Restore problems from manually compressed backup

Hi all,

I’ve some backup/restore issues after a fresh install. Explanation below.

My YunoHost server

Hardware:

admin@yuno:~$ sudo raspinfo 
System Information
------------------

Raspberry Pi 3 Model B Plus Rev 1.3
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"

Raspberry Pi reference 2022-08-08
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 3160118e7105880e064923c3bd4769478ed1955c, stage3

Linux yuno.one 5.15.32-v8+ #1538 SMP PREEMPT Thu Mar 31 19:40:39 BST 2022 aarch64 GNU/Linux
Revision        : a020d3
Serial          : 000000005cc285c6
Model           : Raspberry Pi 3 Model B Plus Rev 1.3
Throttled flag  : throttled=0x80008
Camera          : supported=0 detected=0, libcamera interfaces=0
[...]

YunoHost version:

admin@yuno:~$ sudo yunohost --version
yunohost: 
  repo: stable
  version: 11.0.9.14
yunohost-admin: 
  repo: stable
  version: 11.0.10
moulinette: 
  repo: stable
  version: 11.0.9
ssowat: 
  repo: stable
  version: 11.0.9

I have access to my server : Through SSH
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no

Description of my issue

After spotting some SD card error on my old instance, I’ve created a backup with the CLI, named backup-rpi-yuno.one.
I installed a frsh Yunohost server on a new SD card, but I struggled a little to make it not failing directly for a restore, because I couldn’t see it with the yunohost backup list command. Indeed, the backup had been created as a folder, and was not recognized by Yunohost until I tar.gz it (not even with tgz extension, quite weird). Well, this issue resolved, I have been able to go further, after makin a symlink towards the attached disk hosting the tar.gz file:

admin@yuno:~$ sudo yunohost backup list
archives: backup-rpi-yuno.one

admin@yuno:~$ sudo yunohost backup info backup-rpi-yuno.one
created_at: 2022-09-10 13:59:50
description: Full backup before migration to new SD card.
path: /mnt/seagate/backup-rpi-yuno.one.tar.gz
size: 31652996875

admin@yuno:~$ ll /home/yunohost.backup/archives/
total 4.0K
4.0K -rw-r--r-- 1 root root 3.0K Sep 10 14:02 backup-rpi-yuno.one.info.json
   0 lrwxrwxrwx 1 root root   39 Sep 11 16:07 backup-rpi-yuno.one.tar.gz -> /mnt/seagate/backup-rpi-yuno.one.tar.gz

admin@yuno:~$ tar -tvf /mnt/seagate/backup-rpi-yuno.one.tar.gz 'info.json'
-rw-r--r-- root/root      3023 2022-09-10 14:02 info.json

And here is the content of info.json inside the archive:

{
  "description": "Full backup before migration to new SD card.",
  "created_at": 1662811190,
  "size": 31652996875,
  "size_details": {
    "system": {
      "conf_ldap": 340,
      "data_home": 4745641122,
      "data_multimedia": 6246315,
      "conf_ynh_settings": 2543,
      "conf_ynh_certs": 268762,
      "data_mail": 98484666,
      "data_xmpp": 139264,
      "conf_manually_modified_files": 0
    },
    "apps": {
      "sonarr": 13012515,
      "aeneria": 66615651,
      "ttrss": 125991638,
      "nextcloud": 25039428278,
      "kanboard": 15108997,
      "wallabag2": 191271378,
      "zerobin": 2906970,
      "squid3": 66135,
      "dokuwiki": 26202844,
      "opentracker": 1761218,
      "jirafeau": 1010753,
      "piwigo": 1318658594
    }
  },
  "apps": {
    "sonarr": {
      "version": "3.0.6.1196~ynh2",
      "name": "Sonarr",
      "description": "Gestionnaire de collection de séries pour utilisateurs de Usenet et BitTorrent"
    },
    "aeneria": {
      "version": "1.1.6~ynh2",
      "name": "æneria",
      "description": "Tableau de bord pour analyser votre consomation d'énergie à partir des données d'un Linky et de la météo"
    },
    "ttrss": {
      "version": "20220414~ynh1",
      "name": "Tiny Tiny RSS",
      "description": "Lecteur de flux d’actualité utilisant les protocoles RSS et Atom"
    },
    "nextcloud": {
      "version": "22.2.7~ynh1",
      "name": "Nextcloud",
      "description": "Stockage en ligne, plateforme de partage de fichiers et diverses autres applications"
    },
    "kanboard": {
      "version": "1.2.21~ynh4",
      "name": "Kanboard",
      "description": "Logiciel de gestion de projet Kanban"
    },
    "wallabag2": {
      "version": "2.3.8~ynh4",
      "name": "Wallabag",
      "description": "Une application de lecture-plus-tard auto-hébergeable"
    },
    "zerobin": {
      "version": "1.3.5~ynh2",
      "name": "Zerobin",
      "description": "Pastebin minimaliste où le serveur n'a aucune connaissance des données copiées"
    },
    "squid3": {
      "version": "1.0~ynh3",
      "name": "squid3",
      "description": "Proxy cache pour le web, compatible HTTP, HTTPS, FTP et plus encore."
    },
    "dokuwiki": {
      "version": "2020.07.29~ynh6",
      "name": "Dokuwiki",
      "description": "Wiki léger, simple à utiliser et très polyvalent"
    },
    "opentracker": {
      "version": "1.0~ynh3",
      "name": "OpenTracker",
      "description": "Tracker Bittorrent"
    },
    "jirafeau": {
      "version": "4.4.0~ynh1",
      "name": "Jirafeau",
      "description": "Hébergez simplement un fichier et partagez-le avec un lien unique"
    },
    "piwigo": {
      "version": "12.1.0~ynh1",
      "name": "Piwigo",
      "description": "Galerie de photos open source pour le web"
    }
  },
  "system": {
    "conf_ldap": [
      "/usr/share/yunohost/hooks/backup/05-conf_ldap"
    ],
    "data_home": [
      "/usr/share/yunohost/hooks/backup/17-data_home"
    ],
    "data_multimedia": [
      "/usr/share/yunohost/hooks/backup/18-data_multimedia"
    ],
    "conf_ynh_settings": [
      "/usr/share/yunohost/hooks/backup/20-conf_ynh_settings"
    ],
    "conf_ynh_certs": [
      "/usr/share/yunohost/hooks/backup/21-conf_ynh_certs"
    ],
    "data_mail": [
      "/usr/share/yunohost/hooks/backup/23-data_mail"
    ],
    "data_xmpp": [
      "/usr/share/yunohost/hooks/backup/27-data_xmpp"
    ],
    "conf_manually_modified_files": [
      "/usr/share/yunohost/hooks/backup/50-conf_manually_modified_files"
    ]
  },
  "from_yunohost_version": "4.4.2.10"
}

I’ve tried to restore the full archive, but it failed miserably (a problem related to a no matching hostname, or hostname not set, I don’t remember well).
So I reinstalled the SD card (because the apt update were failing after that), I’ve made a restore test with a simple app, such as ttrss, but also got an error: https://paste.yunohost.org/raw/bugefivixo
I’m surprised about some env variables which do not match my domain name (using maindomain.tld instead of yuno.one):

  YNH_APP_BACKUP_DIR: /home/yunohost.backup/tmp/backup-rpi-maindomain.tld/apps/ttrss/backup
  YNH_APP_BASEDIR: /home/yunohost.backup/tmp/backup-rpi-maindomain.tld/apps/ttrss/settings
  YNH_BACKUP_CSV: /home/yunohost.backup/tmp/backup-rpi-maindomain.tld/backup.csv
  YNH_BACKUP_DIR: /home/yunohost.backup/tmp/backup-rpi-maindomain.tld
  YNH_CWD: /home/yunohost.backup/tmp/backup-rpi-maindomain.tld/apps/ttrss/backup

Did I messed up with my manual tar.gz archive, or is there another problem?

Let me know if you need any other information,
Thanks for you help.

In addition, I’ve tried to restore nextcloud app, but also got errors related to missing files in the archives, which are indeed not present:

admin@yuno:~$ time sudo yunohost backup restore backup-rpi-yuno.one --apps nextcloud --force
Info: Preparing archive for restoration...
Info: Restoring nextcloud...
Info: [....................] > Loading settings...
Info: [+...................] > Validating restoration parameters...
Warning: mv: cannot stat '/home/yunohost.backup/tmp/backup-rpi-yuno.one/apps/nextcloud/backup/etc/nginx/conf.d/yuno.one.d/nextcloud.conf': No such file or directory
Error: Could not restore nextcloud: An error occured inside the app restore script
Info: The operation 'Restore 'nextcloud' from a backup archive' could not be completed. Please share the full log of this operation using the command 'yunohost log share 20220912-001705-backup_restore_app-nextcloud' to get help
Warning: Here's an extract of the logs before the crash. It might help debugging the error:
Info: DEBUG - ++ python3 -c '
Info: DEBUG - import sys
Info: DEBUG - import csv
Info: DEBUG - with open(sys.argv[1], '\''r'\'') as backup_file:
Info: DEBUG -     backup_csv = csv.DictReader(backup_file, fieldnames=['\''source'\'', '\''dest'\''])
Info: DEBUG -     for row in backup_csv:
Info: DEBUG -         if row['\''source'\'']==sys.argv[2].strip('\''"'\''):
Info: DEBUG -             print(row['\''dest'\''])
Info: DEBUG -             sys.exit(0)
Info: DEBUG -     raise Exception('\''Original path for %s not found'\'' % sys.argv[2])
Info: DEBUG -     ' /home/yunohost.backup/tmp/backup-rpi-yuno.one/backup.csv '"/etc/nginx/conf.d/yuno.one.d/nextcloud.conf"'
Info: DEBUG - ++ return 0
Info: DEBUG - + archive_path=/home/yunohost.backup/tmp/backup-rpi-yuno.one/apps/nextcloud/backup/etc/nginx/conf.d/yuno.one.d/nextcloud.conf
Info: DEBUG - + [[ -e /etc/nginx/conf.d/yuno.one.d/nextcloud.conf ]]
Info: DEBUG - ++ dirname /etc/nginx/conf.d/yuno.one.d/nextcloud.conf
Info: DEBUG - + mkdir --parents /etc/nginx/conf.d/yuno.one.d
Info: DEBUG - + mountpoint --quiet /home/yunohost.backup/tmp/backup-rpi-yuno.one
Info: DEBUG - + mv /home/yunohost.backup/tmp/backup-rpi-yuno.one/apps/nextcloud/backup/etc/nginx/conf.d/yuno.one.d/nextcloud.conf /etc/nginx/conf.d/yuno.one.d/nextcloud.conf
Info: WARNING - mv: cannot stat '/home/yunohost.backup/tmp/backup-rpi-yuno.one/apps/nextcloud/backup/etc/nginx/conf.d/yuno.one.d/nextcloud.conf': No such file or directory
Info: DEBUG - + ynh_exit_properly
Info: [....................] > Loading installation settings...
Info: [+++++++++...........] > Removing dependencies...
Warning: E: Unable to locate package nextcloud-ynh-deps
Info: [#########+++........] > Removing the MySQL database...
Warning: Database nextcloud not found
Info: [############+.......] > Removing app main directory...
Info: '/var/www/nextcloud' wasn't deleted because it doesn't exist.
Info: [#############.......] > Removing NGINX web server configuration...
Info: '/etc/nginx/conf.d/yuno.one.d/nextcloud.conf' wasn't deleted because it doesn't exist.
Info: [#############+......] > Removing PHP-FPM configuration...
Info: '/etc/php/7.4/fpm/pool.d/nextcloud.conf' wasn't deleted because it doesn't exist.
Info: [##############+.....] > Removing logrotate configuration...
Info: [###############+++..] > Removing Fail2Ban configuration...
Info: '/etc/fail2ban/jail.d/nextcloud.conf' wasn't deleted because it doesn't exist.
Info: '/etc/fail2ban/filter.d/nextcloud.conf' wasn't deleted because it doesn't exist.
Info: '/etc/cron.d/nextcloud' wasn't deleted because it doesn't exist.
Info: [##################+.] > Removing the dedicated system user...
Warning: The user nextcloud was not found
Info: [####################] > Removal of nextcloud completed
Error: The operation 'Restore 'nextcloud' from a backup archive' could not be completed. Please share the full log of this operation using the command 'yunohost log share 20220912-001705-backup_restore_app-nextcloud' to get help
Error: Nothing was restored

real    54m16.919s
user    27m30.896s
sys     7m44.412s

Full logs: https://paste.yunohost.org/raw/ucosocujag

Here is the content of the directory:

root@yuno:/mnt/seagate/backup-rpi-yuno.one# tree apps/nextcloud/
apps/nextcloud/
├── backup
│   ├── db.sql
│   ├── etc
│   │   ├── cron.d
│   │   ├── fail2ban
│   │   │   ├── filter.d
│   │   │   └── jail.d
│   │   ├── logrotate.d
│   │   ├── nginx
│   │   │   └── conf.d
│   │   │       └── yuno.one.d
│   │   └── php
│   │       └── 7.3
│   │           └── fpm
│   │               └── pool.d
│   ├── home
│   │   └── yunohost.app
│   └── var
│       └── www
└── settings
    ├── actions.toml
    ├── conf
    │   ├── app.src
    │   ├── config_install.json
    │   ├── config.json
    │   ├── extra_php-fpm.conf
    │   ├── f2b_filter.conf
    │   ├── f2b_jail.conf
    │   ├── nextcloud.cron
    │   ├── nginx.conf
    │   └── php-fpm.conf
    ├── config_panel.toml
    ├── doc
    │   ├── DISCLAIMER_fr.md
    │   ├── DISCLAIMER.md
    │   └── screenshots
    │       └── screenshot.png
    ├── hooks
    │   └── post_user_create
    ├── manifest.json
    ├── permissions.yml
    ├── scripts
    │   ├── actions
    │   │   ├── add_multimedia_directories
    │   │   └── disable_maintenance
    │   ├── backup
    │   ├── change_url
    │   ├── _common.sh
    │   ├── config
    │   ├── install
    │   ├── nextcloud-logrotate
    │   ├── remove
    │   ├── restore
    │   ├── upgrade
    │   └── upgrade.d
    │       ├── upgrade.10.sh
    │       ├── upgrade.11.sh
    │       ├── upgrade.12.sh
    │       ├── upgrade.13.sh
    │       ├── upgrade.14.sh
    │       ├── upgrade.15.sh
    │       ├── upgrade.16.sh
    │       ├── upgrade.17.sh
    │       ├── upgrade.18.sh
    │       ├── upgrade.19.sh
    │       ├── upgrade.20.sh
    │       ├── upgrade.21.sh
    │       ├── upgrade.9.sh
    │       └── upgrade.last.sh
    └── settings.yml

26 directories, 44 files

Hi here, does the problem I mentioned is known to someone?

Else, I’ll try to restore apps manually with copying data and config files…