Issues restoring backups when migrating from a dying SSD (Yunohost 11) to a new HDD (Yunohost 12) on the same machine

What type of hardware are you using: Old laptop or computer
What YunoHost version are you running: 12.0.9
How are you able to access your server: The webadmin
SSH
Direct access via physical keyboard/screen
Are you in a special context or did you perform specific tweaking on your YunoHost instance ?: SSD is dying, so I am migrating to a new HDD on the same machine

Describe your issue

TL;DR generic problem

When I yunohost backup restore a whole system backup it:

  • takes a long long time (i.e. overnight!)
  • and does not restore apps
  • (it does restore system settings, users and their homes, as well as shared folder)

Same happens regardless if I restore the whole backup from CLI or just a specific app via WebUI.

Actual problem I am running into

Hardware and software

What we’re dealing with is a dedicated AMD64 (Ryzen 5) box with 32 GB RAM and …

before the migration

  • 1 TB NVMe M.2 SSD (Samsung 990 Pro; less than a year old) – let’s call it dyingSSDthis is where the whole Yunohost 11 system is (Ext4) – mounted to /
  • 1 TB SATA HDD (2.5" WD Red; 7 years old) – let’s call it oldHDD1this is where yunohost backups went to (Btrfs) – mounted to /home/yunohost.backup
  • 1 TB SATA HDD (2.5" WD Red; 5 years old) – let’s call it oldHDD2disk from old borg backup server, so still has old borg backups of family PCs (Ext4)

I happily ran Yunohost 11 on that until one was SMART started screaming that the dyingSSD is failing. So I went and bought a new set of disks.

after migration

  • dyingSSD – unmounted and to be removed ASAP, but still alive, it seems
  • oldHDD1 – mounted to /home/yunohost.backup
  • oldHDD2
  • 4 TB SATA HDD (3.5" Seagate IronWolf, new) – let’s call it newHDD1this is where I have Yunohost 12 installed on now (ESP, Btrfs) – mounted to /
  • 4 TB SATA HDD (3.5" WD Red Plus, new) – let’s call it newHDD2this is currently a spare disk, will be added to Btrfs RAID1 with the above disk soon though

(please let’s not discuss the disk/partition layout or backup strategy here; there is a separate thread for that)

What I have done so far

After much trial-and-error (incl. not finding Yunohost 11 official ISO anymore, and failing to install from Yunohost 12 ISO) and help via the Yunohost XMPP/Matrix chat I did the following:

  1. yunohost backup create when still booting Yunohost 11 from dyingSSD and onto oldHDD1 – this ended up being a 220 GB system backup, with all(?) the apps etc.
  2. install Debian 12 on newHDD1 with Btrfs following Installing Debian for Yunohost | Yunohost Documentation
  3. installed Yunohost 12 following Install YunoHost | Yunohost Documentation
  4. ran yunohost backup restore 20241215-193442 following Backups | Yunohost Documentation

The last step took overnight. Even though the logs claim it was done pretty fast, I was sitting through it for many hours watching the CLI and its Preparing archive for restoration message, went to bed, and continued watching that message for another few hours before it finished. iotop was also clearly showing it was working all that time.

What migrated / restored from the backup and what did not

What seems to have migrated well:

  • system settings
  • domains
  • users
  • user homes, including the shared folders
  • /var/mail/ (although oddly enough I don’t have permission to read my own mailbox there)

What did not migrate though is:

  • apps – I cannot see any apps in either user or admin WebUI
  • app settings
  • /home/yunohost.app/ is also empty

So I logged into the admin WebUI and in /yunohost/admin/#/backup/local selected the same system 220 GB backup and checked to restore just my Pelican blog, a redirect to it and some static HTML (webpage FAQ). Again, it took well overnight, and again, the apps are not restored – not even the simple static HTML.

HELP!

While this topic is about how to restore from backups, ultimately I just want to end up with a working Yunohost server again and migrate from the backup (or the SSD).

My current Yunohost 12 install is basically empty and not really working (see bonus issue below), so I am open to more drastic solutions too.

Bonus big problem

After all this – and potentially unconnected – I am currently in a situation where I can’t update packages either through yunohost or apt.

sudo yunohost tools upgrade
Error: You cannot do this right now because dpkg/APT (the system package managers) seems to be in a broken state… You can try to solve this issue by connecting through SSH and running `sudo apt install --fix-broken` and/or `sudo dpkg --configure -a` and/or `sudo dpkg --audit`.

sudo apt install --fix-broken
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up yunohost (12.0.9) ...
Regenerating configuration, this might take a while...
Error: Could not find scrutiny in the list of installed apps: 
 * 
dpkg: error processing package yunohost (--configure):
 installed yunohost package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 yunohost
E: Sub-process /usr/bin/dpkg returned an error code (1)

sudo dpkg --configure -a
Setting up yunohost (12.0.9) ...
Regenerating configuration, this might take a while...
Error: Could not find scrutiny in the list of installed apps: 
 * 
dpkg: error processing package yunohost (--configure):
 installed yunohost package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 yunohost

sudo dpkg --audit
The following packages are only half configured, probably due to problems
configuring them the first time.  The configuration should be retried using
dpkg --configure <package> or the configure menu option in dselect:
 yunohost             manageable and configured self-hosting server

sudo dpkg --configure yunohost
Setting up yunohost (12.0.9) ...
Regenerating configuration, this might take a while...
Error: Could not find scrutiny in the list of installed apps: 
 * 
dpkg: error processing package yunohost (--configure):
 installed yunohost package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 yunohost

Share relevant logs or error messages

Logs relevant to (post)install and restoring the system backup from CLI:

Logs relevant to restoring my blog and static HTML via the WebAdmin:

So you have a working backup?
Why not just reinstall yunohost 11 from here Index of /images/releases_archive/
That would be less complicated

1 Like

So you have a working backup?

On oldHDD1, yes. At least, I think it’s a working backup.

I also still have everything on dyingSSD too.

Why not just reinstall yunohost 11 from here Index of /images/releases_archive/
That would be less complicated

That is not a bad idea, and was something initially I tried to do. The reason I ended up the way I did was:

  1. I could not find that link, so I brute-forced the typical download URL to get yunohost-bullseye-11.2-amd64-stable.iso
  2. for some reason, when I installed it, it did not boot – in hindsight, it could have been that I booted the USB in legacy BIOS mode and that’s why ESP was not created
  3. someone (I think it was one of the devs) on the Yunohost chat was very confident that Debian 12 + Yunohost 12 and then restoring the Yunohost 11 backup is the best way to go about it

I can try installing 11 tomorrow again and put it onto newHDD2, so if that fails, I still have 12 on newHDD1 :person_shrugging:

You may also install debian 11 and install yunohost 11 using https://install.yunohost.org/bullseye script

1 Like

It is very good to know the old scripts are around still, thank you. I’m surprised I did not find that in the documentation.

Let’s call trying to install from YNH 11 ISO plan A, and going the long way over Debian 11 ISO (+ YNH bullseye script) plan B.

1 Like

Hmmm the sha256sum --check does not pass for either of the Yunohost 11 images:

Just downloaded and compared…

$ sha256sum yunohost-bullseye-11.2-amd64-stable.iso
d18bf0aa2be09bf4a893f109097b8cb94918c7c845469098bc30e85ba1ddce92  yunohost-bullseye-11.2-amd64-stable.iso
$ cat yunohost-bullseye-11.2-amd64-stable.iso.sha256sum
d18bf0aa2be09bf4a893f109097b8cb94918c7c845469098bc30e85ba1ddce92  debian-11.11.0-amd64-netinst-modified.iso

And also got checked gpg signature…

$ curl https://forge.yunohost.org/yunohost.asc | gpg --import
$ wget -q https://repo.yunohost.org/images/releases_archive/yunohost-bullseye-11.2-amd64-stable.iso.sig
$ gpg --verify yunohost-bullseye-11.2-amd64-stable.iso.sig yunohost-bullseye-11.2-amd64-stable.iso
gpg: Signature faite le Sam 28 Sep 2024 20:00:00 CEST
gpg:                avec la clef RSA 1904C5B42E4856DCD4E9CF96360AAF3259A3E6FF
gpg: Bonne signature de « YunoHost <build@yunohost.org> » [inconnu]
gpg: Attention : cette clef n'est pas certifiée avec une signature de confiance.
gpg:             Rien n'indique que la signature appartient à son propriétaire.
Empreinte de clef principale : 1904 C5B4 2E48 56DC D4E9  CF96 360A AF32 59A3 E6FF

Looks fine to me…

1 Like

You’re right. That way they match. Thanks.

I see what I was doing wrong. I should have ran sha256sum --check against the .iso.sha256 file (and not the .iso). :man_facepalming:

But even then (and only for 11.2), I get:

sha256sum --check yunohost-bullseye-11.2-amd64-stable.iso.sha256sum
sha256sum: debian-11.11.0-amd64-netinst-modified.iso: No such file or directory
debian-11.11.0-amd64-netinst-modified.iso: FAILED open or read
sha256sum: WARNING: 1 listed file could not be read

Which makes sense, since yunohost-bullseye-11.2-amd64-stable.iso.sha256sum does actually point to the wrong file:

d18bf0aa2be09bf4a893f109097b8cb94918c7c845469098bc30e85ba1ddce92  debian-11.11.0-amd64-netinst-modified.iso

But yes, should be fine. It’s just a filename “typo”.


update, so I don’t spam with another post:

These did not work:

  • yunohost-bullseye-11.2-amd64-stable.iso works, but the installer gets stuck at not finding any root partition or something similar and partman won’t do anything
  • yunohost-bullseye-11.0.9-amd64-stable.iso does not even show up as a bootable medium in EFI menu

So this is what I’m doing now:

  1. installed Debian 11 on newHDD2debian-11.11.0-amd64-netinst.iso booted and worked fine
  2. ran curl https://install.yunohost.org/bullseye | bash on top of that
  3. mounted oldHDD1 to /home/yunohost.backup with noatime
  4. successfully ran yunohost backup restore 20241215-193442 --system – took several hours
  5. successfully ran yunohost backup restore 20241215-193442 --apps pelican redirect__3 my_webapp to restore my Pelican blog and some static pages (600 MB) – took 4h
  6. successfully ran yunohost backup restore 20241215-193442 --apps pihole nextcloud to restore, well Pi-Hole and Nextcloud (cca. 20 GB) – took 3h40’
    • Pi-Hole itself seems to work, but if I use it as DNS, I can’t access anything on the web anymore. But I had a slightly odd set-up already before, IIRC. I’ll look into it later.
  7. successfully ran yunohost backup restore 20241215-193442 --apps biboumi unattended_upgrades searxng scrutiny piped to restore my Biboumi IRC-XMPP bridge (XMPP works already), search, SMART monitor, and yt sane-ifier (850 MB) – took 5h20’
  8. succesfully ran yunohost backup restore 20241215-193442 --apps languagetool funkwhale collabora calibreweb converse
    • Funkwhale seems to have issues accessing the shared music folder

If anything big happens, I’ll start a new comment; else just update this one.

Well, something happened alright :weary:

After a (mostly) successful restoration from backups, I decided to migrate to Yunohost 12.

So I triggered the migration with:

sudo yunohost tools migrations run --accept-disclaimer

After I booted, I could not access the server other than through physical access anymore.

Here’s the diagnosis:

https://paste.yunohost.org/uhijupiqay

While I did have to force a static local IP for the new MAC address on the router, I don’t think (but could be wrong) I’ve done anything that would trigger this issue. (I was probably wrong and actually broke something on the router.)

I do get e-mails forwarded from the YNH server to my default (non-YNH) e-mail address, though.


I have since removed PiHole, in case that is causing it.

sudo yunohost app remove pihole
sudo yunohost service restart dnsmasq

But this does not seem to have fixed it.


To double-check whether the access issue is on the router or in the system, I booted Yunohost 11 from the dyingSSD and also got the same issues. So I guess, I did mess up something on my router when I was cleaning up old entries …oh man, I hate setting up NAT, I’m really bad at it.

Well, at least it’s not a problem on Yunohost’s side …that’s something :sweat_smile:

After two days of fighting with my router, it started working again. I have no idea what I did to break it and even less of a clue why it just magically started working again. But I’m not complaining :sweat_smile:

Status quo:

  • replaced Pi-Hole with AdGuard Home, but need to figure out how to add my YNH server’s domains to it, so I can still access the server when I used AdGuard as my DNS in the same LAN
  • removed Funkwhale, since it did not work. Might bring it back up later or replace it with something else
  • installed Prosody from testing branch – it works fine, but I had to rmdir /home/yunohost.app/metronome to get it to install
  • Biboumi (IRC to XMPP bridge) does not seem to install without Metronome
  • Piped works fine
  • I had to remove and re-install SearxNG to get it to work
  • CalibreWeb gives me a 502 error – I’ll try to figure that out later
  • Nextcloud works fine, as do Collabora and LanguageTool

In adguard > filters > DNS rewrites > add dns rewrite

1 Like

Awesome, thanks. That was exactly what I needed. You saved me quite a bit of reading :bowing_man:

1 Like