All apps gone + some users gone + YNH version rolled back 😰 + wrong certs after reset due to a stuck app upgrade

hook · May 24, 2026, 6:28pm

What type of hardware are you using: Old laptop or computer
What YunoHost version are you running: 12.0.9
Are you in a special context or did you perform specific tweaking on your YunoHost instance ?: not really

Describe your issue

What I did

(the trigger happened a few days ago, so I’m pulling this from memory)

I updated first the system (I remember a php package) and later a few apps through the YunoHost web interface.
One of the apps (I think it was Piped, but am not sure) got stuck in its update and the progress bar did not budge for a long time, preventing me from using the admin UI for anything else.
At some point, I logged in via ssh and restarted the server. I thought that was a smart idea, due to the several kernel patches recently and I decided to take the opportunity to restart just in case.
When the server got back up, things were very wrong

What is the situation now

SSL certificate is wrong – I can still log in through a (non-Firefox) browser
SSH fingerprint is wrong – I can still log in, if I accept the new fingerprint
all the ynh apps are gone
- YNH portal is empty and says apps are missing
- yunohost app list shows an empty list
- /home/yunohost.app is empty
- even the binaries of the apps (e.g. prosody, pelican) are gone
oddly enough yunohost diagnosis says all is OK with base system and apps

Luckily /home/yunohost.backup/ and /home/yunohost.multimedia/Shared seem to be OK. At least something …

Needless to say, this is a very bad situation and I would be thankful for any help.

Share relevant logs or error messages

Happy to share whatever is relevant, once we identify where to start looking

otm33 · May 24, 2026, 6:33pm

What does return ls -l /var/www ?

jarod5001 · May 24, 2026, 6:44pm

Do you have mount points?

hook · May 24, 2026, 7:51pm

/var/www
└── html
   └── index.nginx-debian.html

hook · May 24, 2026, 8:06pm

My /etc/fstab is:

# <file system>                                                                 <mount point>                   <type>  <options>                                               <dump>  <pass>

## btrfs @ WD 4TB (TODO: put in RAID1 with Seagate 4TB; add reserve)
UUID=fead4eeb-7a1e-45b0-84a5-8a0b98b82b4f               /                                               btrfs   noatime,subvol=@rootfs                  0               0

## ESP @ WD 4TB (TODO: make backup on Seagate 4TB)
UUID=3A67-609C                                                                  /boot/efi                               vfat    umask=0077                                              0   1

## swap @ WD 4TB (TODO: make also on Seagate 4TB)
UUID=93e39306-0943-4f41-a610-07478c4b0b69               none                                    swap    sw                                                              0               0

## TODO: tmpfs

## system backups @ WD 1TB
UUID=37d31240-58a5-475b-aa65-e30669141145               /home/yunohost.backup   btrfs   subvol=@ynh_backup,noatime              0               0

## external backups @ WD 1TB
UUID=37d31240-58a5-475b-aa65-e30669141145               /home/borgwarehouse             btrfs   subvol=@borg_backup,noatime             0               0

## old Olimex server disk @ WD 1 TB
UUID=31df0752-b981-4e7c-90dc-50adeb305c35               /mnt/olimex_server              ext4    noatime,noauto                                  0               0

## dying SSD
UUID=5c417858-46a6-4361-8df0-66281c5a4e4e               /mnt/dying_ssd                  ext4    noauto                                                  0               0


## TODO: second WD 1TB

and lsblk --fs says:

NAME   FSTYPE FSVER LABEL   UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                             
└─sda1 btrfs                37d31240-58a5-475b-aa65-e30669141145  441.4G    52% /home/borgwarehouse
                                                                                /home/yunohost.backup
sdb                                                                             
├─sdb1                                                                          
└─sdb2 ext4   1.0           31df0752-b981-4e7c-90dc-50adeb305c35                
sdc                                                                             
├─sdc1 vfat   FAT32         86B8-F400                                           
├─sdc2 swap   1             0f80d3c5-80cf-413c-9155-e66b5fd2234e                
└─sdc3 btrfs        root@wd af4d5763-2325-44c6-b9f0-69741db1a7b3                
sdd                                                                             
├─sdd1 swap   1             56ed04b7-8b35-4be6-80a3-c5aea4b1ca3d                
├─sdd2 vfat   FAT32         3A67-609C                             463.4M     2% /boot/efi
└─sdd3 btrfs

btrfs filesystem show --all-devices 
Label: '@'  uuid: fead4eeb-7a1e-45b0-84a5-8a0b98b82b4f
        Total devices 1 FS bytes used 154.40GiB
        devid    1 size 3.63TiB used 157.02GiB path /dev/sdd3

Label: 'root@wd'  uuid: af4d5763-2325-44c6-b9f0-69741db1a7b3
        Total devices 1 FS bytes used 561.49GiB
        devid    1 size 3.63TiB used 572.02GiB path /dev/sdc3

Label: none  uuid: 37d31240-58a5-475b-aa65-e30669141145
        Total devices 1 FS bytes used 476.64GiB
        devid    1 size 921.51GiB used 485.02GiB path /dev/sda1

But I get a warning/error during boot that it gave up on waiting for a suspend/resume device. I also notice that the swap is missing.

Even if I do swapon --all I get:

swapon: cannot find the device for UUID=93e39306-0943-4f41-a610-07478c4b0b69

otm33 · May 24, 2026, 9:40pm

Could your system have booted on the wrong disk ?
I see

sdd                                                                             
├─sdd1 swap   1             56ed04b7-8b35-4be6-80a3-c5aea4b1ca3d                
├─sdd2 vfat   FAT32         3A67-609C                             463.4M     2% /boot/efi
└─sdd3 btrfs

The root file system shouldn’t be on that disk ? (partition sdd 3)
Can you run

findmnt

otm33 · May 24, 2026, 9:54pm

I guess grub got installed on sdc…?

hook · May 24, 2026, 9:56pm

Oooh, I haven’t heard of findmnt before – this is useful!

TARGET                                                  SOURCE              FSTYPE      OPTIONS
/                                                       /dev/sdd3[/@rootfs] btrfs       rw,noatime,space_cache=v2,subvolid=256,subvol=/@rootfs
├─/sys                                                  sysfs               sysfs       rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security                                securityfs          securityfs  rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup                                      cgroup2             cgroup2     rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
│ ├─/sys/fs/pstore                                      pstore              pstore      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware/efi/efivars                           efivarfs            efivarfs    rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf                                         bpf                 bpf         rw,nosuid,nodev,noexec,relatime,mode=700
│ ├─/sys/kernel/debug                                   debugfs             debugfs     rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/tracing                                 tracefs             tracefs     rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/fuse/connections                            fusectl             fusectl     rw,nosuid,nodev,noexec,relatime
│ └─/sys/kernel/config                                  configfs            configfs    rw,nosuid,nodev,noexec,relatime
├─/proc                                                 proc                proc        rw,nosuid,nodev,noexec,relatime,hidepid=invisible
│ └─/proc/sys/fs/binfmt_misc                            systemd-1           autofs      rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15243
│   └─/proc/sys/fs/binfmt_misc                          binfmt_misc         binfmt_misc rw,nosuid,nodev,noexec,relatime
├─/dev                                                  udev                devtmpfs    rw,nosuid,relatime,size=16096792k,nr_inodes=4024198,mode=755,inode64
│ ├─/dev/pts                                            devpts              devpts      rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ ├─/dev/shm                                            tmpfs               tmpfs       rw,nosuid,nodev,inode64
│ ├─/dev/hugepages                                      hugetlbfs           hugetlbfs   rw,relatime,pagesize=2M
│ └─/dev/mqueue                                         mqueue              mqueue      rw,nosuid,nodev,noexec,relatime
├─/run                                                  tmpfs               tmpfs       rw,nosuid,nodev,noexec,relatime,size=3221004k,mode=755,inode64
│ ├─/run/lock                                           tmpfs               tmpfs       rw,nosuid,nodev,noexec,relatime,size=5120k,inode64
│ ├─/run/credentials/systemd-sysusers.service           ramfs               ramfs       ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/credentials/systemd-sysctl.service             ramfs               ramfs       ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/credentials/systemd-tmpfiles-setup-dev.service ramfs               ramfs       ro,nosuid,nodev,noexec,relatime,mode=700
│ ├─/run/credentials/systemd-tmpfiles-setup.service     ramfs               ramfs       ro,nosuid,nodev,noexec,relatime,mode=700
│ └─/run/user/44365                                     tmpfs               tmpfs       rw,nosuid,nodev,relatime,size=3221000k,nr_inodes=805250,mode=700,uid=44365,gid=44365,inode64
├─/home/borgwarehouse                                   /dev/sdb1[/@borg_backup]
│                                                                           btrfs       rw,noatime,space_cache,subvolid=258,subvol=/@borg_backup
├─/home/yunohost.backup                                 /dev/sdb1[/@ynh_backup]
│                                                                           btrfs       rw,noatime,space_cache,subvolid=257,subvol=/@ynh_backup
└─/boot/efi                                             /dev/sdd2           vfat        rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-r

/dev/disk/by-uuid/fead4eeb-7a1e-45b0-84a5-8a0b98b82b4f -> ../../sdd3 – so this looks like it should be correct.

Also sdc and sdd should be in btrfs raid1 (will need to double-check if I forgot to do that, TBH; I have it documented as if I did).

In general, the two 4 TB disks (one WD, one Seagate) should be in a btrfs mirror, so other than the /boot/efi partition, they should be identical.

otm33 · May 24, 2026, 10:01pm

ok then false lead…

btrfs subvolume list /

hook · May 24, 2026, 11:25pm

ID 256 gen 12479 top level 5 path @rootfs

hook · May 28, 2026, 8:36pm

Any other ideas?

I’d really like to get my home server back and don’t want to mess around on my own, without first figuring out what went wrong and if there’s anything I can do to restore things.

hook · May 31, 2026, 10:11pm

OK, I think I made some progress towards what went wrong – or at least a new discovery.

I just noticed that a few users are missing (the ones I added the last) and that there YunoHost version is reported as 12.0.9, which is way older than I was running, as I was regularly updating (i.e. I should be around 12.1.40).

So far I have two suspicions – neither I feel comfortable with trying to fix without being sure and knowing what to do:

could it be that the failed update triggered a faulty backup restore and it pulled from a really old backup and also for some reason a whole system backup instead of just the app’s?
since I have two disks in Btrfs RAID1, I may have messed something up and it was not mirroring correctly, but messed something up and when it rebooted, something happened to the primary disk and it booted from the second one and it’s an old snapshot or something?

Help still very welcome. We’re leaving the “I’ll buy you a beer at FOSDEM” and moving into the “I’ll ship a crate of beer to wherever you are” territory.