Yunohost in Read Only mode

Hello everyone,
I had a problem after trying to make a backup through the admin interface in 2.4, as I can’t find the error I can’t paste it here.
Anyway, the result is that I now have a read only filesystem and I can’t solve it.
I’ve tried mount -o remount,rw / without success, of course I can’t use fsck since the system is mounted.
I’ve also tried to reboot without success

Here is an excerpt from /var/log/syslog

May 23 07:34:45 ns311349 kernel: EXT3-fs (sda1): error: couldn't mount because of unsupported optional features (240)
May 23 07:34:45 ns311349 kernel: EXT2-fs (sda1): error: couldn't mount because of unsupported optional features (240)
May 23 07:34:45 ns311349 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
May 23 07:34:45 ns311349 kernel: VFS: Mounted root (ext4 filesystem) readonly on device 8:1.
May 23 07:34:45 ns311349 kernel: devtmpfs: mounted
May 23 07:34:45 ns311349 kernel: Freeing unused kernel memory: 1056K (ffffffff825be000 - ffffffff826c6000)
May 23 07:34:45 ns311349 kernel: grsec: time set by /lib/systemd/systemd[systemd:1] uid/euid:0/0 gid/egid:0/0, parent /[swapper/0:0] uid/euid:0/0 gid/egid:0/0
May 23 07:34:45 ns311349 kernel: random: systemd urandom read with 53 bits of entropy available
May 23 07:34:45 ns311349 kernel: random: nonblocking pool is initialized
May 23 07:34:45 ns311349 kernel: Adding 1047548k swap on /dev/sda3.  Priority:-1 extents:1 across:1047548k
May 23 07:34:45 ns311349 kernel: EXT4-fs (sda1): re-mounted. Opts: errors=remount-ro
May 23 07:34:45 ns311349 kernel: EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)

I managed to force a fsck at boot by creating the file forcefsck at the root of the system with:
touch /forcefsck
After a reboot everything went back to normal !

So the problem is in fact with the backup tool, every time I launch a backup I got these errors (in french sorry):

root@myserver:~# yunohost backup create
Attention : mkdir: impossible de créer le répertoire « /home/yunohost.backup/tmp/20160526-065713/data/home/backups »: Système de fichiers accessible en lecture seulement
Attention : mount: le point de montage /home/yunohost.backup/tmp/20160526-065713/data/home/backups n'existe pas
Attention : cp: impossible de créer le répertoire « /home/yunohost.backup/tmp/20160526-065713/data/home/backups »: Système de fichiers accessible en lecture seulement
Attention : cp: impossible de créer le répertoire « /home/yunohost.backup/tmp/20160526-065713/data/home/downloads »: Système de fichiers accessible en lecture seulement
Attention : cp: impossible de créer le répertoire « /home/yunohost.backup/tmp/20160526-065713/data/home/user »: Système de fichiers accessible en lecture seulement

Hello,

This is related to your hard drive or filesystem, not to YunoHost. I would recommend you to start by doing a total backup of your whole system VERY QUICKLY then to start investigating with tools like smartctl, backblocks and other, there is chances that either your hard drive is dying or that your file system is corrupt.

Good luck,

I’m on a dedicated OVH server and when I reboot I’ve no problems at all with anything it’s only when I try to use yunohost backup create
I can copy the result of smartctl if needed or anything else :slight_smile:

Hi, could you paste the output of df -h and ls -hl /home please - if you do not have sensible information with those commands output?

$ df -h
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/root           20G    2,8G   16G  16% /
devtmpfs           987M       0  987M   0% /dev
tmpfs              988M       0  988M   0% /dev/shm
tmpfs              988M     31M  958M   4% /run
tmpfs              5,0M       0  5,0M   0% /run/lock
tmpfs              988M       0  988M   0% /sys/fs/cgroup
/dev/sda2          439G     68G  349G  17% /home
tmpfs              198M       0  198M   0% /run/user/0
$ ls -hl /home
total 56K
drwxr-xr-x  3 admin                   1007 4,0K avril 24 17:10 admin
drwxrwx---  2 root                root     4,0K mai   27 06:40 backups
drwxr-xr-x  3 www-data            www-data 4,0K mai   23 08:46 downloads
drwx------  2 root                root      16K avril 24 15:54 lost+found
drwxrwxr-x+ 2 thomas                 50866 4,0K avril 24 16:44 thomas
drwxr-xr-x  2 vmail               mail     4,0K avril 24 16:16 vmail
drwxr-xr-x  4 root                root     4,0K mai   22 15:20 yunohost.app
drwxr-xr-x  4 root                root     4,0K mai   23 07:09 yunohost.backup
drwxr-xr-x  4 root                root     4,0K mai   25 07:10 yunohost.conf
drwxr-x---  4 debian-transmission www-data 4,0K avril 24 17:12 yunohost.transmission

Thanks, and what’s the output of mount too please? It would be maybe easier to debug that in the xmpp chatroom…

Here is the output of mount:

/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
devtmpfs on /dev type devtmpfs (rw,relatime,size=1010576k,nr_inodes=252644,mode=755)
sysfs on /sys type sysfs (rw,relatime)
proc on /proc type proc (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=23,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/sda2 on /home type ext4 (rw,relatime,data=ordered)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=202224k,mode=700)

Thanks! I really don’t see what can be the origin of the issue you have… One more thing, could you please paste the verbose output of the backup creation, with only the data_home hook to reduce the output: yunohost backup create --hooks data_home --verbose.

Here is the output, sorry it’s quite long :wink:

Exécution des scripts de sauvegarde...
Exécution du script « /usr/share/yunohost/hooks/backup/17-data_home »...
+ . /usr/share/yunohost/helpers
+++ run-parts --list /usr/share/yunohost/helpers.d
++ for helper in '$(run-parts --list /usr/share/yunohost/helpers.d 2>/dev/null)'
++ '[' -r /usr/share/yunohost/helpers.d/filesystem ']'
++ . /usr/share/yunohost/helpers.d/filesystem
+++ CAN_BIND=1
++ for helper in '$(run-parts --list /usr/share/yunohost/helpers.d 2>/dev/null)'
++ '[' -r /usr/share/yunohost/helpers.d/ip ']'
++ . /usr/share/yunohost/helpers.d/ip
++ for helper in '$(run-parts --list /usr/share/yunohost/helpers.d 2>/dev/null)'
++ '[' -r /usr/share/yunohost/helpers.d/mysql ']'
++ . /usr/share/yunohost/helpers.d/mysql
+++ MYSQL_ROOT_PWD_FILE=/etc/yunohost/mysql
++ for helper in '$(run-parts --list /usr/share/yunohost/helpers.d 2>/dev/null)'
++ '[' -r /usr/share/yunohost/helpers.d/package ']'
++ . /usr/share/yunohost/helpers.d/package
++ for helper in '$(run-parts --list /usr/share/yunohost/helpers.d 2>/dev/null)'
++ '[' -r /usr/share/yunohost/helpers.d/print ']'
++ . /usr/share/yunohost/helpers.d/print
++ for helper in '$(run-parts --list /usr/share/yunohost/helpers.d 2>/dev/null)'
++ '[' -r /usr/share/yunohost/helpers.d/setting ']'
++ . /usr/share/yunohost/helpers.d/setting
++ for helper in '$(run-parts --list /usr/share/yunohost/helpers.d 2>/dev/null)'
++ '[' -r /usr/share/yunohost/helpers.d/string ']'
++ . /usr/share/yunohost/helpers.d/string
++ for helper in '$(run-parts --list /usr/share/yunohost/helpers.d 2>/dev/null)'
++ '[' -r /usr/share/yunohost/helpers.d/user ']'
++ . /usr/share/yunohost/helpers.d/user
++ for helper in '$(run-parts --list /usr/share/yunohost/helpers.d 2>/dev/null)'
++ '[' -r /usr/share/yunohost/helpers.d/utils ']'
++ . /usr/share/yunohost/helpers.d/utils
++ find /home/admin /home/backups /home/downloads /home/lost+found /home/pauline /home/thomas /home/vmail /home/yunohost.app /home/yunohost.backup /home/yunohost.conf /home/yunohost.transmission -type d -prune
++ awk -F/ '{print $NF}'
+ for f in '$(find /home/* -type d -prune | awk -F/ '\''{print $NF}'\'')'
+ [[ ! admin =~ ^yunohost|lost\+found ]]
+ ynh_backup /home/admin /home/yunohost.backup/tmp/20160529-161328/data/home/admin 1
+ local SRCPATH=/home/admin
+ local DESTPATH=/home/yunohost.backup/tmp/20160529-161328/data/home/admin
+ local TO_BIND=1
+ local SUDO_CMD=sudo
+ [[ '' = \1 ]]
+ [[ -e /home/admin ]]
+ [[ -n '' ]]
+ [[ ! -e /home/yunohost.backup/tmp/20160529-161328/data/home/admin ]]
+ [[ 1 = \1 ]]
+ [[ 1 = \1 ]]
+ eval sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home/admin
++ sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home/admin
+ sudo mount --rbind /home/admin /home/yunohost.backup/tmp/20160529-161328/data/home/admin
+ sudo mount -o remount,ro /home/yunohost.backup/tmp/20160529-161328/data/home/admin
+ return 0
+ for f in '$(find /home/* -type d -prune | awk -F/ '\''{print $NF}'\'')'
+ [[ ! backups =~ ^yunohost|lost\+found ]]
+ ynh_backup /home/backups /home/yunohost.backup/tmp/20160529-161328/data/home/backups 1
+ local SRCPATH=/home/backups
+ local DESTPATH=/home/yunohost.backup/tmp/20160529-161328/data/home/backups
+ local TO_BIND=1
+ local SUDO_CMD=sudo
+ [[ '' = \1 ]]
+ [[ -e /home/backups ]]
+ [[ -n '' ]]
+ [[ ! -e /home/yunohost.backup/tmp/20160529-161328/data/home/backups ]]
+ [[ 1 = \1 ]]
+ [[ 1 = \1 ]]
+ eval sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home/backups
++ sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home/backups
Attention : mkdir: impossible de créer le répertoire « /home/yunohost.backup/tmp/20160529-161328/data/home/backups »: Système de fichiers accessible en lecture seulement
+ sudo mount --rbind /home/backups /home/yunohost.backup/tmp/20160529-161328/data/home/backups
Attention : mount: le point de montage /home/yunohost.backup/tmp/20160529-161328/data/home/backups n'existe pas
+ CAN_BIND=0
+ echo 'Bind mounting seems to be disabled on your system.'
Bind mounting seems to be disabled on your system.
+ echo 'You have maybe to check your apparmor configuration.'
You have maybe to check your apparmor configuration.
+ mountpoint -q /home/yunohost.backup/tmp/20160529-161328/data/home/backups
+ eval sudo rm -rf /home/yunohost.backup/tmp/20160529-161328/data/home/backups
++ sudo rm -rf /home/yunohost.backup/tmp/20160529-161328/data/home/backups
++ dirname /home/yunohost.backup/tmp/20160529-161328/data/home/backups
+ eval sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home
++ sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home
+ eval sudo cp -a /home/backups /home/yunohost.backup/tmp/20160529-161328/data/home/backups
++ sudo cp -a /home/backups /home/yunohost.backup/tmp/20160529-161328/data/home/backups
Attention : cp: impossible de créer le répertoire « /home/yunohost.backup/tmp/20160529-161328/data/home/backups »: Système de fichiers accessible en lecture seulement
+ for f in '$(find /home/* -type d -prune | awk -F/ '\''{print $NF}'\'')'
+ [[ ! downloads =~ ^yunohost|lost\+found ]]
+ ynh_backup /home/downloads /home/yunohost.backup/tmp/20160529-161328/data/home/downloads 1
+ local SRCPATH=/home/downloads
+ local DESTPATH=/home/yunohost.backup/tmp/20160529-161328/data/home/downloads
+ local TO_BIND=1
+ local SUDO_CMD=sudo
+ [[ '' = \1 ]]
+ [[ -e /home/downloads ]]
+ [[ -n '' ]]
+ [[ ! -e /home/yunohost.backup/tmp/20160529-161328/data/home/downloads ]]
+ [[ 0 = \1 ]]
++ dirname /home/yunohost.backup/tmp/20160529-161328/data/home/downloads
+ eval sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home
++ sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home
+ eval sudo cp -a /home/downloads /home/yunohost.backup/tmp/20160529-161328/data/home/downloads
++ sudo cp -a /home/downloads /home/yunohost.backup/tmp/20160529-161328/data/home/downloads
Attention : cp: impossible de créer le répertoire « /home/yunohost.backup/tmp/20160529-161328/data/home/downloads »: Système de fichiers accessible en lecture seulement
+ for f in '$(find /home/* -type d -prune | awk -F/ '\''{print $NF}'\'')'
+ [[ ! lost+found =~ ^yunohost|lost\+found ]]
+ for f in '$(find /home/* -type d -prune | awk -F/ '\''{print $NF}'\'')'
+ [[ ! pauline =~ ^yunohost|lost\+found ]]
+ ynh_backup /home/pauline /home/yunohost.backup/tmp/20160529-161328/data/home/pauline 1
+ local SRCPATH=/home/pauline
+ local DESTPATH=/home/yunohost.backup/tmp/20160529-161328/data/home/pauline
+ local TO_BIND=1
+ local SUDO_CMD=sudo
+ [[ '' = \1 ]]
+ [[ -e /home/pauline ]]
+ [[ -n '' ]]
+ [[ ! -e /home/yunohost.backup/tmp/20160529-161328/data/home/pauline ]]
+ [[ 0 = \1 ]]
++ dirname /home/yunohost.backup/tmp/20160529-161328/data/home/pauline
+ eval sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home
++ sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home
+ eval sudo cp -a /home/pauline /home/yunohost.backup/tmp/20160529-161328/data/home/pauline
++ sudo cp -a /home/pauline /home/yunohost.backup/tmp/20160529-161328/data/home/pauline
Attention : cp: impossible de créer le répertoire « /home/yunohost.backup/tmp/20160529-161328/data/home/pauline »: Système de fichiers accessible en lecture seulement
+ for f in '$(find /home/* -type d -prune | awk -F/ '\''{print $NF}'\'')'
+ [[ ! thomas =~ ^yunohost|lost\+found ]]
+ ynh_backup /home/thomas /home/yunohost.backup/tmp/20160529-161328/data/home/thomas 1
+ local SRCPATH=/home/thomas
+ local DESTPATH=/home/yunohost.backup/tmp/20160529-161328/data/home/thomas
+ local TO_BIND=1
+ local SUDO_CMD=sudo
+ [[ '' = \1 ]]
+ [[ -e /home/thomas ]]
+ [[ -n '' ]]
+ [[ ! -e /home/yunohost.backup/tmp/20160529-161328/data/home/thomas ]]
+ [[ 0 = \1 ]]
++ dirname /home/yunohost.backup/tmp/20160529-161328/data/home/thomas
+ eval sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home
++ sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home
+ eval sudo cp -a /home/thomas /home/yunohost.backup/tmp/20160529-161328/data/home/thomas
++ sudo cp -a /home/thomas /home/yunohost.backup/tmp/20160529-161328/data/home/thomas
Attention : cp: impossible de créer le répertoire « /home/yunohost.backup/tmp/20160529-161328/data/home/thomas »: Système de fichiers accessible en lecture seulement
+ for f in '$(find /home/* -type d -prune | awk -F/ '\''{print $NF}'\'')'
+ [[ ! vmail =~ ^yunohost|lost\+found ]]
+ ynh_backup /home/vmail /home/yunohost.backup/tmp/20160529-161328/data/home/vmail 1
+ local SRCPATH=/home/vmail
+ local DESTPATH=/home/yunohost.backup/tmp/20160529-161328/data/home/vmail
+ local TO_BIND=1
+ local SUDO_CMD=sudo
+ [[ '' = \1 ]]
+ [[ -e /home/vmail ]]
+ [[ -n '' ]]
+ [[ ! -e /home/yunohost.backup/tmp/20160529-161328/data/home/vmail ]]
+ [[ 0 = \1 ]]
++ dirname /home/yunohost.backup/tmp/20160529-161328/data/home/vmail
+ eval sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home
++ sudo mkdir -p /home/yunohost.backup/tmp/20160529-161328/data/home
+ eval sudo cp -a /home/vmail /home/yunohost.backup/tmp/20160529-161328/data/home/vmail
++ sudo cp -a /home/vmail /home/yunohost.backup/tmp/20160529-161328/data/home/vmail
Attention : cp: impossible de créer le répertoire « /home/yunohost.backup/tmp/20160529-161328/data/home/vmail »: Système de fichiers accessible en lecture seulement
+ for f in '$(find /home/* -type d -prune | awk -F/ '\''{print $NF}'\'')'
+ [[ ! yunohost.app =~ ^yunohost|lost\+found ]]
+ for f in '$(find /home/* -type d -prune | awk -F/ '\''{print $NF}'\'')'
+ [[ ! yunohost.backup =~ ^yunohost|lost\+found ]]
+ for f in '$(find /home/* -type d -prune | awk -F/ '\''{print $NF}'\'')'
+ [[ ! yunohost.conf =~ ^yunohost|lost\+found ]]
+ for f in '$(find /home/* -type d -prune | awk -F/ '\''{print $NF}'\'')'
+ [[ ! yunohost.transmission =~ ^yunohost|lost\+found ]]
Traceback (most recent call last):
  File "/usr/bin/yunohost", line 212, in <module>
    password=opts.password, parser_kwargs={'top_parser': parser}
  File "/usr/lib/python2.7/dist-packages/moulinette/__init__.py", line 138, in cli
    moulinette.run(args, output_as=output_as, password=password)
  File "/usr/lib/python2.7/dist-packages/moulinette/interfaces/cli.py", line 352, in run
    ret = self.actionsmap.process(args, timeout=5)
  File "/usr/lib/python2.7/dist-packages/moulinette/actionsmap.py", line 467, in process
    return func(**arguments)
  File "/usr/lib/moulinette/yunohost/backup.py", line 176, in backup_create
    filesystem.mkdir(tmp_hooks_dir, 0750, True, uid='admin')
  File "/usr/lib/python2.7/dist-packages/moulinette/utils/filesystem.py", line 36, in mkdir
    mkdir(head, mode, parents, uid, gid, force)
  File "/usr/lib/python2.7/dist-packages/moulinette/utils/filesystem.py", line 44, in mkdir
    os.mkdir(path, mode)
OSError: [Errno 30] Read-only file system: '/home/yunohost.backup/tmp/20160529-161328/hooks'

After that if I don’t do touch /forcefsck followed by a reboot I can’t use my system.

Thanks for that!

So, the first thing to absolutely not do after is rm -rf /home/yunohost.backup/tmp regarding the result… But if you restart your system after that, it’s okay!

Could you try to change the file /usr/share/yunohost/helpers.d/filesystem line 38 as follow please:

-            sudo mount -o remount,ro "${DESTPATH}" >/dev/null 2>&1 \
-              || sudo mount -o remount,ro,bind "${SRCPATH}" "${DESTPATH}" \
+            sudo mount -o remount,ro,bind "${SRCPATH}" "${DESTPATH}" \

And try to execute again the backup… Also, if it fails again, can you paste the output of mount before restarting please?

Thank you for your help in debugging this issue!

Hello,

It seems to work, I don’t have time to wait until the backup is over but at least i has started the creation of the archive !
Thank you very much @jerome !

Glad to know that!! :slight_smile: Thanks for your report and tests, it has been integrated into the new v2.4.0.6!

So the file you asked me to modify will be overwritten with the next release, right ?

Yes, that’s it!