Hi all,
TL;DR
The upgrade of Nextcloud from 18 to 20 takes much longer than I anticipated. I think the cause is the size of the filecache, combined with HDD media. Can that cache be cleared prior to upgrade?
My YunoHost server
Hardware: Container on Proxmox, NAS-like box
YunoHost version: just upgraded from 3.8 to 4(.1?)
I have access to my server : Through SSH | through the webadmin : webadmin is ‘busy’ atm | direct access via keyboard / screen if necessary
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no
Description of my issue
In the Proxmoxbox is an SSD and some HDD, I moved Yunohost from SSD to HDD because the SSD was not large enough to hold all Nextcloud files.
That was before upgrading: the SSD was full enough that I worried about backup space while performing the upgrade from YNH3.8 to YNH4+. That upgrade went, as far as I can see, without trouble (matrix-synapse needed some help, and a downgrade of an SSL package, both documented by others on the forum).
After upgrading YNH, most apps would upgrade. Nextcloud was one that ended in an error, mysqldump: Got errno 28 on write. YNH got some 20G of free space when starting the upgrade, so not very little, but also not a lot.
I extended the disk space by 150G to have some headroom before restarting the upgrade last night. This morning there was an nginX gateway timeout on the webinterface, I could not reconnect. Yunohost commands via SSH would just seem to hang, while the system was not very busy (low CPU and low RAM usage).
The YNH-api log ends in:
# tail -f yunohost-api.log
2021-01-29 00:05:22,345 DEBUG yunohost.hook <lambda> - [331.45] 24006 + ynh_mysql_dump_db --database=nextcloud
2021-01-29 00:05:22,345 DEBUG yunohost.hook <lambda> - [331.45] 24006 + local legacy_args=d
2021-01-29 00:05:22,345 DEBUG yunohost.hook <lambda> - [331.45] 24006 + args_array=([d]=database=)
2021-01-29 00:05:22,345 DEBUG yunohost.hook <lambda> - [331.45] 24006 + local -A args_array
2021-01-29 00:05:22,346 DEBUG yunohost.hook <lambda> - [331.45] 24007 + local database
2021-01-29 00:05:22,346 DEBUG yunohost.hook <lambda> - [331.45] 24007 + ynh_handle_getopts_args --database=nextcloud
2021-01-29 00:05:22,346 DEBUG yunohost.hook <lambda> - [331.45] 24007 + set +o xtrace
2021-01-29 00:05:22,346 DEBUG yunohost.hook <lambda> - [331.45] 24007 ++ cat /etc/yunohost/mysql
2021-01-29 00:05:22,547 DEBUG yunohost.hook <lambda> - [331.45] 24208 + mysqldump --user=root --password=Ex7eziT2uW --single-transaction --skip-dump-date nextcloud
2021-01-29 00:05:22,547 DEBUG yunohost.hook <lambda> - [331.45] 24208 [########++++........] > Backing up the MySQL database...
At first I thought the process had crashed, but comparing disk usage over time, something was still running:
# # before upgrade
# df -hTt ext4
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/yunostmp-vm--104--disk--0 ext4 1.1T 837G 179G 83% /
# # this morning
# df -hTt ext4
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/yunostmp-vm--104--disk--0 ext4 1.1T 910G 107G 90% /
# # a while later
# df -hTt ext4
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/yunostmp-vm--104--disk--0 ext4 1.1T 922G 94G 91% /
iotop gave quite high disk activity, writing a few MB/s.
That let me check the MySQL files,
# du -hs /var/lib/mysql/nextcloud
438G /var/lib/mysql/nextcloud
# ls -hs /var/lib/mysql/nextcloud| grep G
total 438G
438G oc_filecache.ibd
That explains a lot! It also explains where all diskspace went in the first place 
(Nextcloud is mosty filled by snapshots and movies from our telephones, and syncs with some laptop directories, all in all <0,5TB of 1TB SSD)
Looking at the size of this file (400G+), the orinal free disk space (200G-) and the I/O and write speed of the spinning HDD (~5MB/s), it will take the rest of the day before the backup ends in “out of free space”.
What would be the best way forward now? I see these options:
- Do nothing, Nextcloud 18 is quite OK!
- Mount the (now almost empty…) SSD at the right place for backups (/home/yunohost.backup/tmp/nextcloud-pre-upgrade1/apps/nextcloud/backup ?) so only reading MySQL is from the slow disk, but writing is to another, faster disk
- Extend the space for YNH so that it got enough room for the backup
- Kill the upgrade and clear the cache before upgrading
The first option is easiest but least interesting. The last option has my preference. I am not sure of the impact of killing the upgrade (is it handled gracefully? My experience with small upgrades: yes, it is) and how to clear the Nextcloud filecache (what I found as first results, is that cache only can be cleared for deleted files, if at all)
Thanks for any pointers!
