Migration 29 (Postgresql 13 -> 15) failed after upgrade to Yunohost 12

Support
, ,
1

What type of hardware are you using: VPS bought online
What YunoHost version are you running: 12.0.11
How are you able to access your server: The webadmin
Are you in a special context or did you perform specific tweaking on your YunoHost instance ?: no

Describe your issue

After running the migration Yunohost 11 → 12 and the post-uprade migrations, migration 29 (Postgreqsl 13 → 15) fails and is never ending.

The log shows some localized messages concerning no disk space available, but I checked with df that 27.6GB free disk space is available.

Originally I wrote this at the end of my first post (Upgrade to Yunohost 12 failed with unspecific error - #26 by matalb), but this new post has a specific title and I hope for specific help.

Since I don’t undestand the messages, I hope there will be some help with that. Any help is appreciated!

Share relevant logs or error messages

Migration log: https://paste.yunohost.org/raw/zivomoveju

2

Can you share the content of /var/log/postgresql/pg_upgradecluster-13-15-main.dw7h

What’s the output of du -sh /var/lib/postgresql/13/main ?

3

The output is:

51G	/var/lib/postgresql/13/main
4 
/var/log/postgresql/pg_upgradecluster-13-15-main.dw7h

is a directory. Its content is empty.

5

Your database is 51G large. You need twice that of free space to migrate your database.

A workaround is to set the temporary folder on a larger disk
export PGHOST=/path/to/larger/disk/tmp then rerun the migration

Edit : I have searched how to optimize postgresql database, there are some commands “VACUUM FULL” “ANALYZE” “REINDEX DATABASE”. I didn’t read the whole thing but it is possible to reduce the database size on disk by removing leftovers

2 Likes
6

Now I have additional disk storage at /mnt/plusdisk via nfs. df shows 104857280 free bytes, although I ordered 250GB. So it’s only almost double of 51G.

Should I try the migration?

7

I managed to set the quota to 250GB, the maximum. I will run the migration accoring to your recommendation.

8

I ran this migration command

sudo mkdir /mnt/plusdisk/tmp
export PGHOST=/mnt/plusdisk/tmp
sudo yunohost tools migrations run

which resulted in an error message, here is the log:
https://paste.yunohost.org/raw/rucoxigopo

The disk space should be sufficient, df -h output:

myipaddress:/voln353211a2                       250G  384K  250G   1% /mnt/plusdisk
9

Error: could not stop old cluster, please do that manually
Try to stop postgresql 13 before running the migration

1 Like
10

With which command?

11 
systemctl stop postgresql.service
systemctl stop postgresql@13-main.service
12

The same result, I tried three times obeying your recommendations, checking PGHOST and df.

Here is the last log:

https://paste.yunohost.org/raw/punomuxeva

What seems wrong: the logs are in /var/log/postgresql/pg_upgradecluster-13-15-main.0YZD, which is empty.

I wonder what I am doing wrong…

13

I am trying to undestand the logfile better. I improved it by replacing all \n with \n in gedit.

The first error message is:

Copying
  user relation files                                 
Fehler beim Kopieren von Relation
  \xc2\xbbpublic.state_groups_state_type_idx\xc2\xab: konnte Datei \xc2\xbb/var/lib/postgresql/15/main/base/16401/17779.1\xc2\xab
  nicht schreiben: Auf dem Ger\xc3\xa4t ist kein Speicherplatz mehr verf\xc3\xbcgbar
Fehlgeschlagen,
  Programm wird beendet
Cluster is not running.
''

  stderr:

  b''Error: pg_upgrade run failed. Logfiles are in /var/log/postgresql/pg_upgradecluster-13-15-main.0YZD
Error
  during cluster dumping, removing new cluster
Error: could not stop old cluster,
  please do that manually
'

That means the first error says, it could not write the file \xc2\xbb/var/lib/postgresql/15/main/base/16401/17779.1\xc2\xab because there is no more memory/storage available on the device.

Could not stop the old cluster comes later.

This is all I can do. How can we get more debugging information?

14

Can you open a second terminal and monitor the disk usage using df -h during the migration?

15

I even managed the second ssh window. I did df -h frequently, the output didn’t change seemingly. Here it is:

Filesystem                                        Size  Used Avail Use% Mounted on
udev                                              3.9G     0  3.9G   0% /dev
tmpfs                                             795M  656K  794M   1% /run
/dev/sda3                                         153G  119G   27G  82% /
tmpfs                                             3.9G     0  3.9G   0% /dev/shm
tmpfs                                             5.0M     0  5.0M   0% /run/lock
/dev/sda2                                         974M   84M  823M  10% /boot
ipaddress:/voln353211a1                       250G   73G  178G  30% /mnt/storagespace
ipaddress:/voln353211a1/home/yunohost.backup  250G   73G  178G  30% /home/yunohost.backup
tmpfs                                             795M     0  795M   0% /run/user/1007
ipaddress:/voln353211a2                       250G  384K  250G   1% /mnt/plusdisk

The last line is the additional disk.

16

So, there is plenty of disk space. The error message is wrong.

I need more detailed error information.

The question is, why is there no logging of the postgresql migration process?

17

I’m out of ideas may be @Aleks can help here. Sorry to ping you

18

Anyway, thank you very much for your frequent and quick help!

19

Can someone please help and look into this?
Thank you in advance!

20

Help is needed since my Synapse matrix server has not been running for 3 days. Yunohost upgrade is not yet complete.

So far, postgresql and synapse service are not running. Here is the diagnosis output:

https://paste.yunohost.org/raw/zasapapeyo

@Aleks could you again look into this, please, or someone else?

I would be really grateful if I could repair my Yunohost server so that the upgrade to Yunohost 12 is complete and all services are running again.