'Migration 0023_postgresql_11_to_13 gescheitert mit der Ausnahme Failed to run command ''LC_ALL=C pg_upgradecluster -m upgrade 11 main''

Description of my issue

I’ve done the migration to Debian Bullseye - all ok.
After that I wanted to start the next migration that was displayed. But this one was not executed successfully. Now I can not restart the postgresql.

What can I do?

Here are some error messages

https://paste.yunohost.org/raw/axihavujaw

My YunoHost server

Hardware: VPS bought online
YunoHost version: 11.0.9.8
I have access to my server : Through SSH
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no

1 Like

So apparently postgres-13 is missing some libraries:

Checking for presence of required libraries                 fatalY
our installation references loadable libraries that are missing from the
new installation.  You can add these libraries to the new installation,
or remove the functions using them from the old installation.  A list of
problem libraries is in the file:    loadable_libraries.txt

Can you share the output of : cat /var/log/postgresql/pg_upgradecluster-11-13-main.DW3R ? (hopefully that should contain more info about what lib exactly is missing)

I have this files in the folder:
loadable_libraries.txt
pg_upgrade_dump_16385.log
pg_upgrade_dump_1.log
pg_upgrade_utility.log
pg_upgrade_dump_13101.custom
pg_upgrade_dump_17083.custom
pg_upgrade_dump_globals.sql
pg_upgrade_dump_13101.log
pg_upgrade_dump_17083.log
pg_upgrade_internal.log
pg_upgrade_dump_16385.custom
pg_upgrade_dump_1.custom
pg_upgrade_server.log

Which content do you need?

Ah that’s a directory ?

Then let’s look at loadable_libraries.txt :

sudo cat /var/log/postgresql/pg_upgradecluster-11-13-main.DW3R/loadable_libraries.txt 
could not load library "$libdir/postgis-2.5": ERROR:  could not access file "$libdir/postgis-2.5": No such file or directory
In database: mobilizon
could not load library "$libdir/rtpostgis-2.5": ERROR:  could not access file "$libdir/rtpostgis-2.5": No such file or directory
In database: mobilizon

Hmyeah that’s the famous postgis … I saw somebody having the same issue, I think that was in the main Yunohost 11 release thread ? I’m in the train right now so not super easy to check. I think the person managed to fix it but if i remember correctly it’s not that clear what’s the fix

i think thats it
postgres migration

That’s looks good. Now i have

Ver Cluster Port Status Owner    Data directory              Log file
11  main    5433 down   postgres /var/lib/postgresql/11/main /var/log/postgresql/postgresql-11-main.log
Ver Cluster Port Status Owner    Data directory              Log file
13  main    5432 online postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log

But I can not restart

sudo pg_ctlcluster 13 main restart
Job for postgresql@13-main.service failed because the service did not take the steps required by its unit configuration.
See "systemctl status postgresql@13-main.service" and "journalctl -xe" for details.
systemctl status postgresql@13-main.service
Aug 23 09:54:51 systemd[1]: Starting PostgreSQL Cluster 13-main...
Aug 23 09:54:51 postgresql@13-main[28532]: Cluster is already running.
Aug 23 09:54:51 systemd[1]: postgresql@13-main.service: Can't open PID file /run/postgresql/13-main.pid (yet?) after start: Operation not permitted
Aug 23 09:54:51 systemd[1]: postgresql@13-main.service: Failed with result 'protocol'.
Aug 23 09:54:51 systemd[1]: Failed to start PostgreSQL Cluster 13-main.
journalctl -xe
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit user@0.service has entered the 'failed' state with result 'exit-code'.
Aug 23 12:05:12 aaaa.org systemd[1]: Failed to start User Manager for UID 0.
░░ Subject: A start job for unit user@0.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit user@0.service has finished with a failure.
░░ 
░░ The job identifier is 762063 and the job result is failed.
Aug 23 12:05:12 aaaa.org systemd[1]: Started Session 699346 of user root.
░░ Subject: A start job for unit session-699346.scope has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit session-699346.scope has finished successfully.
░░ 
░░ The job identifier is 761999.
Aug 23 12:05:13 aaaa.org mobilizon[5543]: 12:05:13.450 [error] GenServer {Oban.Registry, {Oban, {:plugin, Oban.Plugins.Stager}}} terminating
Aug 23 12:05:13 aaaa.org mobilizon[5543]: ** (Postgrex.Error) ERROR XX000 (internal_error) cache lookup failed for type 19604
Aug 23 12:05:13 aaaa.org mobilizon[5543]:     (ecto_sql 3.7.1) lib/ecto/adapters/sql.ex:760: Ecto.Adapters.SQL.raise_sql_call_error/1
Aug 23 12:05:13 aaaa.org mobilizon[5543]:     (ecto_sql 3.7.1) lib/ecto/adapters/sql.ex:693: Ecto.Adapters.SQL.execute/5
Aug 23 12:05:13 aaaa.org mobilizon[5543]:     (oban 2.10.1) lib/oban/plugins/stager.ex:98: anonymous fn/1 in Oban.Plugins.Stager.lock_and_stage/1
Aug 23 12:05:13 aaaa.org mobilizon[5543]:     (ecto_sql 3.7.1) lib/ecto/adapters/sql.ex:1021: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
Aug 23 12:05:13 aaaa.org mobilizon[5543]:     (db_connection 2.4.1) lib/db_connection.ex:1531: DBConnection.run_transaction/4
Aug 23 12:05:13 aaaa.org mobilizon[5543]:     (oban 2.10.1) lib/oban/plugins/stager.ex:81: anonymous fn/2 in Oban.Plugins.Stager.handle_info/2
Aug 23 12:05:13 aaaa.org mobilizon[5543]:     (telemetry 1.0.0) /builds/framasoft/mobilizon/deps/telemetry/src/telemetry.erl:293: :telemetry.span/3
Aug 23 12:05:13 aaaa.org mobilizon[5543]:     (oban 2.10.1) lib/oban/plugins/stager.ex:80: Oban.Plugins.Stager.handle_info/2
Aug 23 12:05:13 aaaa.org mobilizon[5543]: Last message: :stage
Aug 23 12:05:14 aaaa.org mobilizon[5543]: 12:05:14.457 [error] GenServer {Oban.Registry, {Oban, {:plugin, Oban.Plugins.Stager}}} terminating
Aug 23 12:05:14 aaaa.org mobilizon[5543]: ** (Postgrex.Error) ERROR XX000 (internal_error) cache lookup failed for type 19604
Aug 23 12:05:14 aaaa.org mobilizon[5543]:     (ecto_sql 3.7.1) lib/ecto/adapters/sql.ex:760: Ecto.Adapters.SQL.raise_sql_call_error/1
Aug 23 12:05:14 aaaa.org mobilizon[5543]:     (ecto_sql 3.7.1) lib/ecto/adapters/sql.ex:693: Ecto.Adapters.SQL.execute/5
Aug 23 12:05:14 aaaa.org mobilizon[5543]:     (oban 2.10.1) lib/oban/plugins/stager.ex:98: anonymous fn/1 in Oban.Plugins.Stager.lock_and_stage/1
Aug 23 12:05:14 aaaa.org mobilizon[5543]:     (ecto_sql 3.7.1) lib/ecto/adapters/sql.ex:1021: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
Aug 23 12:05:14 aaaa.org mobilizon[5543]:     (db_connection 2.4.1) lib/db_connection.ex:1531: DBConnection.run_transaction/4
Aug 23 12:05:14 aaaa.org mobilizon[5543]:     (oban 2.10.1) lib/oban/plugins/stager.ex:81: anonymous fn/2 in Oban.Plugins.Stager.handle_info/2
Aug 23 12:05:14 aaaa.org mobilizon[5543]:     (telemetry 1.0.0) /builds/framasoft/mobilizon/deps/telemetry/src/telemetry.erl:293: :telemetry.span/3
Aug 23 12:05:14 aaaa.org mobilizon[5543]:     (oban 2.10.1) lib/oban/plugins/stager.ex:80: Oban.Plugins.Stager.handle_info/2
Aug 23 12:05:14 aaaa.org mobilizon[5543]: Last message: :stage

Is it just a problem with mobilzon? Would it help to uninstall mobilzon?

I am also struggling with postgresql (and consequently, all the apps that use it). I followed @emo’s advice, which was sudo -u postgres pg_upgradecluster 11 main followed by pg_ctlcluster 13 main restart. The results are

pg_ctlcluster 13 main restart                                                                                                                                      
Job for postgresql@13-main.service failed because the service did not take the steps required by its unit configuration.                                                                       
See "systemctl status postgresql@13-main.service" and "journalctl -xe" for details.                                                                                                            
root@domain:/home/user# systemctl status postgresql@13-main.service                                                                                                                        
● postgresql@13-main.service - PostgreSQL Cluster 13-main
     Loaded: loaded (/lib/systemd/system/postgresql@.service; disabled; vendor preset: enabled)                                                                                                
     Active: failed (Result: protocol) since Tue 2022-08-23 12:09:57 CEST; 30s ago
    Process: 2149497 ExecStart=/usr/bin/pg_ctlcluster --skip-systemctl-redirect 13-main start (code=exited, status=2)
        CPU: 112ms                                                                                                                                                                             
                                                                                                                                                                                               
aug. 23 12:09:57 domain systemd[1]: Starting PostgreSQL Cluster 13-main...                                                                                                                
aug. 23 12:09:57 domain postgresql@13-main[2149497]: Cluster is already running.                                                                                                          
aug. 23 12:09:57 domain systemd[1]: postgresql@13-main.service: New main PID 2140103 does not belong to service, and PID file is not owned by root. Refusing.
aug. 23 12:09:57 domain systemd[1]: postgresql@13-main.service: New main PID 2140103 does not belong to service, and PID file is not owned by root. Refusing.
aug. 23 12:09:57 domain systemd[1]: postgresql@13-main.service: Failed with result 'protocol'.
aug. 23 12:09:57 domain systemd[1]: Failed to start PostgreSQL Cluster 13-main.

At least now I know how to drop the new postgresql 13 cluster, so I can get back to the previous (non-functional) state…

Hello, I have also problems with Mobilizon…
finally I remove the app, remove all postgresql and restore mobilizon Comment restaurer ancienne database de Mobilizon? - #3 by rodinux
Not sure it is the solution, as I try other things before… but it is perhaps the solution…

I have now removed Mobilizon, then repeated all the steps of postgres migration .

After that the restart still had no success, I think it has something to do with the PID file.

Aug 23 09:54:51 systemd[1]: postgresql@13-main.service: Can't open PID file /run/postgresql/13-main.pid (yet?) after start: Operation not permitted

After I restarted the server completely, the postgres seems to run correctly now.

However, now wiki.js does not work anymore. I skipped the migration in the yunohost-admin. Are there any other steps that need to be taken?

I think you need complete the migration, have you try repeat sudo yunohost tools migrations run --accept-disclaimer

I get Info: No migrations to run
Unfortunately, I had skipped the migration in yunohost-admin. How can I restart it?

Should be something like : sudo yunohost tools migrations run 0021_migrate_to_bullseye --force-rerun

Ok, thanx. I do sudo yunohost tools migrations run 0023_postgresql_11_to_13 --force-rerun

But it fails:

https://paste.yunohost.org/raw/enijamefew

Zblerg:

Checking for presence of required libraries                 fatal

Your installation references loadable libraries that are missing from the
new installation.  You can add these libraries to the new installation,
or remove the functions using them from the old installation.  A list of
problem libraries is in the file:
    loadable_libraries.txt

Failure, exiting\nCluster is not running.\n'
stderr:
b'Error: pg_upgrade run failed. Logfiles are in /var/log/postgresql/pg_upgradecluster-11-13-main.

So let’s check what libs are missing (I’m suspecting postgis, maybe cf the issue with Mobilizon ?) :

This should be something like :

cat /var/log/postgresql/pg_upgradecluster-11-13-main/loadable_libraries.txt

There is No such file or directory. And Mobilzon is already deleted in the meantime.

Since the environment was not yet productive, maybe I should delete postgres and wiki.js and reinstall again?

I still have
Etherpad
Gitta
HumHub
OpenSondage
on the server. They shouldn’t be affected by this, right?

hi, imho the problem is in pg version 11 some immutable files cant be deleted or overwritten by root. owner of these files is user postgres, so migration went fail for some users.
you can follow the documentation PostgreSQL: Documentation: 13: 18.6. Upgrading a PostgreSQL Cluster
and take care to remove all old stuff. be aware, this one is a gun, make a backup before!

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.