['solved'] Synapse - Failed at step NAMESPACE spawning - permission denied

[wbk]

My YunoHost server

Hardware: laptop or computer at home
YunoHost version:
Server hardware architecture is lxc amd64
Server is running Linux kernel 5.11.22-5-pve
Server is running Debian 11.6
Server is running YunoHost 11.0.10.2 (stable)
I have access to my server : Through SSH / through the webadmin / direct access via keyboard / screen
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : yes, possibly related: the system runs in a container on LXC

Description of my issue

Hi all,

My Synapse is down and won’t get up anymore.

• It just got restored from backup this afternoon
• Then I ran the migration from YNH 4.x to YNH 11, which ran quite well
• Then I ran the upgrades for the apps that could not upgrade anymore in YNH 4.x

Many went right through, but Synapse did not.

• The app upgrade got rolled back because the new version would exit with an error through
  systemd (log)
• Automatic restore of the backup ended with the same error, leaving the server without Synapse installed (in the same log)
• Manual restore of the backup ended in the same error (new log)

Between the automatic restore and the manual restore, I tried to find similar messages on the 'Net. It could be related to the server running in a container on LXC.
People opened a bug report against systemd, there a workaround was mentioned in which nested containers are enabled.
I enabled nesting as suggested, but it did not help me: the manual restore also ended in an error.

Relevant sections of the log print:

Warning: Jan 05 16:45:13 systemd[1]: Starting Synapse Matrix homeserver...
Warning: Jan 05 16:45:13 systemd[102511]: matrix-synapse.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied
Warning: Jan 05 16:45:13 systemd[102511]: matrix-synapse.service: Failed at step NAMESPACE spawning /opt/yunohost/matrix-synapse/bin/python: Permission denied
Warning: Jan 05 16:45:13 systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=226/NAMESPACE
Warning: Jan 05 16:45:13 systemd[1]: matrix-synapse.service: Failed with result 'exit-code'.
Warning: Jan 05 16:45:13 systemd[1]: Failed to start Synapse Matrix homeserver.

and

Warning: Jan 05 18:28:14 systemd[1]: Starting Synapse Matrix homeserver...
Warning: Jan 05 18:28:14 systemd[153710]: matrix-synapse.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied
Warning: Jan 05 18:28:14 systemd[153710]: matrix-synapse.service: Failed at step NAMESPACE spawning /opt/yunohost/matrix-synapse/bin/python: Permission denied
Warning: Jan 05 18:28:14 systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=226/NAMESPACE
Warning: Jan 05 18:28:14 systemd[1]: matrix-synapse.service: Failed with result 'exit-code'.
Warning: Jan 05 18:28:14 systemd[1]: Failed to start Synapse Matrix homeserver.
Error: Could not restore synapse: An error occured inside the app restore script

I am not quite sure where to point: I think Matrix was still running after the migration to Bullseye, but there was no reason to try restarting Synapse prior to upgrading it. Then after the failed upgrade, the backup gave the same error.

Any suggestions where to continue debugging?

[wbk]

Installing a new Matrix-Synapse fails with the same error. Some reports on the net point at #ReadOnlyDirectories = / in the systemd unit file.

To help in troubleshooting, is there a way to prevent rolling back the a failed restoration of a backup?

Now Yunohost tries to start Synapse after restoring, sees it failing, and rolls back the restore of the backup. That’s very clean, but leaves me with nothing to troubleshoot

By the way, since the migration it takes 10-20 seconds to login via SSH, which was instantaneous before. Anyone with such symptoms?

[wbk]

Hu? I just, for fun, tried installing a new instance again and it did not fail !

Changes since yesterday? Upgrading Nextcloud…

Restoring ‘my own’ Synapse over the blanco Synapse does not work of course, so uninstalling and yunohost backup restore synapse-pre-upgrade1 --app

Guess what? It says ‘Restoration completed’

Next up was yunohost app upgrade synapse , which also ran flawless.

The only reason for success I can think of, is that the upgrade of Nextcloud pulled in some dependencies or configuration changes. It is not the most satisfying resolution of the problem, but I’m happy all is well now!