[Solved] Synapse upgrade fails, restore fails (setuptools 60 vs 68? psycopg2?)

[wbk]

Hi all!

My YunoHost server

Hardware: LXC on small box with Proxmox
YunoHost version:
Server is running YunoHost 11.2.5 (stable)
yunohost version: 11.2.5 (stable)
yunohost-admin version: 11.2.3 (stable)
moulinette version: 11.2 (stable)
ssowat version: 11.2 (stable)
I have access to my server : Through SSH and through the webadmin, if necessary direct access via keyboard / screen
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no
If your request is related to an app, specify its name and version: Synapse 1.91.0~ynh1 → 1.92.3~ynh1

Description of my issue

I ran an upgrade of my apps (screen → yunohost app upgrade).

Synapse failed to upgrade. To add insult to injury, the restore of the pre-upgrade version failed as well.

Here is an excerpt of the full log where the error occurs:

2023-10-12 23:26:27,900: DEBUG - + pip3 install --upgrade setuptools==60.8.2 wheel pip
2023-10-12 23:26:29,541: DEBUG - Collecting setuptools==60.8.2
2023-10-12 23:26:29,563: DEBUG -   Using cached setuptools-60.8.2-py3-none-any.whl (1.1 MB)
2023-10-12 23:26:29,578: DEBUG - Requirement already satisfied: wheel in /opt/yunohost/matrix-synapse/lib/python3.9/site-packages (0.41.2)
2023-10-12 23:26:29,848: DEBUG - Requirement already satisfied: pip in /opt/yunohost/matrix-synapse/lib/python3.9/site-packages (23.2.1)
2023-10-12 23:26:31,532: DEBUG - Installing collected packages: setuptools
2023-10-12 23:26:31,533: DEBUG -   Attempting uninstall: setuptools
2023-10-12 23:26:31,534: DEBUG -     Found existing installation: setuptools 68.1.2
2023-10-12 23:26:31,593: DEBUG -     Uninstalling setuptools-68.1.2:
2023-10-12 23:26:31,708: DEBUG -       Successfully uninstalled setuptools-68.1.2
2023-10-12 23:26:33,905: WARNING - ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2023-10-12 23:26:33,905: WARNING - setuptools-rust 1.7.0 requires setuptools>=62.4, but you have setuptools 60.8.2 which is incompatible.
2023-10-12 23:26:33,906: DEBUG - Successfully installed setuptools-60.8.2
2023-10-12 23:26:34,349: DEBUG - + pip3 install --upgrade cffi ndg-httpsclient psycopg2 lxml jinja2
2023-10-12 23:26:35,033: DEBUG - Requirement already satisfied: cffi in /opt/yunohost/matrix-synapse/lib/python3.9/site-packages (1.15.1)
2023-10-12 23:26:36,112: DEBUG - Collecting cffi
2023-10-12 23:26:36,113: DEBUG -   Obtaining dependency information for cffi from https://files.pythonhosted.org/packages/ea/ac/e9e77bc385729035143e54cc8c4785bd480eaca9df17565963556b0b7a93/cffi-1.16.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
2023-10-12 23:26:36,200: DEBUG -   Downloading cffi-1.16.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
2023-10-12 23:26:36,378: DEBUG - Requirement already satisfied: ndg-httpsclient in /opt/yunohost/matrix-synapse/lib/python3.9/site-packages (0.5.1)
2023-10-12 23:26:36,577: DEBUG - Requirement already satisfied: psycopg2 in /opt/yunohost/matrix-synapse/lib/python3.9/site-packages (2.9.7)
2023-10-12 23:26:36,946: DEBUG - Collecting psycopg2
2023-10-12 23:26:36,965: DEBUG -   Downloading psycopg2-2.9.9.tar.gz (384 kB)
2023-10-12 23:26:37,222: DEBUG -      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 384.9/384.9 kB 1.5 MB/s eta 0:00:00
2023-10-12 23:26:37,320: DEBUG -   Preparing metadata (setup.py): started
2023-10-12 23:26:38,195: DEBUG -   Preparing metadata (setup.py): finished with status 'error'
2023-10-12 23:26:38,209: WARNING -   error: subprocess-exited-with-error
2023-10-12 23:26:38,209: DEBUG - 
2023-10-12 23:26:38,209: WARNING -   × python setup.py egg_info did not run successfully.
2023-10-12 23:26:38,209: WARNING -   │ exit code: 1
2023-10-12 23:26:38,210: WARNING -   ╰─> [24 lines of output]
2023-10-12 23:26:38,210: WARNING -       Traceback (most recent call last):
2023-10-12 23:26:38,210: WARNING -         File "<string>", line 2, in <module>
2023-10-12 23:26:38,210: WARNING -         File "<pip-setuptools-caller>", line 34, in <module>
2023-10-12 23:26:38,211: WARNING -         File "/tmp/pip-install-sxu8id8x/psycopg2_a46233964eeb471897fd544fbc90103c/setup.py", line 545, in <module>
2023-10-12 23:26:38,211: WARNING -           setup(name="psycopg2",

Just around this time @Thetrveanthony got a similar, not yet solved, problem. I’ll try @Catslover71’s suggestion regarding if/else for setuptools in bullseye and post back.

If it doesn’t work, I’ll try an install of Synapse from scratch: my wife is on the other side of the globe for a while, I suggested Element as a modern means of communication … So getting up and running in the short while is more urgent for now than getting to the bottom (which I hope to get to before too long as well).

[wbk]

Solved!

The solution was indeed removing the if/else that @CatsLover71 mentioned!

So, for the next victim I’ll copy the commands I used:

cd /home/yunohost.backup/archives/
tar -czf synapse-pre-upgrade2.tar.gz.org synapse-pre-upgrade2.tar
tar tvf synapse-pre-upgrade2.tar |grep common
tar xf synapse-pre-upgrade2.tar apps/synapse/settings/scripts/_common.sh
vi apps/synapse/settings/scripts/_common.sh 
tar uvf synapse-pre-upgrade2.tar apps/synapse/settings/scripts/_common.sh
rm -rf apps/
yunohost backup restore synapse-pre-upgrade2

There was some trialing and erroring, so I redacted the commands for clarity’s sake but did not re-run to test them afterwards.
The long story, based on the commands above, is:

• go to your archives
• make a (compressed) backup of the backup that you might damage while experimenting
• search the Synapse-backup for the common.sh file
• extract the file
• edit to reflect catslovers71’s suggestion
• update the tarball with the changed common.sh file
• tidy: remove the directory that was created when extracting the file
• restore the backup

My actual commands differed a bit, also because I first tried if Synapse still was installable at all on my system:

yunohost app remove synapse
cd /home/yunohost.app/matrix-synapse/
mv matrix-synapse/ matrix-synapse/tmp
cp -r matrix-synapse20231013.223321/ matrix-synapse/

I was smart enough to remove the temporary installation of Synapse, but restore didn’t work: it complained about the app-dir. I removed the app dir, and renamed th ereserve copy that was created on installation (it renamed the existing directory).

Another thing that broke, was the link between the yaml-file and the actual backup. At first I compressed synapse-pre-upgrade2.tar to synapse-pre-upgrade2.tar.gz, which seemed a good idea at that point. I think there is some logic that tries to first find backupfilename.tar.gz with a corresponding backupfilename.yaml, and only if it not exists, tries to find backupfilename.tar. So when I compressed to synapse-pre-upgrade2.tar.gz, that file was found by yunohost backup restore synapse , but no corresponding yaml, ending in error.

Nearing the end of this post, I got this nagging feeling I forgot something. I actually did: the restore is done, but upgrade not yet!

To be continued…

[wbk]

Wohoo! Synapse got through the upgrade