Synapse update fails, impossible to restore

Support
,
1

My YunoHost server

Hardware: RaspberryPi 4
YunoHost version: 11.2.5
I have access to my server : Through SSH | through the webadmin
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no

Description of my issue

logs

En

Hi,

When I updated my system today, the synapse (v1.79) update failed again (because of setuptools, I tried all the solutions I could find on the forum or github, but none worked for me), and also this time, backing up fails as well, the system complains about a file named /opt/yunohost/....../METADATA and cannot install packages because of it.

I found a couple topics with the same issue except they did not have the same error as me, so I prefer asking first before randomly trying their solutions, even if they worked.
How can I fix that ? I thought a touch would do the trick but /opt/yunohost is empty …

Fr

Salut à tous,

En faisant ma mise à jour, j’ai à nouveau eu une erreur quand vint le tour de synapse (à cause de setuptools, j’ai essayé toutes les solutions que j’ai pu trouver sur le forum ou sur github mais rien n’y fait), et cette fois la restauration a aussi échoué, à cause d’un fichier manquant, /opt/yunohost...../METADATA, ce qui empêcherait l’installation de paquets.
Je suis tombé sur deux topics sur le même problème, mais avec un message d’erreur différent, du coup je préfère ne pas me hasarder à tenter leurs solutions, même si elles ont marché.

Merci / Thanks !

2

Hello,
You have had the now famous setuptools issue with synapse.

Maybe, you can try to modify this file inside the backup archive of synapse.

Modify this file by keeping only the else part :
pip3 install --upgrade setuptools wheel pip
And then try again to restore using this modified file inside the archive.
Hope this gonna solve your issue.

3

Thanks for replying,

I tried as you suggested (and realised I need to learn more about tar), unfortunately it still fails to restore, I got the same error : ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/opt/yunohost/matrix-synapse/lib/python3.9/site-packages/cffi-1.15.1.dist-info/METADATA' ,
should I try a touch opt/yunohost/matrix-synapse/lib/python3.9/site-packages/cffi-1.15.1.dist-info/METADATA ?

4

Did the error with setuptool disappeared, at least ?
I don’t know but why not trying the touch of METADATA
Just in case I give you the content of this file on my server :


/opt/yunohost/matrix-synapse/lib/python3.9/site-packages/cffi-1.15.1.dist-info # cat METADATA 
Metadata-Version: 2.1
Name: cffi
Version: 1.15.1
Summary: Foreign Function Interface for Python calling C code.
Home-page: http://cffi.readthedocs.org
Author: Armin Rigo, Maciej Fijalkowski
Author-email: python-cffi@googlegroups.com
License: MIT
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: License :: OSI Approved :: MIT License
License-File: LICENSE
Requires-Dist: pycparser


CFFI
====

Foreign Function Interface for Python calling C code.
Please see the `Documentation <http://cffi.readthedocs.org/>`_.

Contact
-------

`Mailing list <https://groups.google.com/forum/#!forum/python-cffi>`_

There are some other files in this folder :

/opt/yunohost/matrix-synapse/lib/python3.9/site-packages/cffi-1.15.1.dist-info # ls -al
total 44
drwxr-xr-x   2 matrix-synapse root  4096 Sep 27  2022 .
drwxrwxrwx 138 matrix-synapse root 12288 Sep 21 18:06 ..
-rw-r--r--   1 matrix-synapse root    75 Sep 27  2022 entry_points.txt
-rw-r--r--   1 matrix-synapse root     4 Sep 27  2022 INSTALLER
-rw-r--r--   1 matrix-synapse root  1294 Sep 27  2022 LICENSE
-rw-r--r--   1 matrix-synapse root  1144 Sep 27  2022 METADATA
-rw-r--r--   1 matrix-synapse root  2992 Sep 27  2022 RECORD
-rw-r--r--   1 matrix-synapse root    19 Sep 27  2022 top_level.txt
-rw-r--r--   1 matrix-synapse root   150 Sep 27  2022 WHEEL

/opt/yunohost/matrix-synapse/lib/python3.9/site-packages/cffi-1.15.1.dist-info # cat entry_points.txt 
[distutils.setup_keywords]
cffi_modules = cffi.setuptools_ext:cffi_modules

/opt/yunohost/matrix-synapse/lib/python3.9/site-packages/cffi-1.15.1.dist-info # cat INSTALLER 
pip

/opt/yunohost/matrix-synapse/lib/python3.9/site-packages/cffi-1.15.1.dist-info # cat RECORD 
_cffi_backend.cpython-39-aarch64-linux-gnu.so,sha256=mYh4Dy89jojXJX-ab6fqsIjPaYOXIpOKMDkU4DGdzBE,1052976
cffi-1.15.1.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
cffi-1.15.1.dist-info/LICENSE,sha256=BLgPWwd7vtaICM_rreteNSPyqMmpZJXFh72W3x6sKjM,1294
cffi-1.15.1.dist-info/METADATA,sha256=KP4G3WmavRgDGwD2b8Y_eDsM1YeV6ckcG6Alz3-D8VY,1144
cffi-1.15.1.dist-info/RECORD,,
cffi-1.15.1.dist-info/WHEEL,sha256=EFW0IeA9Np1ksorViC508uzcNuLwTraoZLYyYhhpIFI,150
cffi-1.15.1.dist-info/entry_points.txt,sha256=y6jTxnyeuLnL-XJcDv8uML3n6wyYiGRg8MTp_QGJ9Ho,75
cffi-1.15.1.dist-info/top_level.txt,sha256=rE7WR3rZfNKxWI9-jn6hsHCAl7MDkB-FmuQbxWjFehQ,19
cffi.libs/libffi-2a6f5b63.so.8.1.0,sha256=uFcmlDsVhvOzDnZnzRz8x20BiBv18IwLJOzwwnc9-Ko,198248
cffi/__init__.py,sha256=6xB_tafGvhhM5Xvj0Ova3oPC2SEhVlLTEObVLnazeiM,513
cffi/__pycache__/__init__.cpython-39.pyc,,
cffi/__pycache__/api.cpython-39.pyc,,
cffi/__pycache__/backend_ctypes.cpython-39.pyc,,
cffi/__pycache__/cffi_opcode.cpython-39.pyc,,
cffi/__pycache__/commontypes.cpython-39.pyc,,
cffi/__pycache__/cparser.cpython-39.pyc,,
cffi/__pycache__/error.cpython-39.pyc,,
cffi/__pycache__/ffiplatform.cpython-39.pyc,,
cffi/__pycache__/lock.cpython-39.pyc,,
cffi/__pycache__/model.cpython-39.pyc,,
cffi/__pycache__/pkgconfig.cpython-39.pyc,,
cffi/__pycache__/recompiler.cpython-39.pyc,,
cffi/__pycache__/setuptools_ext.cpython-39.pyc,,
cffi/__pycache__/vengine_cpy.cpython-39.pyc,,
cffi/__pycache__/vengine_gen.cpython-39.pyc,,
cffi/__pycache__/verifier.cpython-39.pyc,,
cffi/_cffi_errors.h,sha256=zQXt7uR_m8gUW-fI2hJg0KoSkJFwXv8RGUkEDZ177dQ,3908
cffi/_cffi_include.h,sha256=tKnA1rdSoPHp23FnDL1mDGwFo-Uj6fXfA6vA6kcoEUc,14800
cffi/_embedding.h,sha256=9tnjF44QRobR8z0FGqAmAZY-wMSBOae1SUPqHccowqc,17680
cffi/api.py,sha256=yxJalIePbr1mz_WxAHokSwyP5CVYde44m-nolHnbJNo,42064
cffi/backend_ctypes.py,sha256=h5ZIzLc6BFVXnGyc9xPqZWUS7qGy7yFSDqXe68Sa8z4,42454
cffi/cffi_opcode.py,sha256=v9RdD_ovA8rCtqsC95Ivki5V667rAOhGgs3fb2q9xpM,5724
cffi/commontypes.py,sha256=QS4uxCDI7JhtTyjh1hlnCA-gynmaszWxJaRRLGkJa1A,2689
cffi/cparser.py,sha256=rO_1pELRw1gI1DE1m4gi2ik5JMfpxouAACLXpRPlVEA,44231
cffi/error.py,sha256=v6xTiS4U0kvDcy4h_BDRo5v39ZQuj-IMRYLv5ETddZs,877
cffi/ffiplatform.py,sha256=HMXqR8ks2wtdsNxGaWpQ_PyqIvtiuos_vf1qKCy-cwg,4046
cffi/lock.py,sha256=l9TTdwMIMpi6jDkJGnQgE9cvTIR7CAntIJr8EGHt3pY,747
cffi/model.py,sha256=_GH_UF1Rn9vC4AvmgJm6qj7RUXXG3eqKPc8bPxxyBKE,21768
cffi/parse_c_type.h,sha256=OdwQfwM9ktq6vlCB43exFQmxDBtj2MBNdK8LYl15tjw,5976
cffi/pkgconfig.py,sha256=LP1w7vmWvmKwyqLaU1Z243FOWGNQMrgMUZrvgFuOlco,4374
cffi/recompiler.py,sha256=YgVYTh2CrXIobo-vMk7_K9mwAXdd_LqB4-IbYABQ488,64598
cffi/setuptools_ext.py,sha256=RUR17N5f8gpiQBBlXL34P9FtOu1mhHIaAf3WJlg5S4I,8931
cffi/vengine_cpy.py,sha256=YglN8YS-UaHEv2k2cxgotNWE87dHX20-68EyKoiKUYA,43320
cffi/vengine_gen.py,sha256=5dX7s1DU6pTBOMI6oTVn_8Bnmru_lj932B6b4v29Hlg,26684
cffi/verifier.py,sha256=ESwuXWXtXrKEagCKveLRDjFzLNCyaKdqAgAlKREcyhY,11253

/opt/yunohost/matrix-synapse/lib/python3.9/site-packages/cffi-1.15.1.dist-info # cat top_level.txt 
_cffi_backend
cffi

This one look specific to the architecture (here ARM64)

/opt/yunohost/matrix-synapse/lib/python3.9/site-packages/cffi-1.15.1.dist-info # cat WHEEL 
Wheel-Version: 1.0
Generator: bdist_wheel (0.37.1)
Root-Is-Purelib: false
Tag: cp39-cp39-manylinux_2_17_aarch64
Tag: cp39-cp39-manylinux2014_aarch64
5

The setuptools error is gone, indeed

but my /opt/yunohost folder is empty :

admin@purplehaze:~ $ ls -a /opt/yunohost/ 
.  ..

So I assume the last update deleted the matrix-synapse folder for whatever reason, I my only option is to do a full clean install of synapse ? (Your file contents mus be specific to your server I guess)

6

Well, the .tar file contains the /opt/yunohost/matrix… folder. So I’d try a restauration of this folder manually and then try again a full restore.

7

Still doesnt work ><, here is what I did:

tar -xvf synapse-pre-upgrade1.tar apps/synapse/backup/opt/yunohost/matrix-synapse
cp -r apps/synapse/backup/opt/yunohost/matrix-synapse /opt/yunohost
## then restore

returned There is already a directory: /opt/yunohost/matrix-synapse, so the system complains whether the dir is here or not.

Do you think I can delete the test command that returns that error and retry ? It just does a few chmods after this : lines 62 and 220

8

Well, lost for lost, why not.
But I feel a bit shamy, cause, what I suggested before was a no luck to work, seeing the result.

9

Still doesn’t work, it complains again about metadata not being here although I manually restored it. I’ll just reinstall synapse from scratch, many thanks for your help

10

Hi Anthony,

I read your story. I got the same problem today, but in my case the suggestion by Catlover worked.

Do you still have the original backup file? You could try once more. I documented the steps to update the single file in the backup. It looks like many steps, but it does not take a lot of time.

11

Hi,

I already reinstalled synapse, but I still have the original backup archive indeed, could you show how you did it then ?

12

My bad, sorry! I intended to link to it, but my thinking was less clear at the end of the night. Please have a look at [Solved] Synapse upgrade fails, restore fails (setuptools 60 vs 68? psycopg2?) - #2 by wbk ; keep in mind that you’ll have to replace your own file names of course :wink:

Give it a try!

13

So, I gave it a try, and it failed, I had that metadata error again, which was to be expected, actually I took a look at your logs and it seems you didn’t have that one.
I’ll just reinstall it from scratch, I only have a few users so it’s no big deal, thanks anyway !

14

Hi Anthony,

Sorry to not be of any help.

I actually had a look in the logs you posted before replying, I thought I could not find the metadata error in your logs either. I just had a look again, it seems we both have a metadata error, but not the same metadata error :wink:

On my Yunohost there was an error with metadata, WARNING - error: metadata-generation-failed :


2023-10-12 23:26:35,033: DEBUG - Requirement already satisfied: cffi in /opt/yunohost/matrix-synapse/lib/python3.9/site-packages (1.15.1)
2023-10-12 23:26:36,112: DEBUG - Collecting cffi
2023-10-12 23:26:36,113: DEBUG -   Obtaining dependency information for cffi from https://files.pythonhosted.org/packages/ea/ac/e9e77bc385729035143e54cc8c4785bd480eaca9df17565963556b0b7a93/cffi-1.16.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
2023-10-12 23:26:36,200: DEBUG -   Downloading cffi-1.16.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
2023-10-12 23:26:36,378: DEBUG - Requirement already satisfied: ndg-httpsclient in /opt/yunohost/matrix-synapse/lib/python3.9/site-packages (0.5.1)
2023-10-12 23:26:36,577: DEBUG - Requirement already satisfied: psycopg2 in /opt/yunohost/matrix-synapse/lib/python3.9/site-packages (2.9.7)
2023-10-12 23:26:36,946: DEBUG - Collecting psycopg2
2023-10-12 23:26:36,965: DEBUG -   Downloading psycopg2-2.9.9.tar.gz (384 kB)
2023-10-12 23:26:37,222: DEBUG -      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 384.9/384.9 kB 1.5 MB/s eta 0:00:00
2023-10-12 23:26:37,320: DEBUG -   Preparing metadata (setup.py): started
2023-10-12 23:26:38,195: DEBUG -   Preparing metadata (setup.py): finished with status 'error'
2023-10-12 23:26:38,209: WARNING -   error: subprocess-exited-with-error
2023-10-12 23:26:38,209: DEBUG - 
2023-10-12 23:26:38,209: WARNING -   × python setup.py egg_info did not run successfully.
2023-10-12 23:26:38,209: WARNING -   │ exit code: 1
2023-10-12 23:26:38,210: WARNING -   ╰─> [24 lines of output]
...
2023-10-12 23:26:38,216: WARNING -       [end of output]
2023-10-12 23:26:38,216: DEBUG - 
2023-10-12 23:26:38,216: WARNING -   note: This error originates from a subprocess, and is likely not a problem with pip.
2023-10-12 23:26:38,217: WARNING - error: metadata-generation-failed
2023-10-12 23:26:38,217: DEBUG - 
2023-10-12 23:26:38,217: WARNING - × Encountered error while generating package metadata.
2023-10-12 23:26:38,217: WARNING - ╰─> See above for output.
2023-10-12 23:26:38,217: DEBUG -

It’s a pity when that is necessary, but it is a solution as well. Good luck with it! :slight_smile:

15

Hello,
I have got the same problem on Raspberry Pi 4 : Synapse upgrade from 1.91.2 to 1.92.3 failed and I can’t restore 1.91.2.
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: ‘/opt/yunohost/matrix-synapse/lib/python3.9/site-packages/cffi-1.15.1.dist-info/METADATA’
Did anybody find way to fix this issue ?
Thank you for your help.

16

Strange, I searched for cffi-1.15.1.dist-info, and got a result that does not mention cffi, but it does concern METADATA.

People seem to just mix and match metadata,

My error mentioned metadata as well, but it was another case than yours. I can’t test the suggestions in the Stackoverflow-thread, but maybe it got something that helps you on your way.

17

Thank you for your feed back.
I performed some test

  • Creation of METADATA empty file : Restore script failed (directory exist and script can’t create it)
  • Restoration thru SSH : Same error
  • I tried to fix it with solution in Github Issue 414 : Same error
  • I tried to restore an old synapse backup (September 28) : Same error
  • I installed an brand new yunohost on another hard drive and try to restore September 28 complete backup. All application restoration are OK except synapse (https://paste.yunohost.org/raw/wasiwudaru). There is only warning :
    2023-10-18 20:17:12,059: WARNING - …/settings/scripts/_common.sh: /opt/yunohost/matrix-synapse/bin/pip3: /opt/yunohost/matrix-synapse/bin/python3: bad interpreter: No such file or directory
    Does anybody can give me some advices ?
18

You have been quite busy!

I’m out of suggestions. Just one left: does a new instance of Synapse install without problems? Perhaps you can compare the log of installing a new instance with the restore of the backup and find out which file is needed.

19

Both logs are very different. It is difficult to compare them. (Failed restore https://paste.yunohost.org/raw/fixidohive, new Synapse installation https://paste.yunohost.org/raw/fuwayemume).

20

:frowning: Yes, quite so.

There is a whole block almost identical; I put the logs in two windows beside each other and searched for synapse-ynh-deps; it occurs in the middle 30% of the restore log which corresponds with the area 20% to 40% (the second fifth) of the install log.

After that it diverts, restoring on the one hand and creating a database on the other (search for Adding user turnserver to group ssl-cert on both sides).

Then search for /home/yunohost.backup/tmp/ on the ‘restore’ side, which runs almost till the bottom. The two files convert on chown matrix-synapse:root -R /opt/yunohost/matrix-synapse, which is at 2023-10-19 19:53:03,174 on the restore side and at 2023-10-19 19:59:50,234 on the install side (and once more, at the bottom of the log).

Finally, the warning in the restore log occurs while running setuptools. It still tries to install the old version:


2023-10-19 19:53:03,622: DEBUG - ++ '[' -n /bin/bash -o -n '' ']'
2023-10-19 19:53:03,623: DEBUG - ++ hash -r
2023-10-19 19:53:03,623: DEBUG - + set -u
2023-10-19 19:53:03,623: DEBUG - ++ lsb_release --codename --short
2023-10-19 19:53:03,748: DEBUG - + '[' bullseye == bullseye ']'
2023-10-19 19:53:03,749: DEBUG - + pip3 install --upgrade setuptools==60.8.2 wheel pip
2023-10-19 19:53:03,751: WARNING - ../settings/scripts/_common.sh: /opt/yunohost/matrix-synapse/bin/pip3: /opt/yunohost/matrix-synapse/bin/python3: bad interpreter: No such file or directory

The install log does not have a specific version in its command, and installs a much newer version:


2023-10-19 19:59:56,824: DEBUG - ++ '[' -n /bin/bash -o -n '' ']'
2023-10-19 19:59:56,824: DEBUG - ++ hash -r
2023-10-19 19:59:56,824: DEBUG - + set -u
2023-10-19 19:59:56,824: DEBUG - + pip3 install --upgrade setuptools wheel pip
2023-10-19 20:00:00,647: DEBUG - Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
2023-10-19 20:00:00,649: DEBUG - Requirement already satisfied: setuptools in /opt/yunohost/matrix-synapse/lib/python3.9/site-packages (44.1.1)
2023-10-19 20:00:05,317: DEBUG - Collecting setuptools
2023-10-19 20:00:05,446: DEBUG -   Downloading https://www.piwheels.org/simple/setuptools/setuptools-68.2.2-py3-none-any.whl (807 kB)
2023-10-19 20:00:06,185: DEBUG - Collecting wheel

Please have a look at the suggestion behind the link in my earlier post . I wrote the step-by-step instructions to replace the file in the archive. Take care to use the correct name for the backup, of course!