[Synapse] Le service redémarre en boucle

Artanux · May 15, 2024, 8:02am

Bonjour tout le monde,

Après mon problème de backups, je m’attaque à un autre problème sur mon serveur. Ça fait un moment que ça dure, j’ai des déconnexion régulière quand j’utilise mon client Matrix (Element ou autre).

Le service synapse sur le serveur redémarre régulièrement.

Mon serveur YunoHost

Matériel: VPS en ligne
Version de YunoHost: :

# yunohost --version
yunohost: 
  repo: stable
  version: 11.2.11.3
yunohost-admin: 
  repo: stable
  version: 11.2.5
moulinette: 
  repo: stable
  version: 11.2
ssowat: 
  repo: stable
  version: 11.2

J’ai accès à mon serveur : En SSH et par la webadmin
Êtes-vous dans un contexte particulier ou avez-vous effectué des modifications particulières sur votre instance ? : non, à part les mises à jours régulières.

Description du problème

Le service synapse redémarre plusieurs fois par jour. Je ne sais pas encore à quelle fréquence.

# systemctl status synapse
● synapse.service - Synapse Matrix homeserver
     Loaded: loaded (/etc/systemd/system/synapse.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-05-15 07:17:51 UTC; 19min ago

Rien qu’en écrivant ce message, il a déjà redémarré :

root@artanux:/home/admin# systemctl status synapse
● synapse.service - Synapse Matrix homeserver
     Loaded: loaded (/etc/systemd/system/synapse.service; enabled; vendor preset: enabled)
     Active: activating (start-pre) since Wed 2024-05-15 07:41:13 UTC; 7s ago

Je suis en train d’essayer de comprendre ce qu’il se passe mais je me demande si ça ne fait pas ça depuis que Synapse demande à être seul sur le domaine. (voir Synapse now require to be alone on the domain · Issue #443 · YunoHost-Apps/synapse_ynh · GitHub )

Dans les tests de base :

Il reste plein de place sur le disque dur du serveur
Les 4 CPU tournent calmement
Le serveur utilise actuellement 1,32G/3,82G pour la RAM
Le serveur utilise 316M/511M de Swap
Les dernières mises à jours ont été réalisées hier (mais le problème est là depuis plusieurs semaines…)

36 redémarrages en mai et 31 en avril. 0 en mars. Au moins ça permet d’identifier quand ça a commencé à merdouiller.

# grep STARTING /var/log/matrix-synapse/homeserver.log | grep "2024-05-" | wc -l
36

# grep STARTING /var/log/matrix-synapse/homeserver.log | grep "2024-04-" | wc -l
31

Les premiers redémarrages en avril sont à partir du 23/04 :

# grep STARTING /var/log/matrix-synapse/homeserver.log | grep "2024-04-"
2024-04-23 12:27:26,251 - root - 355 - WARNING - main- ***** STARTING SERVER *****
2024-04-25 11:26:40,877 - root - 355 - WARNING - main- ***** STARTING SERVER *****
2024-04-25 11:27:14,517 - root - 355 - WARNING - main- ***** STARTING SERVER *****
(...)

(investigation en cours)

Josue · May 15, 2024, 7:25pm

Hello,

Would be interesting to see the last log before synapse crash, it might explain why.

On my side I can’t reproduce your issue.

Flamm3 · May 16, 2024, 3:53pm

Je viens de faire quelques recherches. Il y a des forums en anglais qui en parle. Je vous les met dans le cas où vous pourriez checker pour voir s’il y a des solutions

bonne chance

Artanux · May 16, 2024, 3:57pm

@Josue : Sorry, I am quiete busy for now but I’ll do it as soon as possible.

Artanux · May 16, 2024, 3:58pm

@Flamm3 : C’est gentil, mais ça n’a rien à voir. Je parle du serveur Matrix. ( GitHub - element-hq/synapse: Synapse: Matrix homeserver written in Python/Twisted. ) Rien à voir avec Razer. Merci quand même.

Artanux · May 18, 2024, 1:04pm

Bon, je reprends mes investigations.

# systemctl status synapse
● synapse.service - Synapse Matrix homeserver
     Loaded: loaded (/etc/systemd/system/synapse.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2024-05-18 12:28:03 UTC; 1min 26s ago

Le service vient de redémarrer. Et effectivement, il y a eu 9 redémarrages depuis mon message d’il y a trois jours :

# grep STARTING /var/log/matrix-synapse/homeserver.log | grep "2024-05-" | wc -l
45

Rien qu’aujourd’hui, il y a 14 000 lignes dans le fichier de logs…

# cat /var/log/matrix-synapse/homeserver.log | grep "2024-05-18" | wc -l
14246

J’essaye de trier un peu pour ne pas avoir trop de logs à analyser.

# cat /var/log/matrix-synapse/homeserver.log | grep "2024-05-" | grep -v INFO | wc -l
550

# cat /var/log/matrix-synapse/homeserver.log | grep "2024-05-" | grep -v WARNING | wc -l
481213

Si je regarde dans ces logs :

# # cat /var/log/matrix-synapse/homeserver.log | grep "2024-05-18 12:28:13,647 - root" -B 10 -A 10
2024-05-18 12:27:49,402 - synapse.http.matrixfederationclient - 362 - INFO - federation_transaction_transmission_loop-70- {PUT-O-29} [t2bot.io] Completed request: 200 OK in 0.14 secs, got 11 bytes - PUT matrix-federation://t2bot.io/_matrix/federation/v1/send/1716034870500
2024-05-18 12:27:49,403 - synapse.federation.sender.transaction_manager - 180 - INFO - federation_transaction_transmission_loop-70- TX [t2bot.io] {1716034870500} got 200 response
2024-05-18 12:27:49,504 - synapse.http.matrixfederationclient - 362 - INFO - federation_transaction_transmission_loop-71- {PUT-O-30} [matrix.mygaia.org] Completed request: 200 OK in 0.13 secs, got 11 bytes - PUT matrix-federation://matrix.mygaia.org/_matrix/federation/v1/send/1716034870501
2024-05-18 12:27:49,505 - synapse.federation.sender.transaction_manager - 180 - INFO - federation_transaction_transmission_loop-71- TX [matrix.mygaia.org] {1716034870501} got 200 response
2024-05-18 12:27:50,058 - synapse.http.client - 426 - INFO - GET-85- Received response to GET https://www.youtube.com/watch?v=rwwnDAss03o&t=670s: 200
2024-05-18 12:27:50,064 - synapse.access.http.8008 - 472 - INFO - GET-80- 193.121.114.248 - 8008 - {@simon:artanux.be} Processed request: 0.557sec/0.013sec (0.100sec, 0.008sec) (0.029sec/0.104sec/14) 16149B 200 "GET /_matrix/client/v3/rooms/!QXEsKNOMscCaAUthmX%3Aartanux.be/context/%24dpasCmxFT2coiPmzNTi4pkckSslII6zE6KRnwzpUUhs?filter=%7B%22lazy_load_members%22%3Atrue%7D HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36.0 (KHTML, like Gecko) Chrome/118.0.5993.159 Safari/537.36.0" [22 dbevts]
2024-05-18 12:27:50,082 - synapse.http.client - 426 - INFO - GET-87- Received response to GET https://www.youtube.com/watch?v=liHKITEUAQ8: 200
2024-05-18 12:27:50,219 - synapse.http.client - 426 - INFO - GET-86- Received response to GET https://www.amazon.fr/dp/B09NPZBH2P: 200
2024-05-18 12:27:50,226 - synapse.access.http.8008 - 472 - INFO - GET-89- 193.121.114.248 - 8008 - {None} Processed request: 0.091sec/0.064sec (0.009sec, 0.000sec) (0.003sec/0.009sec/2) 80B 200 "GET /_matrix/client/v3/profile/%40simon%3Aartanux.be HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36.0 (KHTML, like Gecko) Chrome/118.0.5993.159 Safari/537.36.0" [0 dbevts]
2024-05-18 12:27:50,328 - synapse.access.http.8008 - 472 - INFO - GET-90- 193.121.114.248 - 8008 - {@simon:artanux.be} Processed request: 0.067sec/0.003sec (0.039sec, 0.004sec) (0.003sec/0.025sec/3) 16149B 200 "GET /_matrix/client/v3/rooms/!QXEsKNOMscCaAUthmX%3Aartanux.be/context/%24dpasCmxFT2coiPmzNTi4pkckSslII6zE6KRnwzpUUhs?filter=%7B%22lazy_load_members%22%3Atrue%7D HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36.0 (KHTML, like Gecko) Chrome/118.0.5993.159 Safari/537.36.0" [0 dbevts]
2024-05-18 12:28:13,647 - root - 355 - WARNING - main- ***** STARTING SERVER *****
2024-05-18 12:28:13,648 - root - 356 - WARNING - main- Server /opt/yunohost/matrix-synapse/lib/python3.9/site-packages/synapse/app/homeserver.py version 1.105.1
2024-05-18 12:28:13,649 - root - 361 - WARNING - main- Copyright (c) 2023 New Vector, Inc
2024-05-18 12:28:13,650 - root - 362 - WARNING - main- Licensed under the AGPL 3.0 license. Website: https://github.com/element-hq/synapse
2024-05-18 12:28:13,650 - root - 365 - INFO - main- Server hostname: artanux.be
2024-05-18 12:28:13,652 - root - 366 - INFO - main- Instance name: master
2024-05-18 12:28:13,652 - root - 367 - INFO - main- Twisted reactor: EPollReactor
2024-05-18 12:28:13,653 - synapse.app.homeserver - 356 - INFO - main- Setting up server
2024-05-18 12:28:13,654 - synapse.server - 345 - INFO - main- Setting up.
2024-05-18 12:28:13,707 - synapse.storage.databases - 73 - INFO - main- [database config 'master']: Checking database server
2024-05-18 12:28:13,713 - synapse.storage.databases - 76 - INFO - main- [database config 'master']: Preparing for databases ['main', 'state']

Je ne vois rien qui explique le redémarrage du service.

En regardant avec journal

# journalctl -xe | grep synapse | grep "exited" -B 20
(...)
May 18 12:20:46 artanux.be systemd[1]: synapse.service: Failed with result 'core-dump'.
░░ The unit synapse.service has entered the 'failed' state with result 'core-dump'.
May 18 12:20:46 artanux.be systemd[1]: synapse.service: Consumed 24.675s CPU time.
░░ The unit synapse.service completed and consumed the indicated resources.
May 18 12:20:50 artanux.be systemd[1]: synapse.service: Scheduled restart job, restart counter is at 15.
░░ Automatic restarting of the unit synapse.service has been scheduled, as the result for
░░ Subject: A stop job for unit synapse.service has finished
░░ A stop job for unit synapse.service has finished.
May 18 12:20:50 artanux.be systemd[1]: synapse.service: Consumed 24.675s CPU time.
░░ The unit synapse.service completed and consumed the indicated resources.
░░ Subject: A start job for unit synapse.service has begun execution
░░ A start job for unit synapse.service has begun execution.
May 18 12:20:58 artanux.be python[396928]: File '/etc/matrix-synapse/conf.d/app_service.yaml' is empty or doesn't parse into a key-value map. IGNORING.
░░ Subject: A start job for unit synapse.service has finished successfully
░░ A start job for unit synapse.service has finished successfully.
May 18 12:21:09 artanux.be python[396938]: 2024-05-18 12:21:09,377 - root - 356 - WARNING - main- Server /opt/yunohost/matrix-synapse/lib/python3.9/site-packages/synapse/app/homeserver.py version 1.105.1
May 18 12:21:09 artanux.be python[396938]: 2024-05-18 12:21:09,379 - root - 362 - WARNING - main- Licensed under the AGPL 3.0 license. Website: https://github.com/element-hq/synapse
May 18 12:21:10 artanux.be python[396938]: 2024-05-18 12:21:10,743 - synapse.app - 56 - WARNING - sentinel- Failed to listen on 0.0.0.0, continuing because listening on [::]
May 18 12:27:47 artanux.be python[396938]: 2024-05-18 12:27:47,835 - synapse.http.server - 782 - WARNING - GET-43- Not sending response to request <XForwardedForRequest at 0x7f4020799d90 method='GET' uri='/_matrix/client/v3/sync?filter=3&timeout=30000&set_presence=unavailable&since=s33101_2492534_0_68086_4621_4_2115_141691_0_3' clientproto='HTTP/1.0' site='8008'>, already disconnected.
May 18 12:27:51 artanux.be systemd[1]: synapse.service: Main process exited, code=dumped, status=4/ILL
░░ An ExecStart= process belonging to unit synapse.service has exited.

Artanux · May 19, 2024, 8:18am

Si je supprime l’application synapse, ça n’efface pas ses données, exact ? Je peux donc tenter de supprimer et réinstaller l’application sans risque de perte de données ?

(C’est ce qu’il se passe lors d’une mise à jour, si je ne dis pas de bêtises?)

Josue · May 19, 2024, 11:45am

Hello,

No, when you remove the app all data into the database are lost. Note that in synapse case, the data directory are kept but the database are removed.

So if you remove than install again the app no data will be restored.

Artanux · May 19, 2024, 12:17pm

Thank you @Josue !

Perhaps I can execute a backup (local), remove the app, install the app, restore the backup ?

Artanux · May 19, 2024, 2:42pm

J’ai arrêté le service synapse, fait un backup local et un backup via borg et puis supprimé l’application.

# systemctl stop synapse

# yunohost backup create -n manual-synapse --apps synapse

# yunohost backup create -n manual-borg-synapse --apps synapse --method borg

# yunohost app remove synapse

J’ai ensuite créé un sous-domaine dédié à Synapse ( synapse.mondomaine.be ), installé un certificat let’s encrypt dessus et j’ai ensuite réinstallé Synapse via la webadmin.

J’ai ajouté une entrée DNS pour rediriger vers mon nouveau sous-domaine :

_matrix._tcp.synapse.mondomaine.be. 3600    IN      SRV     10 0 8448 synapse.mondomaine.be.

Mon idée maintenant, c’est de restaurer les données sur cette nouvelle installation. Je pensais pouvoir faire quelque chose comme ça :

# yunohost backup restore manual-synapse --apps synaps

Sauf que ça me renvoie une erreur (“l’application existe déjà”). Ok. J’ai une instance de Synapse. Comment je restaure mes données dessus maintenant ?

Je suis tombé sur ce post : Synapse backup/restore failed - #9 by ludovic

# cd /home/yunohost.backup/archives/

# tar -xvf manual-synapse.tar -C ../tmp/

# cd /home/yunohost.backup/tmp/apps/synapse/settings/scripts/

J’ai regardé s’il fallait modifier le script :

# vim restore

# chmod +x restore

# ./restore

Et voici le retour du script :

./restore: line 10: ../settings/scripts/_common.sh: No such file or directory
./restore: line 11: ../settings/scripts/experimental_helper.sh: No such file or directory
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[....................] > Loading settings...'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ app=synapse
+ ynh_script_progression '--message=Configuring system groups'
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[....................] > Configuring system groups'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ adduser synapse ssl-cert
The user `synapse' is already a member of `ssl-cert'.
+ adduser turnserver ssl-cert
The user `turnserver' is already a member of `ssl-cert'.
+ ynh_script_progression '--message=Fixing database type...' --weight=1
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[....................] > Fixing database type...'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ ynh_psql_execute_as_root '--sql=update pg_database set datcollate='\''C'\'', datctype='\''C'\'' where datname='\'''\'';'
+ local legacy_args=sd
+ args_array=(['s']='sql=' ['d']='database=')
+ local -A args_array
+ local sql
+ local database
+ ynh_handle_getopts_args '--sql=update pg_database set datcollate='\''C'\'', datctype='\''C'\'' where datname='\'''\'';'
+ set +o xtrace
+ database=
+ '[' -n '' ']'
++ cat /etc/yunohost/psql
+ ynh_psql_connect_as --user=postgres --password=blablabla
+ local legacy_args=upd
+ args_array=(['u']='user=' ['p']='password=' ['d']='database=')
+ local -A args_array
+ local user
+ local password
+ local database
+ ynh_handle_getopts_args --user=postgres --password=blablabla
+ set +o xtrace
+ database=
+ sudo --login --user=postgres PGUSER=postgres PGPASSWORD=blablabla psql ''
UPDATE 0
+ ynh_script_progression '--message=Restoring directory and configuration...' --weight=10
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[++..................] > Restoring directory and configuration...'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ cp -r ../backup/var/lib/matrix-synapse /var/lib/matrix-synapse
cp: cannot stat '../backup/var/lib/matrix-synapse': No such file or directory
+ cp -r ../backup/var/log/matrix-synapse /var/log/matrix-synapse
cp: cannot stat '../backup/var/log/matrix-synapse': No such file or directory
+ cp -r ../backup/etc/matrix-synapse /etc/matrix-synapse
cp: cannot stat '../backup/etc/matrix-synapse': No such file or directory
+ mkdir -p /etc/matrix-synapse/app-service
+ ynh_script_progression '--message=Check for source up to date...' --weight=5
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[##+.................] > Check for source up to date...'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ install_sources
./restore: line 55: install_sources: command not found
+ ynh_script_progression '--message=Reload Fail2Ban...' --weight=6
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[###+................] > Reload Fail2Ban...'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ ynh_systemd_action --action=restart --service_name=fail2ban
+ local legacy_args=nalpte
+ args_array=(['n']='service_name=' ['a']='action=' ['l']='line_match=' ['p']='log_path=' ['t']='timeout=' ['e']='length=')
+ local -A args_array
+ local service_name
+ local action
+ local line_match
+ local length
+ local log_path
+ local timeout
+ ynh_handle_getopts_args --action=restart --service_name=fail2ban
+ set +o xtrace
+ service_name=fail2ban
+ action=restart
+ line_match=
+ length=20
+ log_path=/var/log/fail2ban/fail2ban.log
+ timeout=300
+ '[' restart == stop ']'
+ [[ -n '' ]]
+ '[' restart == reload ']'
++ date --utc --rfc-3339=seconds
++ cut -d+ -f1
+ local 'time_start=2024-05-19 14:26:53 UTC'
+ systemctl restart fail2ban
+ [[ -n '' ]]
+ ynh_script_progression '--message=Restoring the PostgreSQL database...' --weight=13
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[####++..............] > Restoring the PostgreSQL database...'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ ynh_psql_execute_file_as_root --file=/dump.sql --database=
+ local legacy_args=fd
+ args_array=(['f']='file=' ['d']='database=')
+ local -A args_array
+ local file
+ local database
+ ynh_handle_getopts_args --file=/dump.sql --database=
+ set +o xtrace
+ database=
+ '[' -n '' ']'
++ cat /etc/yunohost/psql
+ ynh_psql_connect_as --user=postgres --password=blablabla
/usr/share/yunohost/helpers.d/postgresql: line 78: /dump.sql: No such file or directory
+ ynh_script_progression '--message=Enable systemd services' --weight=2
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[######..............] > Enable systemd services'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ systemctl enable synapse.service --quiet
+ systemctl enable synapse-coturn.service --quiet
+ yunohost service add synapse --log /var/log/matrix-synapse/homeserver.log --needs_exposed_ports --description 'Main matrix server service.'
usage: yunohost service add name [-h] [-d DESCRIPTION] [-l LOG [LOG ...]] [--test_status TEST_STATUS] [--test_conf TEST_CONF] [--needs_exposed_ports PORT [PORT ...]] [-n]
yunohost service add: error: argument --needs_exposed_ports: expected at least one argument
+ yunohost service add synapse-coturn --needs_exposed_ports --description 'Turn server for matrix server. Used for audio and video call.'
usage: yunohost service add name [-h] [-d DESCRIPTION] [-l LOG [LOG ...]] [--test_status TEST_STATUS] [--test_conf TEST_CONF] [--needs_exposed_ports PORT [PORT ...]] [-n]
yunohost service add: error: argument --needs_exposed_ports: expected at least one argument
+ ynh_script_progression '--message=Creating a dh file...' --weight=40
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[######+++++++.......] > Creating a dh file...'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ '[' '!' -e /etc/ssl/private/dh2048.pem ']'
+ ynh_script_progression '--message=Reconfiguring Coturn...' --weight=23
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[#############++++...] > Reconfiguring Coturn...'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ configure_coturn
./restore: line 108: configure_coturn: command not found
+ ynh_script_progression '--message=Configuring log rotation...'
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[#################...] > Configuring log rotation...'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ ynh_use_logrotate --logfile /var/log/matrix-synapse
+ all_args=('--logfile' '/var/log/matrix-synapse')
+ local all_args
++ seq 0 1
+ for I in $(seq 0 $(($# - 1)))
+ [[ --logfile == \-\-\n\o\n\-\a\p\p\e\n\d ]]
+ [[ --logfile == \-\-\n\o\n\a\p\p\e\n\d ]]
+ for I in $(seq 0 $(($# - 1)))
+ [[ /var/log/matrix-synapse == \-\-\n\o\n\-\a\p\p\e\n\d ]]
+ [[ /var/log/matrix-synapse == \-\-\n\o\n\a\p\p\e\n\d ]]
+ set -- --logfile /var/log/matrix-synapse
+ local legacy_args=lu
+ args_array=(['l']='logfile=' ['u']='specific_user=')
+ local -A args_array
+ local logfile
+ local specific_user
+ ynh_handle_getopts_args --logfile /var/log/matrix-synapse
+ set +o xtrace
+ logfile=/var/log/matrix-synapse
+ specific_user=
+ set -o noglob
+ [[ -z /var/log/matrix-synapse ]]
+ [[ /var/log/matrix-synapse != \l\o\g ]]
+ [[ /var/log/matrix-synapse != \t\x\t ]]
+ logfile='/var/log/matrix-synapse/*.log'
+ set +o noglob
+ for stuff in $logfile
++ dirname /var/log/matrix-synapse/homeserver.log
+ mkdir --parents /var/log/matrix-synapse
+ for stuff in $logfile
++ dirname /var/log/matrix-synapse/turnserver.log
+ mkdir --parents /var/log/matrix-synapse
+ local su_directive=
+ [[ -n '' ]]
++ mktemp
+ local tempconf=/tmp/tmp.paPnHDF0i7
+ cat
+ [[ true == \t\r\u\e ]]
+ cat /tmp/tmp.paPnHDF0i7
+ FIRST_CALL_TO_LOGROTATE=false
+ chmod 644 /etc/logrotate.d/synapse
+ mkdir -p /var/log/synapse
+ chmod 750 /var/log/synapse
+ ynh_script_progression '--message=Protecting directories...' --weight=3
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[#################+..] > Protecting directories...'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ set_permissions data
./restore: line 124: set_permissions: command not found
+ ynh_script_progression '--message=Restarting Synapse services...' --weight=7
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[##################+.] > Restarting Synapse services...'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ ynh_systemd_action --service_name=synapse-coturn.service --action=restart
+ local legacy_args=nalpte
+ args_array=(['n']='service_name=' ['a']='action=' ['l']='line_match=' ['p']='log_path=' ['t']='timeout=' ['e']='length=')
+ local -A args_array
+ local service_name
+ local action
+ local line_match
+ local length
+ local log_path
+ local timeout
+ ynh_handle_getopts_args --service_name=synapse-coturn.service --action=restart
+ set +o xtrace
+ service_name=synapse-coturn.service
+ action=restart
+ line_match=
+ length=20
+ log_path=/var/log/synapse-coturn.service/synapse-coturn.service.log
+ timeout=300
+ '[' restart == stop ']'
+ [[ -n '' ]]
+ '[' restart == reload ']'
++ date --utc --rfc-3339=seconds
++ cut -d+ -f1
+ local 'time_start=2024-05-19 14:27:04 UTC'
+ systemctl restart synapse-coturn.service
+ [[ -n '' ]]
+ ynh_systemd_action --service_name=synapse.service --action=restart '--line_match=Synapse now listening on TCP port ' --log_path=/var/log/matrix-synapse/homeserver.log --timeout=300
+ local legacy_args=nalpte
+ args_array=(['n']='service_name=' ['a']='action=' ['l']='line_match=' ['p']='log_path=' ['t']='timeout=' ['e']='length=')
+ local -A args_array
+ local service_name
+ local action
+ local line_match
+ local length
+ local log_path
+ local timeout
+ ynh_handle_getopts_args --service_name=synapse.service --action=restart '--line_match=Synapse now listening on TCP port ' --log_path=/var/log/matrix-synapse/homeserver.log --timeout=300
+ set +o xtrace
+ service_name=synapse.service
+ action=restart
+ line_match='Synapse now listening on TCP port '
+ length=20
+ log_path=/var/log/matrix-synapse/homeserver.log
+ timeout=300
+ '[' restart == stop ']'
+ [[ -n Synapse now listening on TCP port  ]]
++ mktemp
+ local templog=/tmp/tmp.WW9fPIqUj9
+ '[' /var/log/matrix-synapse/homeserver.log == systemd ']'
+ local pid_tail=483005
+ tail --follow=name --retry --lines=0 /var/log/matrix-synapse/homeserver.log
+ '[' restart == reload ']'
++ date --utc --rfc-3339=seconds
++ cut -d+ -f1
+ local 'time_start=2024-05-19 14:27:06 UTC'
+ systemctl restart synapse.service
+ [[ -n Synapse now listening on TCP port  ]]
+ set +x
+ echo 'The service synapse.service has correctly executed the action restart.'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ break
+ set -x
+ '[' 12 -ge 3 ']'
+ echo ''

+ '[' 12 -eq 300 ']'
+ ynh_clean_check_starting
+ '[' -n 483005 ']'
+ kill -SIGTERM 483005
+ '[' -n /tmp/tmp.WW9fPIqUj9 ']'
+ ynh_secure_remove --file=/tmp/tmp.WW9fPIqUj9
+ local legacy_args=f
+ args_array=(['f']='file=')
+ local -A args_array
+ local file
+ ynh_handle_getopts_args --file=/tmp/tmp.WW9fPIqUj9
+ set +o xtrace
+ set +o xtrace
+ ynh_script_progression '--message=Reloading NGINX web server...'
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[###################.] > Reloading NGINX web server...'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace
+ ynh_systemd_action --service_name=php-fpm --action=reload
+ local legacy_args=nalpte
+ args_array=(['n']='service_name=' ['a']='action=' ['l']='line_match=' ['p']='log_path=' ['t']='timeout=' ['e']='length=')
+ local -A args_array
+ local service_name
+ local action
+ local line_match
+ local length
+ local log_path
+ local timeout
+ ynh_handle_getopts_args --service_name=php-fpm --action=reload
+ set +o xtrace
+ service_name=php-fpm
+ action=reload
+ line_match=
+ length=20
+ log_path=/var/log/php-fpm/php-fpm.log
+ timeout=300
+ '[' reload == stop ']'
+ [[ -n '' ]]
+ '[' reload == reload ']'
+ action=reload-or-restart
++ date --utc --rfc-3339=seconds
++ cut -d+ -f1
+ local 'time_start=2024-05-19 14:27:29 UTC'
+ systemctl reload-or-restart php-fpm
Failed to reload-or-restart php-fpm.service: Unit php-fpm.service not found.
+ ynh_exec_err journalctl --quiet --no-hostname --no-pager --lines=20 --unit=php-fpm
+ [[ 6 -eq 1 ]]
++ journalctl --quiet --no-hostname --no-pager --lines=20 --unit=php-fpm
+ ynh_print_err --message=
+ local legacy_args=m
+ args_array=(['m']='message=')
+ local -A args_array
+ local message
+ ynh_handle_getopts_args --message=
+ set +o xtrace
+ ynh_print_log '[Error] '
+ echo -e '[Error] '
[Error] 
+ '[' -e /var/log/php-fpm/php-fpm.log ']'
+ ynh_clean_check_starting
+ '[' -n '' ']'
+ '[' -n '' ']'
+ return 1
+ ynh_systemd_action --service_name=nginx --action=reload
+ local legacy_args=nalpte
+ args_array=(['n']='service_name=' ['a']='action=' ['l']='line_match=' ['p']='log_path=' ['t']='timeout=' ['e']='length=')
+ local -A args_array
+ local service_name
+ local action
+ local line_match
+ local length
+ local log_path
+ local timeout
+ ynh_handle_getopts_args --service_name=nginx --action=reload
+ set +o xtrace
+ service_name=nginx
+ action=reload
+ line_match=
+ length=20
+ log_path=/var/log/nginx/nginx.log
+ timeout=300
+ '[' reload == stop ']'
+ [[ -n '' ]]
+ '[' reload == reload ']'
+ action=reload-or-restart
++ date --utc --rfc-3339=seconds
++ cut -d+ -f1
+ local 'time_start=2024-05-19 14:27:35 UTC'
+ systemctl reload-or-restart nginx
+ [[ -n '' ]]
+ ynh_script_progression '--message=Restoration completed for synapse' --last
+ set +o xtrace
+ set +o xtrace
+ echo '! Helper used in legacy mode !'
+ set +x
+ echo '[####################] > Restoration completed for synapse'
/usr/share/yunohost/helpers.d/logging: line 38: $YNH_STDINFO: ambiguous redirect
+ set -o xtrace

Le service tourne. Des données ont été restaurées. Sauf que… La nouvelle configuration liée au sous-domaine (synapse.mondomaine.be) a été écrasée je suppose. Donc il faut que je répare ça…

Finalement, j’ai à nouveau supprimé synapse et réinstallé pour pouvoir analyser où est géré le sous-domaine dans la configuration avant de tout écraser en restaurant le backup.

# cd /etc/matrix-synapse/

# grep "synapse.mondomaine" *
grep: app-service: Is a directory
grep: conf.d: Is a directory
coturn.conf:realm=synapse.mondomaine.be
coturn.conf:cert=/etc/yunohost/certs/synapse.mondomaine.be/crt.pem
coturn.conf:pkey=/etc/yunohost/certs/synapse.mondomaine.be/key.pem
homeserver.yaml:public_baseurl: https://synapse.mondomaine.be/
homeserver.yaml:tls_certificate_path: "/etc/yunohost/certs/synapse.mondomaine.be/crt.pem"
homeserver.yaml:tls_private_key_path: "/etc/yunohost/certs/synapse.mondomaine.be/key.pem"
homeserver.yaml:turn_uris: [ "turns:synapse.mondomaine.be:5349", "turns:synapse.mondomaine.be:5350" ]
homeserver.yaml:  server_url: "https://synapse.mondomaine.be/_matrix/cas_server.php"
homeserver.yaml:      - synapse.mondomaine.be
homeserver.yaml:  smtp_host: "synapse.mondomaine.be"
homeserver.yaml:  notif_from: "Your Friendly %(app)s Home Server <synapse@synapse.mondomaine.be>"

Et là, je réalise qu’il y a aussi surement un gros paquet de choses dans la base de données qui pointe vers le sous-domaine ?

Finalement, j’ai à nouveau supprimé synapse et restauré depuis le backup :

# yunohost backup restore manual-synapse --apps synapse

Retour au point de départ.

DrPi · May 20, 2024, 3:24pm

Je vais sûrement dire une grosse bêtise, je ne connais pas le fonctionnement interne de Yunohost.
Est-ce que le problème pourrait venir d’une librairie Python mise à jour (à cause des dépendances d’une autre application ?) mais incompatible avec Synapse ?

Artanux · May 20, 2024, 6:00pm

Aucune idée @DrPi …

J’ai remarqué aujourd’hui que le serveur redémarre quand je l’utilise… Le service tournait depuis 19h. J’ai démarré Element sur mon ordi et le service a planté.

jarod5001 · May 20, 2024, 6:20pm

Intéressant. Que dit la console du navigateur ?

Artanux · May 20, 2024, 6:38pm

Je confirme que ça a l’air de planter -au moins- à chaque initialisation du client (sur mobile, sur un onglet navigateur, dans Ferdium, …).

J’ai pas mal d’erreurs quand j’ouvre la console dans le navigateur.

20:24:34,727 Uncaught (in promise) ReferenceError: can't access lexical declaration 'Q' before initialization
    J Markdown.ts:395
    start ReadyWatchingStore.ts:89
    get instance WidgetLayoutStore.ts:142
    ts WidgetLayoutStore.ts:517
    l tslib.es6.mjs:369
    ts ModuleUiDialog.tsx:86
    l tslib.es6.mjs:369
    ts AccessSecretStorageDialog.tsx:454
    l tslib.es6.mjs:369
    ts MatrixActionCreators.ts:378
    l tslib.es6.mjs:369
    ts MatrixClientBackedController.ts:30
    l tslib.es6.mjs:369
    tsx SettingLevel.ts:20
    l tslib.es6.mjs:369
    ts ConfigSettingsHandler.ts:68
    l tslib.es6.mjs:369
    tsx IntegrationManagers.ts:184
    l tslib.es6.mjs:369
    tsx DevtoolsDialog.tsx:147
    l tslib.es6.mjs:369
    ts RecorderWorklet.ts:1
    l tslib.es6.mjs:369
    c tslib.es6.mjs:369
    O tslib.es6.mjs:369
    <anonymous> tslib.es6.mjs:369
    <anonymous> tslib.es6.mjs:369
Markdown.ts:395:4

20:24:40,525 Got room state event for unknown room !BAnjRZqAOOJaqoTmFX:artanux.be! 8 rageshake.ts:67:77

20:24:44,491 Blocage d’une requête multiorigines (Cross-Origin Request) : la politique « Same Origin » ne permet pas de consulter la ressource distante située sur https://artanux.be/_matrix/client/v3/rooms/!QXEsKNOMscCaAUthmX%3Aartanux.be/messages?limit=50&dir=b&from=s33133_2493533_0_68109_4646_4_2135_141751_0_3&filter=%7B%22lazy_load_members%22%3Atrue%7D. Raison : l’en-tête CORS « Access-Control-Allow-Origin » est manquant. Code d’état : 302.

20:24:44,493 Blocage d’une requête multiorigines (Cross-Origin Request) : la politique « Same Origin » ne permet pas de consulter la ressource distante située sur https://artanux.be/_matrix/client/v3/keys/query. Raison : l’en-tête CORS « Access-Control-Allow-Origin » est manquant. Code d’état : 302.

20:24:44,515 Error downloading keys for @tetsumaki:matrix.tetsumaki.net: ConnectionError: fetch failed: NetworkError when attempting to fetch resource.
    l errors.ts:68
    requestOtherUrl fetch.ts:323
    request fetch.ts:240
    authedRequest fetch.ts:154
    downloadKeysForUsers client.ts:9015
    doQueuedQueries DeviceList.ts:747
    B utils.ts:454
    B utils.ts:454
    doQueuedQueries DeviceList.ts:750
    doQueuedQueries DeviceList.ts:799
    promise callback*doQueuedQueries DeviceList.ts:788
    updateDevicesForUsers DeviceList.ts:723
    doKeyDownload DeviceList.ts:604
    refreshOutdatedDeviceLists DeviceList.ts:552
    onSyncCompleted index.ts:3423
    doSync sync.ts:914
    sync sync.ts:769
rageshake.ts:67:77

20:24:44,538 Failed to get URL preview:  ConnectionError: fetch failed: NetworkError when attempting to fetch resource.
    l errors.ts:68
    requestOtherUrl fetch.ts:323
    request fetch.ts:240
    authedRequest fetch.ts:154
    getUrlPreview client.ts:5364
    ze LinkPreviewGroup.tsx:95
    ze LinkPreviewWidget.tsx:146
    d LinkPreviewGroup.tsx:38
    o Ember
    Bl React
    unstable_runWithPriority scheduler.production.min.js:17
    React 4
    unstable_runWithPriority scheduler.production.min.js:17
    React 6
    Nd TimelinePanel.tsx:664
    Nd TimelinePanel.tsx:663
    promise callback*./node_modules/matrix-react-sdk/src/components/structures/MatrixChat.tsx/Nd/< TimelinePanel.tsx:628
    maybeFill ScrollPanel.tsx:494
rageshake.ts:67:77

J’ai sélectionné ce qui me semblait pertinent mais j’ai beaucoup d’erreurs liées au blocage d’une requete multiorigines (Corss-Origin Request).

Josue · May 20, 2024, 7:29pm

Je pense ce qui reste le plus intéressant serai d’avoir le log de synapse ainsi que systemd pour le service synapse

DrPi · May 20, 2024, 7:54pm

Il y a quelques années, j’avais installé Synapse à la mano sur mon serveur. J’avais bien galéré, et je me rappelle que les Cross-Origin-Requests faisaient partie du problème.

J’ai l’habitude de noter tout ce que je fais lorsque je fais une installation complexe et je suis à peu près sûr de l’avoir fait pour Synapse. Mais impossible de remettre la main sur mes notes.

DrPi · May 20, 2024, 7:57pm

Je crois me rappeler que c’est en modifiant la configuration de nginx que j’avais réglé le problème des Cross-Origin-Request
.

jarod5001 · May 20, 2024, 9:05pm

e class="onebox allowlistedgeneric" data-onebox-src="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin">

MDN Web Docs

Access-Control-Allow-Origin - HTTP | MDN

The Access-Control-Allow-Origin response header indicates whether the response can be shared with requesting code from the given origin.

Ça pourrait t’aider

Artanux · May 21, 2024, 9:47am

@Josue : Je ne peux pas partager tout le logs de Synapse, il fait >500Mo. Mais j’ai mis pas mal d’extraits intéressants dans ce post, en retirant les doublons pour simplifier. Tu cherches quelque chose d’autre ?

Josue · May 22, 2024, 6:39am

etant donné que maintenant tu peux reproduire facilement le crash la solution serait de faire la chose suivante:

lancer ces 2 commandes dans 2 terminal différents:

tail -f -n0 /var/log/matrix-synapse/homeserver.log
journalctl -n0 -f

puis “faire crasher synapse”.
Puis ensuite analyser les log généré si on peut trouver une explication du crash.