Bases Mysql et MongoDB HS apres shutdown intempestif

[Bimbo]

What type of hardware are you using: Old laptop or computer
What YunoHost version are you running: 12.1.25
How are you able to access your server: The webadmin
SSH
Direct access via physical keyboard/screen
Are you in a special context or did you perform specific tweaking on your YunoHost instance ?: no

Describe your issue

Bonjour,

Suite à une coupure de courant plus longue que la capacité de mon onduileur, mon serveur s’est arrêté brutalement.
Je ne peux plus accéder à des application comme roundcube ou nextcloud. Ils n’arrivent pas se connecter à leur base respective.
Effectivement mysql et mongodb n’arrivent pas à redémarrer.
J’ai voulu suivre la procédure indiqué sur cette page

How to recover from corrupted mysql

Hello, I just experienced bad situation this week-end with mysql and thought this procedure might interest other people. My yunohost server rebooted serveral times due to electrical outage and finally mysql could not start anymore due to corrupted databases. This is how I managed to get back to normal. After starting the server I had this kind of traces when trying to run mysqld service /etc/init.d/mysql[9489]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf…

Mais je suis bloqué dès le début, la commande mysqld_safe --skip-grant-tables bloque à l’étape "mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql
"
Je n’arrive pas à copier les logs de mongodb, j’ai l’erreur “La connexion avec le serveur a été fermée pour des raisons inconnues. Peut-être que ‘yunohost-api’ ne répond plus ?”

Y a 'til quelque chose a faire avant de tout réinstaller??

Merci d’avance

Share relevant logs or error messages

Les logs mysql : hastebin

[otm33]

Les tables ne sont pas forcément corrompues. Sauvegarde le dossier dans un premier temps.

Pour MariaDB
Tu peux commencer par chercher s’il y a des processus de mariadb encore actifs ps aux | grep mariadb et, le cas échéant, les arrêter (sudo kill + n° du processus)
[EDIT]: Plus précisément, chercher s’il y a un processus qui utilise aria_log_control

lsof /var/lib/mysql/aria_log_control

l’arrêter et essayer de démarrer mariadb.
Il ne sera peut-être même pas nécessaire de passer à l’étape suivante.

Ensuite, tu peux aller dans le dossier /var/lib/mysql et renommer le fichier qui bloque (plus sûr que de le supprimer)

sudo mv /var/lib/mysql/aria_log_control /var/lib/mysql/aria_log_control.bak

Puis essayer de redémarrer mariadb. Parfois, cela suffit.

Pour mongodb, c’est plus compliqué sans log mais il y a fort à parier que c’est aussi un fichier de journal ou de verrou qui pose problème.

sudo journalctl -u mongod -n 100 --no-pager

How to free exclusive lock of Mariadb · photoprism photoprism · Discussion #3523

[Bimbo]

Bonjour,

Merci d’avoir pris le temps de me répondre.

Pour Mysql , la commande ps aux | grep mariadb renvoi :
mysql 1308912 12.2 2.0 1217952 160912 ? Sl 10:20 0:00 /usr/sbin/mariadbd --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-grant-tables --skip-log-error --pid-file=/run/mysqld/mysqld.pid --socket=/run/mysqld/mysqld.sock
root 1308927 0.0 0.0 6360 2208 pts/0 S+ 10:20 0:00 grep mariadb

Mais un sudo kill 1308912 et sudo kill 1308927 ne fonctionnent pas : Aucun processus de ce type

la commande lsof /var/lib/mysql/aria_log_control ne renvoi rien,
J’ai renommé le fichier ariaz_log_control (sudo mv /var/lib/mysql/aria_log_control /var/lib/mysql/aria_log_control.bak)

Au redémarrage même erreur : hastebin

Pour mongodb, voici le logs :
oct. 02 11:06:38 XXX.eu systemd[1]: Started mongod.service - MongoDB Database Server.
oct. 02 11:07:05 XXX.eu mongod[850]: {“t”:{“$date”:“2025-10-02T09:07:05.325Z”},“s”:“I”, “c”:“CONTROL”, “id”:7484500, “ctx”:“main”,“msg”:“Environment variable MONGODB_CONFIG_OVERRIDE_NOFORK == 1, overriding "processManagement.fork" to false”}
oct. 02 11:07:18 XXX.eu systemd[1]: mongod.service: Main process exited, code=exited, status=62/n/a
oct. 02 11:07:18 XXX.eu systemd[1]: mongod.service: Failed with result ‘exit-code’.
oct. 02 11:07:18 XXX.eu systemd[1]: mongod.service: Consumed 1.085s CPU time.

[otm33]

@Bimbo

mysql 1308912 12.2 2.0 1217952 160912 ? Sl 10:20 0:00 /usr/sbin/mariadbd --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-grant-tables --skip-log-error --pid-file=/run/mysqld/mysqld.pid --socket=/run/mysqld/mysqld.sock

En fait, c’est le processus que tu as lancé avec cette commande :

@Bimbo

Mais je suis bloqué dès le début, la commande mysqld_safe --skip-grant-tables bloque à l’étape "mysqld_safe Starting
mariadbd daemon with databases from /var/lib/mysql

…qui est encore en cours.
Lancé ainsi, il monopolise le terminal dans lequel il est lancé et il faut en ouvrir un autre pour passer les autres commandes.
Pour l’arrêter: mysqladmin --socket=/run/mysqld/mysqld.sock shutdown

=> Tu peux l’arrêter, remettre le aria_log (cp -pr /var/lib/mysql/aria_log_control.bak /var/lib/mysql/aria_log_control), le relancer (mysqld_safe --skip-grant-tables), ouvrir un autre terminal et lancer la vérification des tables mysqlcheck -c -u root --all-databases, et faire le backup (reprends le tuto que tu as commencé à utiliser). Tu verras si tu as des tables corrompues. Si pas de tables corrompues, relance mariadb et retourne le log d’erreur qu’il renvoie.

=> Pour mongodb, je ne vois pas trop… le statut 62 renvoie apparemment à une incompatibilité entre la version installée et le fichier de conf… Regarde si les logs sont plus précis dans Webadmin > Tools > Services > Mongod ou sudo tail -n 50 /var/log/mongodb/mongod.log . Quelles sont tes applications en panne qui l’utilisent ?

[Bimbo]

@otm33

…qui est encore en cours.
Lancé ainsi, il monopolise le terminal dans lequel il est lancé et il faut en ouvrir un autre pour passer les autres commandes.
Pour l’arrêter: mysqladmin --socket=/run/mysqld/mysqld.sock shutdown

La commande me renvoi :

mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local server through socket '/run/mysqld/mysqld.sock' (111)'
Check that mariadbd is running and that the socket: '/run/mysqld/mysqld.sock' exists!

Celle ne me semble pas top mais j’ai continué

@otm33

Tu peux l’arrêter, remettre le aria_log (cp -pr /var/lib/mysql/aria_log_control.bak /var/lib/mysql/aria_log_control), le relancer (mysqld_safe --skip-grant-tables)

Cela me donne

251003 12:54:58 mysqld_safe Logging to syslog.
251003 12:54:58 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql

@otm33

ouvrir un autre terminal et lancer la vérification des tables mysqlcheck -c -u root --all-databases

La commande dans un autre terminal me donne

mysqlcheck: Got error: 2013: Lost connection to server at 'handshake: reading initial communication packet', system error: 104 when trying to connect

Pour MongoDB, cela plante lorsque je passe par l’interface (‘yunohost-api’ ne répond plus)
la commande

sudo tail -n 50 /var/log/mongodb/mongod.log

Renvoi

{"t":{"$date":"2025-10-02T11:07:17.413+02:00"},"s":"W",  "c":"CONTROL",  "id":5123300, "ctx":"initandlisten","msg":"vm.max_map_count is too low","attr":{"currentValue":65530,"recommendedMinimum":102400,"maxConns":51200},"tags":["startupWarnings"]}
{"t":{"$date":"2025-10-02T11:07:17.693+02:00"},"s":"F",  "c":"CONTROL",  "id":20573,   "ctx":"initandlisten","msg":"Wrong mongod version","attr":{"error":"UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document (ERROR: Location4926900: Invalid featureCompatibilityVersion document in admin.system.version: { _id: \"featureCompatibilityVersion\", version: \"4.4\" }. See https://docs.mongodb.com/master/release-notes/6.0-compatibility/#feature-compatibility. :: caused by :: Invalid feature compatibility version value '4.4'; expected '6.0' or '6.3' or '7.0'. See https://docs.mongodb.com/master/release-notes/6.0-compatibility/#feature-compatibility.). If the current featureCompatibilityVersion is below 6.0, see the documentation on upgrading at https://docs.mongodb.com/master/release-notes/6.0/#upgrade-procedures."}}
{"t":{"$date":"2025-10-02T11:07:17.693+02:00"},"s":"I",  "c":"REPL",     "id":4784900, "ctx":"initandlisten","msg":"Stepping down the ReplicationCoordinator for shutdown","attr":{"waitTimeMillis":15000}}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"REPL",     "id":4794602, "ctx":"initandlisten","msg":"Attempting to enter quiesce mode"}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"-",        "id":6371601, "ctx":"initandlisten","msg":"Shutting down the FLE Crud thread pool"}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"COMMAND",  "id":4784901, "ctx":"initandlisten","msg":"Shutting down the MirrorMaestro"}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"SHARDING", "id":4784902, "ctx":"initandlisten","msg":"Shutting down the WaitForMajorityService"}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"NETWORK",  "id":20562,   "ctx":"initandlisten","msg":"Shutdown: going to close listening sockets"}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"NETWORK",  "id":4784905, "ctx":"initandlisten","msg":"Shutting down the global connection pool"}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"CONTROL",  "id":4784906, "ctx":"initandlisten","msg":"Shutting down the FlowControlTicketholder"}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"-",        "id":20520,   "ctx":"initandlisten","msg":"Stopping further Flow Control ticket acquisitions."}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"CONTROL",  "id":4784908, "ctx":"initandlisten","msg":"Shutting down the PeriodicThreadToAbortExpiredTransactions"}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"REPL",     "id":4784909, "ctx":"initandlisten","msg":"Shutting down the ReplicationCoordinator"}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"SHARDING", "id":4784910, "ctx":"initandlisten","msg":"Shutting down the ShardingInitializationMongoD"}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"REPL",     "id":4784911, "ctx":"initandlisten","msg":"Enqueuing the ReplicationStateTransitionLock for shutdown"}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"-",        "id":4784912, "ctx":"initandlisten","msg":"Killing all operations for shutdown"}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"-",        "id":4695300, "ctx":"initandlisten","msg":"Interrupted all currently running operations","attr":{"opsKilled":3}}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"TENANT_M", "id":5093807, "ctx":"initandlisten","msg":"Shutting down all TenantMigrationAccessBlockers on global shutdown"}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"TenantMigrationBlockerNet","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"ASIO",     "id":6529201, "ctx":"initandlisten","msg":"Network interface redundant shutdown","attr":{"state":"Stopped"}}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"initandlisten","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2025-10-02T11:07:17.746+02:00"},"s":"I",  "c":"COMMAND",  "id":4784913, "ctx":"initandlisten","msg":"Shutting down all open transactions"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"REPL",     "id":4784914, "ctx":"initandlisten","msg":"Acquiring the ReplicationStateTransitionLock for shutdown"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"INDEX",    "id":4784915, "ctx":"initandlisten","msg":"Shutting down the IndexBuildsCoordinator"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"NETWORK",  "id":4784918, "ctx":"initandlisten","msg":"Shutting down the ReplicaSetMonitor"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"SHARDING", "id":4784921, "ctx":"initandlisten","msg":"Shutting down the MigrationUtilExecutor"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"MigrationUtil-TaskExecutor","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"COMMAND",  "id":4784923, "ctx":"initandlisten","msg":"Shutting down the ServiceEntryPoint"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"CONTROL",  "id":4784928, "ctx":"initandlisten","msg":"Shutting down the TTL monitor"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"CONTROL",  "id":6278511, "ctx":"initandlisten","msg":"Shutting down the Change Stream Expired Pre-images Remover"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"CONTROL",  "id":4784929, "ctx":"initandlisten","msg":"Acquiring the global lock for shutdown"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"CONTROL",  "id":4784930, "ctx":"initandlisten","msg":"Shutting down the storage engine"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"STORAGE",  "id":22320,   "ctx":"initandlisten","msg":"Shutting down journal flusher thread"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"STORAGE",  "id":22321,   "ctx":"initandlisten","msg":"Finished shutting down journal flusher thread"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"STORAGE",  "id":22322,   "ctx":"initandlisten","msg":"Shutting down checkpoint thread"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"STORAGE",  "id":22323,   "ctx":"initandlisten","msg":"Finished shutting down checkpoint thread"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"STORAGE",  "id":20282,   "ctx":"initandlisten","msg":"Deregistering all the collections"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"STORAGE",  "id":22317,   "ctx":"initandlisten","msg":"WiredTigerKVEngine shutting down"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"STORAGE",  "id":22318,   "ctx":"initandlisten","msg":"Shutting down session sweeper thread"}
{"t":{"$date":"2025-10-02T11:07:17.747+02:00"},"s":"I",  "c":"STORAGE",  "id":22319,   "ctx":"initandlisten","msg":"Finished shutting down session sweeper thread"}
{"t":{"$date":"2025-10-02T11:07:17.748+02:00"},"s":"I",  "c":"STORAGE",  "id":4795902, "ctx":"initandlisten","msg":"Closing WiredTiger","attr":{"closeConfig":"leak_memory=true,use_timestamp=false,"}}
{"t":{"$date":"2025-10-02T11:07:17.748+02:00"},"s":"I",  "c":"WTCHKPT",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1759396037,"ts_usec":748635,"thread":"850:0x7f7efa2c5c80","session_name":"close_ckpt","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 2, snapshot max: 2 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 4183595"}}}
{"t":{"$date":"2025-10-02T11:07:18.256+02:00"},"s":"I",  "c":"WTRECOV",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1759396038,"ts_usec":256139,"thread":"850:0x7f7efa2c5c80","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"shutdown checkpoint has successfully finished and ran for 508 milliseconds"}}}
{"t":{"$date":"2025-10-02T11:07:18.256+02:00"},"s":"I",  "c":"WTRECOV",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1759396038,"ts_usec":256317,"thread":"850:0x7f7efa2c5c80","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"shutdown was completed successfully and took 508ms, including 0ms for the rollback to stable, and 508ms for the checkpoint."}}}
{"t":{"$date":"2025-10-02T11:07:18.879+02:00"},"s":"I",  "c":"STORAGE",  "id":4795901, "ctx":"initandlisten","msg":"WiredTiger closed","attr":{"durationMillis":1131}}
{"t":{"$date":"2025-10-02T11:07:18.879+02:00"},"s":"I",  "c":"STORAGE",  "id":22279,   "ctx":"initandlisten","msg":"shutdown: removing fs lock..."}
{"t":{"$date":"2025-10-02T11:07:18.879+02:00"},"s":"I",  "c":"-",        "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2025-10-02T11:07:18.879+02:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"initandlisten","msg":"Now exiting"}
{"t":{"$date":"2025-10-02T11:07:18.879+02:00"},"s":"I",  "c":"CONTROL",  "id":8423404, "ctx":"initandlisten","msg":"mongod shutdown complete","attr":{"Summary of time elapsed":{"Statistics":{"Enter terminal shutdown":"0 ms","Step down the replication coordinator for shutdown":"53 ms","Time spent in quiesce mode":"0 ms","Shut down FLE Crud subsystem":"0 ms","Shut down MirrorMaestro":"0 ms","Shut down WaitForMajorityService":"0 ms","Shut down the transport layer":"0 ms","Shut down the global connection pool":"0 ms","Shut down the flow control ticket holder":"0 ms","Kill all operations for shutdown":"0 ms","Shut down all tenant migration access blockers on global shutdown":"0 ms","Shut down all open transactions":"0 ms","Acquire the RSTL for shutdown":"1 ms","Shut down the IndexBuildsCoordinator and wait for index builds to finish":"0 ms","Shut down the replica set monitor":"0 ms","Shut down the migration util executor":"0 ms","Shut down the TTL monitor":"0 ms","Shut down expired pre-images and documents removers":"0 ms","Shut down the storage engine":"1132 ms","Wait for the oplog cap maintainer thread to stop":"0 ms","Shut down full-time data capture":"0 ms","shutdownTask total elapsed time":"1186 ms"}}}}
{"t":{"$date":"2025-10-02T11:07:18.879+02:00"},"s":"I",  "c":"CONTROL",  "id":23138,   "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":62}}

Cela me bloque pour ampache nextcloud et roundcube

[otm33]

@Bimbo

{"t":{"$date":"2025-10-02T11:07:17.693+02:00"},"s":"F", "c":"CONTROL", "id":20573, "ctx":"initandlisten","msg":"Wrong mongod version","attr":{"error":"UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document (ERROR: Location4926900: Invalid featureCompatibilityVersion document in admin.system.version: { _id: \"featureCompatibilityVersion\", version: \"4.4\" }. See https://docs.mongodb.com/master/release-notes/6.0-compatibility/#feature-compatibility. :: caused by :: Invalid feature compatibility version value '4.4'; expected '6.0' or '6.3' or '7.0'. See https://docs.mongodb.com/master/release-notes/6.0-compatibility/#feature-compatibility.). If the current featureCompatibilityVersion is below 6.0, see the documentation on upgrading at https://docs.mongodb.com/master/release-notes/6.0/#upgrade-procedures."}}

C’est bien un problème d’incompatibilité, donc… et qui n’est peut être pas nouveau. Je ne crois pas que les trois applications utilisent mongod.

Pour mariadb, je regarderais de nouveau les processus en cours:

ps aux | grep mariadb 

Si tu vois
mysql 1308912 12.2 2.0 1217952 160912 ? Sl 10:20 0:00 /usr/sbin/mariadbd --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-grant-tables --skip-log-error --pid-file=/run/mysqld/mysqld.pid --socket=/run/mysqld/mysqld.sock

tenter mysqladmin --socket=/run/mysqld/mysqld.sock shutdown
ou mysqladmin shutdown

jusqu’à ce que ps aux | grep mariadb ne renvoie plus qu’une ligne avec root et grep mariadb.
Il peut y avoir un processus “fantôme” qui bloque.

[Bimbo]

Bonsoir,

Excusez mon retour tardif mais je n’ai malheureusement pas pu faire les manips avant ce soir.

Tu peux l’arrêter, remettre le aria_log (cp -pr /var/lib/mysql/aria_log_control.bak /var/lib/mysql/aria_log_control ), le relancer (mysqld_safe --skip-grant-tables ), ouvrir un autre terminal et lancer la vérification des tables mysqlcheck -c -u root --all-databases

Cela me mets l’erreur suivante :

mysqlcheck: Got error: 2002: Can’t connect to local server through socket ‘/run/mysqld/mysqld.sock’ (111) when trying to connect

[Bimbo]

Question surement con, si je récupère les fichiers des bases qui m’intéresse, je ne peux pas réinstaller puis essayer de les réimporter?