What type of hardware are you using: Other(?)
What YunoHost version are you running: 12.0.11
How are you able to access your server: SSH
Are you in a special context or did you perform specific tweaking on your YunoHost instance ?: No
Describe your issue
Hello,
Since this morning, I’ve encountered continuous errors related to MariaDB, with messages like this in the logs:
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './ibdata1' page [page id: space=0, page number=54]. You may have to r>
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [Note] InnoDB: Page dump (16384 bytes):
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [Note] InnoDB: 9248670400000036000000000000000000000016bc1b836f0006000000000000
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [Note] InnoDB: 000000000000000000000000000000000000ffffffff0000ffffffff00000000
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [Note] InnoDB: 0000000000021232ffffffffffffffffffffffffffffffffffffffffffffffff
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
Feb 05 06:01:10 example.domain mariadbd[2163]: 2025-02-05 6:01:10 0 [Note] InnoDB: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
[...]
I noticed these errors while trying to access Nextcloud. Thinking a reinstall might fix the problem, I uninstalled Nextcloud and tried to reinstall it. However, the installation failed with this error:
Warning: Here's an extract of the logs before the crash. It might help debugging the error:
Info: DEBUG - ++ toml_to_json
Info: DEBUG - ++ python3 -c 'import toml, json, sys; print(json.dumps(toml.load(sys.stdin)))'
Info: DEBUG - + local group=www-data
Info: DEBUG - + [[ -z www-data ]]
Info: DEBUG - + chmod -R u=rwX,g=rX,o=--- /var/www/nextcloud
Info: DEBUG - + chown -R nextcloud:www-data /var/www/nextcloud
Info: DEBUG - + return
Info: INFO - [##++................] > Restoring the MySQL database...
Info: DEBUG - + ynh_mysql_db_shell
Info: DEBUG - + local database=nextcloud
Info: DEBUG - + local default_character_set=
Info: DEBUG - + [[ -n nextcloud ]]
Info: DEBUG - ++ mysql -B nextcloud
Info: DEBUG - ++ tail -n1
Info: DEBUG - ++ cut -f2
Info: DEBUG - + default_character_set=utf8mb4
Info: DEBUG - + default_character_set=--default-character-set=utf8mb4
Info: DEBUG - + mysql --default-character-set=utf8mb4 -B nextcloud
Info: WARNING - ERROR 1036 (HY000) at line 22: Table 'memories_planet_geometry' is read only
Info: DEBUG - + ynh_exit_properly
Info: Removing nextcloud…
Info: [++++++++++..........] > Removing system configurations related to nextcloud...
Info: '/etc/nginx/conf.d/cloud.example.domain.d/nextcloud.conf' wasn't deleted because it doesn't exist.
Info: '/etc/nginx/conf.d/cloud.example.domain.d/nextcloud.d' wasn't deleted because it doesn't exist.
Info: '/etc/php/8.3/fpm/pool.d/nextcloud.conf' wasn't deleted because it doesn't exist.
Info: '/etc/fail2ban/jail.d/nextcloud.conf' wasn't deleted because it doesn't exist.
Info: '/etc/fail2ban/filter.d/nextcloud.conf' wasn't deleted because it doesn't exist.
Info: '/etc/cron.d/nextcloud' wasn't deleted because it doesn't exist.
Info: [####################] > Removal of nextcloud completed
Info: Deprovisioning database...
Warning: ERROR 1036 (HY000) at line 1: Table 'memories_planet_geometry' is read only
Info: Deprovisioning apt...
Info: Deprovisioning permissions...
Info: Deprovisioning data_dir...
Info: Deprovisioning install_dir...
Info: Deprovisioning system_user...
Info: Deprovisioning sources..
Other services seem to be affected as well. For instance, I can’t access Grafana. Here are some log excerpts:
Feb 05 14:49:10 example.domain influxd-systemd-start.sh[2497]: [httpd] ::1 - - [05/Feb/2025:14:49:10 +0100] "POST /query?db=opentsdb&epoch=ms HTTP/1.1 {'q': 'SELECT mean("value") FROM "netdata.sensors.sensor_chip_cpu_thermal_virtual_0_feature_cpu_temp_subfeature_temp1_input_temperature.temperature" WHERE time >= 1738763050000ms and time <= 1738763350000ms GROUP BY time(1s) fill(none) ORDER BY time ASC'}" 200 57 "-" "Grafana/10.4.2" f9c43705-e3c7-11ef-8013-001e06490802 2139
Feb 05 14:49:10 example.domain grafana[4063]: logger=ngalert.state.manager.persist t=2025-02-05T14:49:10.021354183+01:00 level=error msg="Failed to save alert state" labels="__alert_rule_namespace_uid__=Jbn7cTMVz, __alert_rule_uid__=sx7ncTGVk, alertname=Temperature alert, datasource_uid=0, grafana_folder=General Alerting, ref_id=A, rule_uid=sx7ncTGVk" state="unsupported value type" error="Error 1836 (HY000): Running in read-only mode"
Feb 05 14:49:12 example.domain grafana[4063]: logger=secrets t=2025-02-05T14:49:12.475268507+01:00 level=error msg="Failed to get current data key" error="Error 1836 (HY000): Running in read-only mode" label=2025-02-05/root@secretKey.v1
Feb 05 14:49:12 example.domain grafana[4063]: logger=login.authinfo t=2025-02-05T14:49:12.475395177+01:00 level=error msg="failed to set auth info in cache" error="Error 1836 (HY000): Running in read-only mode"
Feb 05 14:49:12 example.domain grafana[4063]: logger=user.sync t=2025-02-05T14:49:12.483248518+01:00 level=error msg="Failed to update user" error="Error 1836 (HY000): Running in read-only mode" auth_module=ldap auth_id="uid=user,ou=users,dc=yunohost,dc=org"
Feb 05 14:49:12 example.domain grafana[4063]: logger=authn.service t=2025-02-05T14:49:12.483415189+01:00 level=error msg="Failed to run post auth hook" client=auth.client.proxy id= error="[user.sync.internal] unable to update user"
I’ve also seen similar errors on Yourls and other services using MariaDB.
Running mysqlcheck --all-databases gives me many corruption errors on several tables:
grafana.alert
Warning : InnoDB: Index 'IDX_alert_org_id_id' contains 0 entries, should be 1.
Warning : InnoDB: Index 'IDX_alert_state' contains 0 entries, should be 1.
Warning : InnoDB: Index 'IDX_alert_dashboard_id' contains 0 entries, should be 1.
error : Corrupt
grafana.alert_configuration
Warning : InnoDB: Index 'UQE_alert_configuration_org_id' contains 0 entries, should be 1.
error : Corrupt
grafana.alert_configuration_history OK
grafana.alert_image OK
grafana.alert_instance
Warning : InnoDB: Index 'IDX_alert_instance_rule_org_id_rule_uid_current_state' contains 0 entries, should be 1.
Warning : InnoDB: Index 'IDX_alert_instance_rule_org_id_current_state' contains 0 entries, should be 1.
error : Corrupt
grafana.alert_notification
Warning : InnoDB: Index 'UQE_alert_notification_org_id_uid' contains 0 entries, should be 1.
error : Corrupt
grafana.alert_notification_state
Warning : InnoDB: Index 'UQE_alert_notification_state_org_id_alert_id_notifier_id' contains 0 entries, should be 1.
Warning : InnoDB: Index 'IDX_alert_notification_state_alert_id' contains 0 entries, should be 1.
error : Corrupt
grafana.alert_rule
Warning : InnoDB: Index 'UQE_alert_rule_org_id_uid' contains 0 entries, should be 1.
Warning : InnoDB: Index 'UQE_alert_rule_org_id_namespace_uid_title' contains 0 entries, should be 1.
Warning : InnoDB: Index 'IDX_alert_rule_org_id_namespace_uid_rule_group' contains 0 entries, should be 1.
Warning : InnoDB: Index 'IDX_alert_rule_org_id_dashboard_uid_panel_id' contains 0 entries, should be 1.
error : Corrupt
grafana.alert_rule_tag OK
grafana.alert_rule_version
Warning : InnoDB: Index 'UQE_alert_rule_version_rule_org_id_rule_uid_version' contains 0 entries, should be 3.
Warning : InnoDB: Index 'IDX_alert_rule_version_rule_org_id_rule_namespace_uid_rule_group' contains 0 entries, should be 3.
error : Corrupt
grafana.annotation
Warning : InnoDB: Index 'IDX_annotation_org_id_alert_id' contains 0 entries, should be 1024.
Warning : InnoDB: Index 'IDX_annotation_org_id_type' contains 0 entries, should be 1024.
Warning : InnoDB: Index 'IDX_annotation_org_id_created' contains 0 entries, should be 1024.
Warning : InnoDB: Index 'IDX_annotation_org_id_updated' contains 0 entries, should be 1024.
Warning : InnoDB: Index 'IDX_annotation_org_id_dashboard_id_epoch_end_epoch' contains 0 entries, should be 1024.
Warning : InnoDB: Index 'IDX_annotation_org_id_epoch_end_epoch' contains 0 entries, should be 1024.
Warning : InnoDB: Index 'IDX_annotation_alert_id' contains 0 entries, should be 1024.
error : Corrupt
grafana.annotation_tag OK
grafana.anon_device OK
grafana.api_key OK
grafana.builtin_role
Warning : InnoDB: Index 'UQE_builtin_role_org_id_role_id_role' contains 0 entries, should be 3.
Warning : InnoDB: Index 'IDX_builtin_role_role_id' contains 0 entries, should be 3.
Warning : InnoDB: Index 'IDX_builtin_role_role' contains 0 entries, should be 3.
Warning : InnoDB: Index 'IDX_builtin_role_org_id' contains 0 entries, should be 3.
error : Corrupt
grafana.cache_data OK
grafana.correlation OK
grafana.dashboard
Warning : InnoDB: Index 'UQE_dashboard_org_id_uid' contains 0 entries, should be 3.
Warning : InnoDB: Index 'IDX_dashboard_org_id' contains 0 entries, should be 3.
Warning : InnoDB: Index 'IDX_dashboard_gnet_id' contains 0 entries, should be 3.
Warning : InnoDB: Index 'IDX_dashboard_org_id_plugin_id' contains 0 entries, should be 3.
Warning : InnoDB: Index 'IDX_dashboard_title' contains 0 entries, should be 3.
Warning : InnoDB: Index 'IDX_dashboard_is_folder' contains 0 entries, should be 3.
Warning : InnoDB: Index 'UQE_dashboard_org_id_folder_uid_title_is_folder' contains 0 entries, should be 3.
Warning : InnoDB: Index 'IDX_dashboard_org_id_folder_id_title' contains 0 entries, should be 3.
error : Corrupt
grafana.dashboard_acl
Warning : InnoDB: Index 'UQE_dashboard_acl_dashboard_id_user_id' contains 0 entries, should be 2.
Warning : InnoDB: Index 'UQE_dashboard_acl_dashboard_id_team_id' contains 0 entries, should be 2.
Warning : InnoDB: Index 'IDX_dashboard_acl_dashboard_id' contains 0 entries, should be 2.
Warning : InnoDB: Index 'IDX_dashboard_acl_user_id' contains 0 entries, should be 2.
Warning : InnoDB: Index 'IDX_dashboard_acl_team_id' contains 0 entries, should be 2.
Warning : InnoDB: Index 'IDX_dashboard_acl_org_id_role' contains 0 entries, should be 2.
Warning : InnoDB: Index 'IDX_dashboard_acl_permission' contains 0 entries, should be 2.
error : Corrupt
grafana.dashboard_provisioning OK
grafana.dashboard_public OK
grafana.dashboard_snapshot OK
grafana.dashboard_tag OK
grafana.dashboard_version
Warning : InnoDB: Index 'UQE_dashboard_version_dashboard_id_version' contains 0 entries, should be 29.
Warning : InnoDB: Index 'IDX_dashboard_version_dashboard_id' contains 0 entries, should be 29.
error : Corrupt
grafana.data_keys OK
grafana.data_source
Warning : InnoDB: Index 'UQE_data_source_org_id_name' contains 0 entries, should be 2.
Warning : InnoDB: Index 'UQE_data_source_org_id_uid' contains 0 entries, should be 2.
Warning : InnoDB: Index 'IDX_data_source_org_id' contains 0 entries, should be 2.
Warning : InnoDB: Index 'IDX_data_source_org_id_is_default' contains 0 entries, should be 2.
error : Corrupt
grafana.entity_event OK
grafana.file OK
grafana.file_meta OK
grafana.folder
[...]
I tried a server reboot, mariadb is not starting now.
Feb 05 15:14:11 example.domain systemd[1]: mariadb.service: Deactivated successfully.
Feb 05 15:14:11 example.domain systemd[1]: Stopped mariadb.service - MariaDB 10.11.6 database server.
Feb 05 15:14:11 example.domain systemd[1]: mariadb.service: Consumed 1min 2.053s CPU time.
Feb 05 15:14:11 example.domain systemd[1]: Starting mariadb.service - MariaDB 10.11.6 database server...
Feb 05 15:14:11 example.domain mariadbd[24829]: 2025-02-05 15:14:11 0 [Note] Starting MariaDB 10.11.6-MariaDB-0+deb12u1 source revision as process 24829
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: Number of transaction pools: 1
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: Using liburing
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: Completed initialization of buffer pool
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: End of log at LSN=97652302122
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [ERROR] InnoDB: Database page corruption on disk or a failed read of file './ibdata1' page [page id: space=0, page number=54]. You may have to >
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: Page dump (16384 bytes):
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: 9248670400000036000000000000000000000016bc1b836f0006000000000000
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: 000000000000000000000000000000000000ffffffff0000ffffffff00000000
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: 0000000000021232ffffffffffffffffffffffffffffffffffffffffffffffff
[...]
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: End of page dump
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [ERROR] InnoDB: File './ibdata1' is corrupted
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: You can use CHECK TABLE to scan your table for corruption. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-mo>
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: Retry with innodb_force_recovery=5
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [ERROR] InnoDB: Plugin initialization aborted with error Page read from tablespace is corrupted.
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] InnoDB: Starting shutdown...
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [Note] Plugin 'FEEDBACK' is disabled.
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [ERROR] Unknown/unsupported storage engine: InnoDB
Feb 05 15:14:12 example.domain mariadbd[24829]: 2025-02-05 15:14:12 0 [ERROR] Aborting
Feb 05 15:14:12 example.domain systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
It seems the MariaDB database is partially corrupted. Before considering a full restore, I’d like to know if anyone has encountered this type of issue and if there’s a method to try repairing the database without data loss.
Here are the db I have
MariaDB [(none)]> show databases
-> ;
+--------------------+
| Database |
+--------------------+
| grafana |
| information_schema |
| mysql |
| nextcloud |
| performance_schema |
| snappymail |
| sys |
| yourls |
+--------------------+
If not, I can tolerate data loss, as I’m sure I have nothing important from this morning. In fact, except for nextcloud, for which I have a yunohost backup already present on the server from last upgrade 2 days ago, and almost no change since, I could almost remove and reinstall other apps if needed as it should be much faster for than getting the offsite backup from S3.
Thanks in advance for your help!
Namely: if you run on an Orange Pi with no-name uSD cards from Aliexpress, the chance of “failing storage” has another probability than when running “second hand server with enterprise SSD”, for example.
.
.
?