MariaDB keeps restarting

hovancik · October 11, 2022, 6:22am

My YunoHost server

Hardware: Raspberry Pi at home
YunoHost version: 11.0.9.15 (stable)
I have access to my server : Through SSH | through the webadmin | direct access via keyboard / screen | …
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no

Description of my issue

I noticed my apps that use MariaDB stopped working, so I checked services and noticed that MariaDB keeps relaunching itself. I found the logs, don’t understand a thing it says. I basically goes in circles.

I don’t remember doing anything on YNH, I basically noticed that one my lights from Home Assistant went off then then noticed none of the MariaDB apps work.

mariadb-server:
  Installed: 1:10.5.15-0+deb11u1

Oct 11 06:59:40 noho.st systemd[1]: Starting MariaDB 10.5.15 database server...
Oct 11 06:59:40 noho.st mariadbd[2778341]: 2022-10-11  6:59:40 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 2778341 ...
Oct 11 06:59:40 noho.st mariadbd[2778341]: 2022-10-11  6:59:40 0 [Note] InnoDB: Uses event mutexes
Oct 11 06:59:40 noho.st mariadbd[2778341]: 2022-10-11  6:59:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Oct 11 06:59:40 noho.st mariadbd[2778341]: 2022-10-11  6:59:40 0 [Note] InnoDB: Number of pools: 1
Oct 11 06:59:40 noho.st mariadbd[2778341]: 2022-10-11  6:59:40 0 [Note] InnoDB: Using ARMv8 crc32 instructions
Oct 11 06:59:40 noho.st mariadbd[2778341]: 2022-10-11  6:59:40 0 [Note] InnoDB: Using Linux native AIO
Oct 11 06:59:40 noho.st mariadbd[2778341]: 2022-10-11  6:59:40 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
Oct 11 06:59:40 noho.st mariadbd[2778341]: 2022-10-11  6:59:40 0 [Note] InnoDB: Completed initialization of buffer pool
Oct 11 06:59:40 noho.st mariadbd[2778341]: 2022-10-11  6:59:40 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=22467617435,22467617435
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 4 row operations to undo
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] InnoDB: Trx id counter is 22340157
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] InnoDB: Starting final batch to recover 254 pages from redo log.
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] InnoDB: 128 rollback segments are active.
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11 06:59:41 0x7f6affd100  InnoDB: Assertion failure in file ./storage/innobase/btr/btr0cur.cc line 336
Oct 11 06:59:41 noho.st mariadbd[2778341]: InnoDB: Failing assertion: btr_page_get_prev(get_block->frame) == block->page.id().page_no()
Oct 11 06:59:41 noho.st mariadbd[2778341]: InnoDB: We intentionally generate a memory trap.
Oct 11 06:59:41 noho.st mariadbd[2778341]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
Oct 11 06:59:41 noho.st mariadbd[2778341]: InnoDB: If you get repeated assertion failures or crashes, even
Oct 11 06:59:41 noho.st mariadbd[2778341]: InnoDB: immediately after the mysqld startup, there may be
Oct 11 06:59:41 noho.st mariadbd[2778341]: InnoDB: corruption in the InnoDB tablespace. Please refer to
Oct 11 06:59:41 noho.st mariadbd[2778341]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
Oct 11 06:59:41 noho.st mariadbd[2778341]: InnoDB: about forcing recovery.
Oct 11 06:59:41 noho.st mariadbd[2778341]: 221011  6:59:41 [ERROR] mysqld got signal 6 ;
Oct 11 06:59:41 noho.st mariadbd[2778341]: This could be because you hit a bug. It is also possible that this binary
Oct 11 06:59:41 noho.st mariadbd[2778341]: or one of the libraries it was linked against is corrupt, improperly built,
Oct 11 06:59:41 noho.st mariadbd[2778341]: or misconfigured. This error can also be caused by malfunctioning hardware.
Oct 11 06:59:41 noho.st mariadbd[2778341]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Oct 11 06:59:41 noho.st mariadbd[2778341]: We will try our best to scrape up some info that will hopefully help
Oct 11 06:59:41 noho.st mariadbd[2778341]: diagnose the problem, but since we have already crashed,
Oct 11 06:59:41 noho.st mariadbd[2778341]: something is definitely wrong and this may fail.
Oct 11 06:59:41 noho.st mariadbd[2778341]: Server version: 10.5.15-MariaDB-0+deb11u1
Oct 11 06:59:41 noho.st mariadbd[2778341]: key_buffer_size=134217728
Oct 11 06:59:41 noho.st mariadbd[2778341]: read_buffer_size=131072
Oct 11 06:59:41 noho.st mariadbd[2778341]: max_used_connections=0
Oct 11 06:59:41 noho.st mariadbd[2778341]: max_threads=153
Oct 11 06:59:41 noho.st mariadbd[2778341]: thread_count=0
Oct 11 06:59:41 noho.st mariadbd[2778341]: It is possible that mysqld could use up to
Oct 11 06:59:41 noho.st mariadbd[2778341]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467879 K  bytes of memory
Oct 11 06:59:41 noho.st mariadbd[2778341]: Hope that's ok; if not, decrease some variables in the equation.
Oct 11 06:59:41 noho.st mariadbd[2778341]: Thread pointer: 0x0
Oct 11 06:59:41 noho.st mariadbd[2778341]: Attempting backtrace. You can use the following information to find out
Oct 11 06:59:41 noho.st mariadbd[2778341]: where mysqld died. If you see no messages after this, something went
Oct 11 06:59:41 noho.st mariadbd[2778341]: terribly wrong...
Oct 11 06:59:41 noho.st mariadbd[2778341]: stack_bottom = 0x0 thread_stack 0x49000
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] InnoDB: 10.5.15 started; log sequence number 22476629569; transaction id 22340158
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] InnoDB: Cannot open '/var/lib/mysql/ib_buffer_pool' for reading: No such file or directory
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] Plugin 'FEEDBACK' is disabled.
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] Server socket created on IP: '127.0.0.1'.
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(my_print_stacktrace)[0x557c99ee00]
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(handle_fatal_signal)[0x557c4dd114]
Oct 11 06:59:41 noho.st mariadbd[2778359]: addr2line: 'linux-vdso.so.1': No such file
Oct 11 06:59:41 noho.st mariadbd[2778341]: linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7fa28e5788]
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] Reading of all Master_info entries succeeded
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] Added new Master_info '' to hash table
Oct 11 06:59:41 noho.st mariadbd[2778341]: 2022-10-11  6:59:41 0 [Note] /usr/sbin/mariadbd: ready for connections.
Oct 11 06:59:41 noho.st mariadbd[2778341]: Version: '10.5.15-MariaDB-0+deb11u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 11
Oct 11 06:59:41 noho.st systemd[1]: Started MariaDB 10.5.15 database server.
Oct 11 06:59:41 noho.st mariadbd[2778341]: linux/raise.c:51(__GI_raise)[0x7fa1f24eac]
Oct 11 06:59:41 noho.st mariadbd[2778341]: stdlib/abort.c:81(__GI_abort)[0x7fa1f11aa0]
Oct 11 06:59:41 noho.st /etc/mysql/debian-start[2778369]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Oct 11 06:59:41 noho.st /etc/mysql/debian-start[2778369]: Looking for 'mysql' as: /usr/bin/mysql
Oct 11 06:59:41 noho.st /etc/mysql/debian-start[2778369]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Oct 11 06:59:41 noho.st /etc/mysql/debian-start[2778369]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB.
Oct 11 06:59:41 noho.st /etc/mysql/debian-start[2778369]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB.
Oct 11 06:59:41 noho.st /etc/mysql/debian-start[2778369]: You can use --force if you still want to run mysql_upgrade
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x557c228dc0]
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(std::unique_lock<std::mutex>::unlock())[0x557c898ff8]
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(std::unique_lock<std::mutex>::unlock())[0x557c89c710]
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(std::unique_lock<std::mutex>::unlock())[0x557c83c824]
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(std::unique_lock<std::mutex>::unlock())[0x557c83c978]
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(std::_Rb_tree<std::pair<unsigned long, unsigned long>, std::pair<unsigned long, unsigned long>, std::_Identity<std::pair<unsigned long, unsigned long> >, std::less<std::pair<unsigned long, unsigned long> >, std::allocator<std::pair<unsigned long, unsigned long> > >::equal_range(std::pair<unsigned long, unsigned long> const&))[0x557c93f0a4]
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(std::_Rb_tree<std::pair<unsigned long, unsigned long>, std::pair<unsigned long, unsigned long>, std::_Identity<std::pair<unsigned long, unsigned long> >, std::less<std::pair<unsigned long, unsigned long> >, std::allocator<std::pair<unsigned long, unsigned long> > >::equal_range(std::pair<unsigned long, unsigned long> const&))[0x557c93f424]
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(std::unique_lock<std::mutex>::unlock())[0x557c846b20]
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(std::unique_lock<std::mutex>::unlock())[0x557c804500]
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(std::unique_lock<std::mutex>::unlock())[0x557c86fbe0]
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(std::unique_lock<std::mutex>::unlock())[0x557c870074]
Oct 11 06:59:41 noho.st mariadbd[2778341]: ??:0(std::unique_lock<std::mutex>::unlock())[0x557c870320]
Oct 11 06:59:41 noho.st mariadbd[2778341]: nptl/pthread_create.c:477(start_thread)[0x7fa2304648]
Oct 11 06:59:41 noho.st mariadbd[2778341]: aarch64/clone.S:81(thread_start)[0x7fa1fc3c1c]
Oct 11 06:59:41 noho.st mariadbd[2778341]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
Oct 11 06:59:41 noho.st mariadbd[2778341]: information that should help you find out what is causing the crash.
Oct 11 06:59:41 noho.st mariadbd[2778341]: Writing a core file...
Oct 11 06:59:41 noho.st mariadbd[2778341]: Working directory at /var/lib/mysql
Oct 11 06:59:41 noho.st mariadbd[2778341]: Resource Limits:
Oct 11 06:59:41 noho.st mariadbd[2778341]: Limit                     Soft Limit           Hard Limit           Units
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max cpu time              unlimited            unlimited            seconds
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max file size             unlimited            unlimited            bytes
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max data size             unlimited            unlimited            bytes
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max stack size            8388608              unlimited            bytes
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max core file size        0                    unlimited            bytes
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max resident set          unlimited            unlimited            bytes
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max processes             29950                29950                processes
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max open files            32768                32768                files
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max locked memory         65536                65536                bytes
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max address space         unlimited            unlimited            bytes
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max file locks            unlimited            unlimited            locks
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max pending signals       29950                29950                signals
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max msgqueue size         819200               819200               bytes
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max nice priority         0                    0
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max realtime priority     0                    0
Oct 11 06:59:41 noho.st mariadbd[2778341]: Max realtime timeout      unlimited            unlimited            us
Oct 11 06:59:41 noho.st mariadbd[2778341]: Core pattern: core
Oct 11 06:59:41 noho.st systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
Oct 11 06:59:41 noho.st systemd[1]: mariadb.service: Failed with result 'signal'.
Oct 11 06:59:41 noho.st systemd[1]: mariadb.service: Consumed 1.164s CPU time.

Any help appreciated, no idea where to look.

aoz · October 11, 2022, 7:05am

I think I have the same issue on my server (topic "Internal Server Error" after Nextcloud upgrade (mySQL issue))

But no idea how to fix that

hovancik · October 14, 2022, 8:38am

I’ve create issue in MariaDB jira: [MDEV-29790] MariaDB keeps restarting - Jira

hovancik · October 16, 2022, 10:12am

@aoz so I think I might have fixed it. (Haven’t had time to check everything). Not sure if HA runs correctly, because I have other issues with my YNH server.

sudo vim /etc/mysql/my.cnf and add

[MYSQLD]
innodb_force_recovery=3

restart mariadb and it should start

then run sudo mysqlcheck --all-databases and see which db is corrupted.

backup that db, drop it, then stop mariadb server and remove sudo rm -rf /var/lib/mysql/homeassistant/ (in my case). start server again and import the backup.

aoz · October 16, 2022, 8:44pm

Hi @hovancik ,

I tried your proposal but it did not fixed the problem. For you information, running sudo mysqlcheck --all-databases outputs that all db are “OK”.

After restarting the server I always have the same issues (Nextcloud & Baikal app are down).

hovancik · October 17, 2022, 6:21am

Sorry to hear that It must be some other issue, then.

aoz · October 20, 2022, 7:56am

@hovancik , is the problem definitively fixed on your side?

If it is the case, could you please explain how to perform the steps you described above:

backup that database: ???
drop it : ???
then stop mariadb server: sudo systemctl stop mariadb.service
remove the database folder sudo rm -rf /var/lib/mysql/databaseFolderName/
start server again: sudo systemctl start mariadb.service
import the backup : ???

Thanks a lot!

hovancik · October 20, 2022, 8:28am

Hi,

I have some issues with some components of Home Assistant not working as they should, but did not have the time to look into it yet. Overall Home Assistant works but it says something is wrong with logging or something similar.

Backup: regular sudo mysqldump homeassistant > ha.sql
drop: sudo mysql and then if I remember DROP DATABASE homeassistant;
import: sudo mysql and CREATE DATABASE homeassistant; (please google, don’t remember exactly)
and then sudo mysql homeassistant < ha.sql