Mysql and postgresql go down

What type of hardware are you using: Raspberry Pi 3, 4+
What YunoHost version are you running: 11.2.30.2 (stable).
How are you able to access your server: The webadmin
SSH
Direct access via physical keyboard/screen

Describe your issue

in order to update YH i had to do

sudo apt-key del 95BD4743; wget -nv -O - “https://packages.sury.org/php/apt.gpg” | sudo apt-key add -

like it is writen here

update to 11.2.30.2 could be done!

But now i have an unstable system- mysql and postgresql go down and updates of apps do not work- what could i try? thanks

Share relevant logs or error messages

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/moulinette/interfaces/api.py”, line 450, in process
ret = self.actionsmap.process(arguments, timeout=30, route=_route)
File “/usr/lib/python3/dist-packages/moulinette/actionsmap.py”, line 574, in process
return func(**arguments)
File “/usr/lib/python3/dist-packages/yunohost/tools.py”, line 388, in tools_update
upgradable_apps = _list_upgradable_apps()
File “/usr/lib/python3/dist-packages/yunohost/app.py”, line 2752, in _list_upgradable_apps
manifest, extracted_app_folder = _extract_app(absolute_app_name)
File “/usr/lib/python3/dist-packages/yunohost/app.py”, line 2584, in _extract_app
return _extract_app_from_gitrepo(
File “/usr/lib/python3/dist-packages/yunohost/app.py”, line 2699, in _extract_app_from_gitrepo
extracted_app_folder = _make_tmp_workdir_for_app()
File “/usr/lib/python3/dist-packages/yunohost/app.py”, line 3176, in _make_tmp_workdir_for_app
shutil.rmtree(path)
File “/usr/lib/python3.9/shutil.py”, line 718, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File “/usr/lib/python3.9/shutil.py”, line 655, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File “/usr/lib/python3.9/shutil.py”, line 655, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File “/usr/lib/python3.9/shutil.py”, line 655, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
[Previous line repeated 1 more time]
File “/usr/lib/python3.9/shutil.py”, line 675, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File “/usr/lib/python3.9/shutil.py”, line 673, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 117] Structure needs cleaning: ‘master’


postgresql journalctl

Sep 13 15:17:30 systemd[1]: Starting PostgreSQL Cluster 13-main…
Sep 13 15:17:32 postgresql@13-main[575]: Removed stale pid file.
Sep 13 15:17:39 systemd[1]: Started PostgreSQL Cluster 13-main.


mysql journalctl

Sep 14 13:07:52 mariadbd[732]: 2024-09-14 13:07:52 187 [ERROR] mariadbd: Lock wait timeout exceeded; try restarting transaction
Sep 14 13:07:52 mariadbd[732]: 2024-09-14 13:07:52 187 [Warning] Sort aborted, host: localhost, user: xxx, thread: 187, query: SELECT workerq_id, workerq_cmd FROM workerq WHERE workerq_reservationid IS NULL ORDER BY workerq_priority DESC, workerq_id ASC LIMIT 1 FOR UPDATE NOWAIT

update by the webinterface do not work

by command line it does work sometimes…

this is what i got right now:

root@my:~# apt update                                                                                                                               
Hit:1 http://deb.debian.org/debian bullseye InRelease
Hit:2 http://deb.debian.org/debian bullseye-updates InRelease
Hit:3 http://archive.raspberrypi.org/debian bullseye InRelease
Hit:4 http://forge.yunohost.org/debian bullseye InRelease                                                                          
Hit:5 https://packages.sury.org/php bullseye InRelease                                                                       
Get:6 http://security.debian.org/debian-security bullseye-security InRelease [27.2 kB]
Err:2 http://deb.debian.org/debian bullseye-updates InRelease
  Splitting up /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye-updates_InRelease into data and signature failed                             
Err:5 https://packages.sury.org/php bullseye InRelease
  Splitting up /var/lib/apt/lists/packages.sury.org_php_dists_bullseye_InRelease into data and signature failed                                     
Hit:7 http://security.debian.org/debian-security bullseye-security/main arm64 Packages
Get:8 http://security.debian.org/debian-security bullseye-security/main armhf Packages [283 kB]
Hit:8 http://security.debian.org/debian-security bullseye-security/main armhf Packages
Hit:10 http://security.debian.org/debian-security bullseye-security/main Translation-en
Err:7 http://security.debian.org/debian-security bullseye-security/main arm64 Packages
  Hash Sum mismatch                                                                                                                                 
  Hashes of expected file:                                                                                                                          
   - Filesize:1816054 [weak]                                                                                                                        
   - SHA256:dacfed400e644a792de85f855de8cc0259217ff18d870b5eeeb63d84ceef171f                                                                        
  Hashes of received file:                                                                                                                          
   - SHA256:7c22159b082b43ab47d76641d5c2c395aff50cd57d9d303236a0542f63c0d48e                                                                        
   - Filesize:1816054 [weak]                                                                                                                        
  Release file created at: Sun, 15 Sep 2024 12:12:16 +0000                                                                                          
Reading package lists... Done  
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://d
eb.debian.org/debian bullseye-updates InRelease: Splitting up /var/lib/apt/lists/deb.debian.org_debian_dists_bullseye-updates_InRelease into data an
d signature failed                                                                                                                                  
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://
packages.sury.org/php bullseye InRelease: Splitting up /var/lib/apt/lists/packages.sury.org_php_dists_bullseye_InRelease into data and signature fai
led                                                                                                                                                 
W: Symlinking file /var/lib/apt/lists/security.debian.org_debian-security_dists_bullseye-security_main_binary-armhf_Packages to /var/lib/apt/lists/p
artial/security.debian.org_debian-security_dists_bullseye-security_main_binary-armhf_Packages failed - pkgAcqIndex::StageDownloadDone (17: File exis
ts)                                                                                                                                                 
W: Symlinking file /var/lib/apt/lists/security.debian.org_debian-security_dists_bullseye-security_main_i18n_Translation-en to /var/lib/apt/lists/par
tial/security.debian.org_debian-security_dists_bullseye-security_main_i18n_Translation-en failed - pkgAcqIndex::StageDownloadDone (17: File exists) 
W: Failed to fetch http://deb.debian.org/debian/dists/bullseye-updates/InRelease  Splitting up /var/lib/apt/lists/deb.debian.org_debian_dists_bullse
ye-updates_InRelease into data and signature failed                                                                                                 
W: Failed to fetch https://packages.sury.org/php/dists/bullseye/InRelease  Splitting up /var/lib/apt/lists/packages.sury.org_php_dists_bullseye_InRe
lease into data and signature failed                                                                                                                
E: Failed to fetch store:/var/lib/apt/lists/partial/security.debian.org_debian-security_dists_bullseye-security_main_binary-arm64_Packages  Hash Sum
 mismatch                                                                                                                                           
   Hashes of expected file:                                                                                                                         
    - Filesize:1816054 [weak]                                                                                                                       
    - SHA256:dacfed400e644a792de85f855de8cc0259217ff18d870b5eeeb63d84ceef171f                                                                       
   Hashes of received file:                                                                                                                         
    - SHA256:7c22159b082b43ab47d76641d5c2c395aff50cd57d9d303236a0542f63c0d48e                                                                       
    - Filesize:1816054 [weak]                                                                                                                       
   Release file created at: Sun, 15 Sep 2024 12:12:16 +0000                                                                                         
W: Some index files failed to download. They have been ignored, or old ones used instead.
                                                                                                                       

hello, the signature for packages.surry.org is not here…

have you try this ?

yunohost tools regen-conf -f

Since YunoHost 11.2.10.1, yunohost tools regen-conf is clearing the APT keys

1 Like

thanks - yes this helped.

had to reboot the system a couple of times but now the system is up… let’s see

the system is still unstable and Mysql and postgresql keeps cashing.

I can´t even sometimes open the page

/yunohost/admin/#/services

it gives me just a black page - sometimes

What else could i try ? THANKS

there is also still a problem with keys… if i try to lunch my CryptPad i get this kind of error:


HTTP Strict Transport Security: false

HTTP Public Key Pinning: false



for cryptpad it is important have let’s encrypt ssl for the 2 domains https://cryptpad.domain.tld and https://sandbox-cryptpad.domain.tld

1 Like

right - did set it that up a while ago but did not check again

while trying to renew the certificate i get this:
https://paste.yunohost.org/raw/gupimijibe

this problem seams so not responsible for the unstable sever and the db crashing

i noticed that even after an reboot

the uptime notice for services shown under

/yunohost/admin/#/services

do not update… they still show “running for X days” even the server was just new started

So I think you need force renew the certificate !! It’s very important, cryptpad can not works without it !

Try this

yunohost domain cert renew sandbox-domain2.tld --force
root@my:~# yunohost domain cert renew sandbox-domain2.tld  --force                                                                  
Error: The certificate for the domain 'sandbox-domain2.tld ' is not issued by Let's Encrypt. Cannot renew it automatically!
root@my:~# yunohost domain cert install sandbox-domain2.tld --force                                                                
Error: The DNS records for domain 'sandbox-domain2.tld' are different to this server's IP. Please check the 'DNS records' (basic) c
ategory in the diagnosis for more info. If you recently modified your A record, please wait for it to propagate (some DNS propagation 
checkers are available online). (If you know what you are doing, use '--no-checks' to turn off these checks.)                         
root@my:~#                                                                                                                            

Have you add the dns entries A and AAA in your registar for sandbox-domain2.tld ? it seems to be the problem, related in the diagnosis ?

yunohost tools run diagnosis
yunohost domain dns suggest sandbox-domain2.tld

Cryptpad must be on a dedicated domain, it can be on something like cryptpad.domain.tld and it create or must have a domain sandbox sandbox-cryptpad.domain.tld. I say that because it is strange to see a domain2.tld ?

facepalm
you are right the record was missing…

i set the record and i hope this will help…

meanwhile mysql is again down

this are the last entries i see in the logs:

 journalctl

-- Journal begins at Fri 2023-06-23 09:41:47 BST, ends at Mon 2024-09-16 15:16:18 BST. --
Sep 16 14:53:42 mariadbd[14801]: We think the query pointer is invalid, but we will try to print it anyway.
Sep 16 14:53:42 mariadbd[14801]: Query:
Sep 16 14:53:42 mariadbd[14801]: Writing a core file...
Sep 16 14:53:42 mariadbd[14801]: Working directory at /var/lib/mysql
Sep 16 14:53:42 mariadbd[14801]: Resource Limits:
Sep 16 14:53:42 mariadbd[14801]: Limit                     Soft Limit           Hard Limit           Units
Sep 16 14:53:42 mariadbd[14801]: Max cpu time              unlimited            unlimited            seconds
Sep 16 14:53:42 mariadbd[14801]: Max file size             unlimited            unlimited            bytes
Sep 16 14:53:42 mariadbd[14801]: Max data size             unlimited            unlimited            bytes
Sep 16 14:53:42 mariadbd[14801]: Max stack size            8388608              unlimited            bytes
Sep 16 14:53:42 mariadbd[14801]: Max core file size        0                    unlimited            bytes
Sep 16 14:53:42 mariadbd[14801]: Max resident set          unlimited            unlimited            bytes
Sep 16 14:53:42 mariadbd[14801]: Max processes             29185                29185                processes
Sep 16 14:53:42 mariadbd[14801]: Max open files            32768                32768                files
Sep 16 14:53:42 mariadbd[14801]: Max locked memory         8388608              8388608              bytes
Sep 16 14:53:42 mariadbd[14801]: Max address space         unlimited            unlimited            bytes
Sep 16 14:53:42 mariadbd[14801]: Max file locks            unlimited            unlimited            locks
Sep 16 14:53:42 mariadbd[14801]: Max pending signals       29185                29185                signals
Sep 16 14:53:42 mariadbd[14801]: Max msgqueue size         819200               819200               bytes
Sep 16 14:53:42 mariadbd[14801]: Max nice priority         0                    0
Sep 16 14:53:42 mariadbd[14801]: Max realtime priority     0                    0
Sep 16 14:53:42 mariadbd[14801]: Max realtime timeout      unlimited            unlimited            us
Sep 16 14:53:42 mariadbd[14801]: Core pattern: core
Sep 16 14:53:42 mariadbd[14801]: Kernel version: Linux version 6.1.21-v8+ (dom@buildbot) (aarch64-linux-gnu-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023
Sep 16 14:53:42 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
Sep 16 14:53:42 systemd[1]: mariadb.service: Failed with result 'signal'.
Sep 16 14:53:42 systemd[1]: mariadb.service: Consumed 1.433s CPU time.
Sep 16 14:53:47 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 110.
Sep 16 14:53:47 systemd[1]: Stopped MariaDB 10.5.23 database server.
Sep 16 14:53:47 systemd[1]: mariadb.service: Consumed 1.433s CPU time.
Sep 16 14:53:47 systemd[1]: Starting MariaDB 10.5.23 database server...
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [Note] Starting MariaDB 10.5.23-MariaDB-0+deb11u1 source revision 6cfd2ba397b0ca689d8ff1bdb9fc4a4dc516a5eb as process 15526
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [Note] InnoDB: Uses event mutexes
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [Note] InnoDB: Number of pools: 1
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [Note] InnoDB: Using ARMv8 crc32 instructions
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [Note] InnoDB: Using Linux native AIO
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [Note] InnoDB: Completed initialization of buffer pool
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=270632462823,270632462823
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [ERROR] InnoDB: Missing FILE_CHECKPOINT between the checkpoint 270632462823 and the end 270634887680.
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [Note] InnoDB: Starting shutdown...
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [Note] Plugin 'FEEDBACK' is disabled.
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [ERROR] Unknown/unsupported storage engine: InnoDB
Sep 16 14:53:47 mariadbd[15526]: 2024-09-16 14:53:47 0 [ERROR] Aborting
Sep 16 14:53:48 systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Sep 16 14:53:48 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Sep 16 14:53:48 systemd[1]: Failed to start MariaDB 10.5.23 database server.

Perhaps a problem of space on the disk / ?? Or a corrupt DB ? Or a problem of memory swap, perhaps you need add more swap…

what returns

free -h

Perhaps you can try reboot also…

root@my:~# free -h                                                                                                                                                      
               total        used        free      shared  buff/cache   available                                                                                        
Mem:           7.6Gi       1.1Gi        89Mi        31Mi       6.4Gi       6.3Gi                                                                                        
Swap:           99Mi          0B        99Mi                                                                                                                            
root@my:~# df                                                                                                                                                           
Filesystem     1K-blocks      Used Available Use% Mounted on                                                                                                            
/dev/root      122653120 110209152   7424932  94% /                                                                                                                     
devtmpfs         3735680         0   3735680   0% /dev                                                                                                                  
tmpfs            3999904       100   3999804   1% /dev/shm                                                                                                              
tmpfs            1599964       940   1599024   1% /run                                                                                                                  
tmpfs               5120         4      5116   1% /run/lock                                                                                                             
/dev/mmcblk0p1    258095     31156    226940  13% /boot                                                                                                                 
tmpfs             799980         0    799980   0% /run/user/0                                                                                                           
root@my:~#                                                                                                                                                              

after a reboot is works for some time and than i goes down again

I think the problem is here, you need add place somewhere… for a raspberry, is better have a external disk and mount for exemple all the /home disk on external storage or all the /

If you keep all on a SD card, all the writes on the card Disk stays and after a while you can’t use it no more…

There are a lot of topics related how-to use external disk on a raspberry here.

You can also first try doing place on your \ folder… This command delete a lot of logs and clean system

yunohost tools basic-space-cleanup

Perhaps also a good idea on a raspberry is to edit the /etc/fstab to keep the temporally writes on the ram, not on the disk, adding this lines on the file /etc/fstab

tmpfs /tmp tmpfs defaults,noatime 0 0
tmpfs /var/log tmpfs defaults,noatime,size=16m 0 0

and perhaps reboot or do a mount -a

Here some documentation about https://pip.raspberrypi.com/categories/685/documents/RP-003610-WP/RP-003610-WP-1.pdf

the storage seams still fine to me and
YH is running for year now on the raspy … it must be something else

have a look here
Firefox_Screenshot_2024-09-16T16-00-12.005Z

even so i rebooted the system 19 minutes ago it says that e.g. nginx is running since 3 days… something is not updating

root@my:~# df                                                                                                                                   
Filesystem     1K-blocks      Used Available Use% Mounted on                                                                                    
/dev/root      122653120 108295504   9338580  93% /                                                                                             
devtmpfs         3735680         0   3735680   0% /dev                                                                                          
tmpfs            3999904        52   3999852   1% /dev/shm                                                                                      
tmpfs            1599964       940   1599024   1% /run                                                                                          
tmpfs               5120         4      5116   1% /run/lock                                                                                     
/dev/mmcblk0p1    258095     31156    226940  13% /boot                                                                                         
tmpfs             799980         0    799980   0% /run/user/0                                                                                   
root@my:~#

so i deleted same files but it does not make a difference - mysql goes down

So did all what you suggested …
but after some minutes the DB goes down again…
Firefox_Screenshot_2024-09-19T08-23-33.147Z

do you have an idea what else i could try?

first reboot did not work and i got this:

"500"

Aktion: "PUT" /yunohost/api/reboot?force

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/moulinette/interfaces/api.py", line 450, in process
    ret = self.actionsmap.process(arguments, timeout=30, route=_route)
  File "/usr/lib/python3/dist-packages/moulinette/actionsmap.py", line 574, in process
    return func(**arguments)
  File "/usr/lib/python3/dist-packages/yunohost/log.py", line 483, in func_wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/yunohost/tools.py", line 566, in tools_reboot
    operation_logger.start()
  File "/usr/lib/python3/dist-packages/yunohost/log.py", line 630, in start
    self.flush()
  File "/usr/lib/python3/dist-packages/yunohost/log.py", line 684, in flush
    with open(self.md_path, "w") as outfile:
OSError: [Errno 5] Input/output error: '/var/log/yunohost/categories/operation/20240919-071642-tools_reboot.yml'

does this hint anywhere?