Collabora: Daemon.log and syslog suddenly exploding in size to many GBs

wasserkrug · September 21, 2024, 12:57pm

What type of hardware are you using: VPS bought online
What YunoHost version are you running: 11.2.30.2 (stable)
How are you able to access your server: The webadmin
SSH

Describe your issue

Yesterday night, all of a sudden, functions of my server stopped working. The diagnosis this morning alerted me that my storage “has only 0B (0%) space remaining.” I also saw an automatic diagnosis message stating that there were only 4% storage remaining last night at 19h00, so there must have been something that gradually slurped away my storage space at a very quick pace.

There was no obvious other error message available anywhere. I was still able to log in to my VPS via SSH and figured out that two files have swollen uncontrollably:

/var/log/daemon.log and
/var/log/syslog

are suddenly both each 35 GB big.

How do I proceed? Is there any way to find out why these files have gotten so enormously big? Which commands would I need to be able to inspect these files properly? Or should I just delete them and pray that it doesn’t happen again? Can I even delete them without breaking anything else?

Thank you very much for your help and have a good weekend!

Share relevant logs or error messages

jarod5001 · September 21, 2024, 3:23pm

Did you update collabora recently?

wasserkrug · September 21, 2024, 3:27pm

Hi! Yes, Collabora is currently installed in version 24.04.7.2~ynh1.

I checked the logs and I updated it on 17.09.2024; here is the corresponding update log: https://paste.yunohost.org/raw/momiboxipi

The other installed apps also include Nextcloud in version 29.0.7~ynh1, if that might be of any relevance? Thanks!

jarod5001 · September 21, 2024, 3:51pm

I also had my vps out of storage because of log filling space after I updated collabora to 24.

Try this to cleanup space
sudo yunohost tools basic-space-cleanup
And
sudo journalctl -f
The last command will show real-time logging

wasserkrug · September 21, 2024, 4:05pm

Thank you!

It seems to have been Collabora indeed: I did as you suggested; monitored via journalctl -f. Everything seemed ‘relatively’ peaceful; as soon as I manually started postgresql via the webadmin, coolwsd started spitting out a seemingly infinite number of lines of logs per second.

I now just swiftly uninstalled Collabora via the webadmin; can I now safely remove/delete the logfiles in any way? Or will a scheduled task take this over automatically?

jarod5001 · September 21, 2024, 4:08pm

Can you share the content of the log? I have deleted the logs and restored v22 of collabora as I had very low space on my vps and I needed the vps to be running correctly as I had collaborative work these days
sudo journalctl -xe

wasserkrug · September 21, 2024, 4:25pm

From what I could gather it’s essentially the following, endlessly on repeat:

coolwsd[174438]: Backtrace 174438 - kit startup of 24.04.7.2 d5ebff5:
coolwsd[174438]: /usr/bin/coolforkit-ns(_ZN7SigUtil13dumpBacktraceEv+0x73)[0x61c2d3]
coolwsd[174438]: /usr/bin/coolforkit-ns[0x61c6f5]
coolwsd[174438]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f622e216140]
coolwsd[174438]: [0x131c580]
coolwsd[377]: wsd-00377-01470 2024-09-21 17:58:57.333599 +0200 [ prisoner_poll ] ERR  #17: Read failed, have 0 buffered bytes (ECONNRESET: Connection reset by peer)| net/Socket.hpp:1156
coolwsd[377]: wsd-00377-01470 2024-09-21 17:58:57.333654 +0200 [ prisoner_poll ] WRN  #17: Unassociated Kit (174438) disconnected unexpectedly| wsd/COOLWSD.cpp:3726
coolwsd[1471]: frk-01471-01471 2024-09-21 17:58:57.338072 +0200 [ forkit ] ERR  Couldn't create file: /opt/cool/child-roots/377-0dc7c9f3/pCKRwB1qTciNTnxg/tmp/kit-crashed due to error: No such file or directory| kit/ForKit.cpp:300
coolwsd[1471]: frk-01471-01471 2024-09-21 17:58:57.338115 +0200 [ forkit ] WRN  Successfully sent 'segfaultcount' message segfaultcount 1
coolwsd[1471]: | kit/ForKit.cpp:343
coolwsd[174439]: kit-174439-174439 2024-09-21 17:58:57.339208 +0200 [ kit_spare_1c59b ] WRN  Failed to open /proc/self/smaps_rollup. Memory stats will be slower| kit/Kit.cpp:3287
coolwsd[174439]: kit-174439-174439 2024-09-21 17:58:57.340347 +0200 [ kit_spare_1c59b ] SIG   Fatal signal received: SIGSEGV code: 2 for address: 0x131c580
coolwsd[174439]: Recent activity:
coolwsd[174439]: Backtrace 174439 - kit startup of 24.04.7.2 d5ebff5:
coolwsd[174439]: /usr/bin/coolforkit-ns(_ZN7SigUtil13dumpBacktraceEv+0x73)[0x61c2d3]
coolwsd[174439]: /usr/bin/coolforkit-ns[0x61c6f5]
coolwsd[174439]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f622e216140]
coolwsd[174439]: [0x131c580]
coolwsd[377]: wsd-00377-01470 2024-09-21 17:58:57.345148 +0200 [ prisoner_poll ] ERR  #17: Read failed, have 0 buffered bytes (ECONNRESET: Connection reset by peer)| net/Socket.hpp:1156
coolwsd[377]: wsd-00377-01470 2024-09-21 17:58:57.345171 +0200 [ prisoner_poll ] WRN  #17: Unassociated Kit (174439) disconnected unexpectedly| wsd/COOLWSD.cpp:3726
coolwsd[1471]: frk-01471-01471 2024-09-21 17:58:57.350392 +0200 [ forkit ] ERR  Couldn't create file: /opt/cool/child-roots/377-0dc7c9f3/21mEXEMhGHFERpXt/tmp/kit-crashed due to error: No such file or directory| kit/ForKit.cpp:300
coolwsd[1471]: frk-01471-01471 2024-09-21 17:58:57.350429 +0200 [ forkit ] WRN  Successfully sent 'segfaultcount' message segfaultcount 1
coolwsd[1471]: | kit/ForKit.cpp:343
coolwsd[174440]: kit-174440-174440 2024-09-21 17:58:57.351377 +0200 [ kit_spare_1c59c ] WRN  Failed to open /proc/self/smaps_rollup. Memory stats will be slower| kit/Kit.cpp:3287
coolwsd[174440]: kit-174440-174440 2024-09-21 17:58:57.354088 +0200 [ kit_spare_1c59c ] SIG   Fatal signal received: SIGSEGV code: 2 for address: 0x131c580
coolwsd[174440]: Recent activity:
coolwsd[174440]: Backtrace 174440 - kit startup of 24.04.7.2 d5ebff5:
coolwsd[174440]: /usr/bin/coolforkit-ns(_ZN7SigUtil13dumpBacktraceEv+0x73)[0x61c2d3]
coolwsd[174440]: /usr/bin/coolforkit-ns[0x61c6f5]
coolwsd[174440]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f622e216140]
coolwsd[174440]: [0x131c580]
coolwsd[377]: wsd-00377-01470 2024-09-21 17:58:57.360976 +0200 [ prisoner_poll ] ERR  #17: Read failed, have 0 buffered bytes (ECONNRESET: Connection reset by peer)| net/Socket.hpp:1156
coolwsd[377]: wsd-00377-01470 2024-09-21 17:58:57.361013 +0200 [ prisoner_poll ] WRN  #17: Unassociated Kit (174440) disconnected unexpectedly| wsd/COOLWSD.cpp:3726
coolwsd[1471]: frk-01471-01471 2024-09-21 17:58:57.364199 +0200 [ forkit ] ERR  Couldn't create file: /opt/cool/child-roots/377-0dc7c9f3/8hfNcMwVrBxxYADm/tmp/kit-crashed due to error: No such file or directory| kit/ForKit.cpp:300
coolwsd[1471]: frk-01471-01471 2024-09-21 17:58:57.364242 +0200 [ forkit ] WRN  Successfully sent 'segfaultcount' message segfaultcount 1
coolwsd[1471]: | kit/ForKit.cpp:343
coolwsd[174441]: kit-174441-174441 2024-09-21 17:58:57.366683 +0200 [ kit_spare_1c59d ] WRN  Failed to open /proc/self/smaps_rollup. Memory stats will be slower| kit/Kit.cpp:3287
coolwsd[174441]: kit-174441-174441 2024-09-21 17:58:57.368863 +0200 [ kit_spare_1c59d ] SIG   Fatal signal received: SIGSEGV code: 2 for address: 0x131c580
coolwsd[174441]: Recent activity:
coolwsd[174441]: Backtrace 174441 - kit startup of 24.04.7.2 d5ebff5:
coolwsd[174441]: /usr/bin/coolforkit-ns(_ZN7SigUtil13dumpBacktraceEv+0x73)[0x61c2d3]
coolwsd[174441]: /usr/bin/coolforkit-ns[0x61c6f5]
coolwsd[174441]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f622e216140]
coolwsd[174441]: [0x131c580]
coolwsd[377]: wsd-00377-01470 2024-09-21 17:58:57.374753 +0200 [ prisoner_poll ] ERR  #17: Read failed, have 0 buffered bytes (ECONNRESET: Connection reset by peer)| net/Socket.hpp:1156
coolwsd[377]: wsd-00377-01470 2024-09-21 17:58:57.374793 +0200 [ prisoner_poll ] WRN  #17: Unassociated Kit (174441) disconnected unexpectedly| wsd/COOLWSD.cpp:3726
coolwsd[1471]: frk-01471-01471 2024-09-21 17:58:57.382906 +0200 [ forkit ] ERR  Couldn't create file: /opt/cool/child-roots/377-0dc7c9f3/6YjHpEw2A7bxFCqs/tmp/kit-crashed due to error: No such file or directory| kit/ForKit.cpp:300
coolwsd[1471]: frk-01471-01471 2024-09-21 17:58:57.382968 +0200 [ forkit ] WRN  Successfully sent 'segfaultcount' message segfaultcount 1
coolwsd[1471]: | kit/ForKit.cpp:343
coolwsd[174443]: kit-174443-174443 2024-09-21 17:58:57.384264 +0200 [ kit_spare_1c59e ] WRN  Failed to open /proc/self/smaps_rollup. Memory stats will be slower| kit/Kit.cpp:3287
coolwsd[174443]: kit-174443-174443 2024-09-21 17:58:57.385824 +0200 [ kit_spare_1c59e ] SIG   Fatal signal received: SIGSEGV code: 2 for address: 0x131c580
coolwsd[174443]: Recent activity:
coolwsd[174443]: Backtrace 174443 - kit startup of 24.04.7.2 d5ebff5:
coolwsd[174443]: /usr/bin/coolforkit-ns(_ZN7SigUtil13dumpBacktraceEv+0x73)[0x61c2d3]
coolwsd[174443]: /usr/bin/coolforkit-ns[0x61c6f5]
coolwsd[174443]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f622e216140]
coolwsd[174443]: [0x131c580]
coolwsd[377]: wsd-00377-01470 2024-09-21 17:58:57.391050 +0200 [ prisoner_poll ] ERR  #17: Read failed, have 0 buffered bytes (ECONNRESET: Connection reset by peer)| net/Socket.hpp:1156
coolwsd[377]: wsd-00377-01470 2024-09-21 17:58:57.391075 +0200 [ prisoner_poll ] WRN  #17: Unassociated Kit (174443) disconnected unexpectedly| wsd/COOLWSD.cpp:3726
coolwsd[174444]: kit-174444-174444 2024-09-21 17:58:57.396529 +0200 [ kit_spare_1c59f ] WRN  Failed to open /proc/self/smaps_rollup. Memory stats will be slower| kit/Kit.cpp:3287
coolwsd[174444]: kit-174444-174444 2024-09-21 17:58:57.398032 +0200 [ kit_spare_1c59f ] SIG   Fatal signal received: SIGSEGV code: 2 for address: 0x131c580

fprocess · September 21, 2024, 5:01pm

Hi, I had exactly the same problem this week.

I also had Collabora installed (I deleted it to save space)
here is the size of my log

— /var/log ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
9.0 GiB [##########] syslog
9.0 GiB [######### ] daemon.log
56.0 MiB [ ] /journal
21.0 MiB [ ] btmp
18.1 MiB [ ] /yunohost
6.9 MiB [ ] mail.log
6.9 MiB [ ] mail.info
6.3 MiB [ ] auth.log
5.4 MiB [ ] messages
5.4 MiB [ ] user.log
3.3 MiB [ ] /nginx
3.0 MiB [ ] mail.warn
2.7 MiB [ ] fail2ban.log
512.0 KiB [ ] cloud-init.log
440.0 KiB [ ] bootstrap.log
344.0 KiB [ ] /dolibarr
300.0 KiB [ ] yunohost-installation_20210605_160956.log
216.0 KiB [ ] /nextcloud
200.0 KiB [ ] debug
180.0 KiB [ ] dpkg.log

thanks in advance for the help

jarod5001 · September 21, 2024, 5:59pm

Did sudo yunohost tools basic-space-cleanup reduce the log folder size?

wasserkrug · September 21, 2024, 6:08pm

In my case it did reduce the overall log folder size, but it just cleaned up a lot of the older/archived files; the two big >35GB files are still there. (I ran it after the removal of Collabora a second time, that didn’t change anything).

fprocess · September 21, 2024, 6:17pm

this command delete 800 MB in /var/log, i still have 2 * 9GiB file log

jarod5001 · September 21, 2024, 6:25pm

OK, and sudo journalctl --vacuum-size=500M
This will reduce the log size to 500M, so you may change it to whatever you want. But depending on your situation, you may want to download the logs locally in case you need it before running this command.

fprocess · September 21, 2024, 7:08pm

It’s complicated to save logs in ssh (no root permission)
do you think there’s a risk of “crashing” Yunohost by executing this command?
or log backup are just for inspection?

jarod5001 · September 21, 2024, 7:38pm

Not crashing but you loose old entries.
Check the latest entries of both files with sudo tail filename

Edit : try rebooting the server and see if it releases some space

elgee · September 22, 2024, 4:21am

Thank you very much for this report and discussion! I ran into the same problem.

Following the recommendations found here

I removed the culprit (Collabora Online) and restored it from the pre-upgrade backup (22.05.14.3~ynh2).
Managed to save space by emptying the syslog by the following command:

sudo sh -c 'echo > /var/log/syslog'

And the demon.log with:

sudo sh -c 'echo > /var/log/daemon.log'

Things have returned to normal.

fprocess · September 22, 2024, 8:15am

I executed the command and I rebooted.
the logs syslog.1 and deamon.log still the same size as before

Lapineige · September 25, 2024, 6:24am

Same issue here, collabora going wrong

fprocess · September 30, 2024, 1:29pm

A quick update on the problem:
I do sudo journalctl --vacuum-size=500M
and few days later, the server automatically reduced the log size.

so everything is ok for me now!

thank you for the help

Luncheon3462 · October 8, 2024, 5:46pm

I am having same issue. I uninstalled the collobora app and downgraded to the previous version. I will monitor to see if it fixes the problem. thank you.

Does anyone know if collabora is aware or if there is a new version of the app for nextcloud being tested that fixes the issue?

OniriCorpe · October 8, 2024, 6:05pm

Moved the topic on the “support apps” section and renamed its title to include “collabora”