Lots of services not starting after I randomly deleted a bunch of things

pl0 · December 16, 2020, 4:38am

My YunoHost server

Hardware: physical server
YunoHost version: 4.0.1
I have access to my server : Through SSH and direct access via keyboard / screen.
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no

Description of my issue

I copied a bunch of data to my YNH server and filled up the drive. The volume got stuck and i had to manually unmount it, fsck it, and delete a bunch of stuff (the data I copied over, rm -rf /var/ltmp/) in order to be able to log into my window manager again.

Some YNH services now start, but many of the important ones aren’t running:

summary: Service fail2ban is failed
summary: Service funkwhale-worker is failed
summary: Service haste is failed
summary: Service nginx is failed
summary: Service php7.3-fpm is failed
summary: Service redis-server is failed
summary: Service rspamd is failed
summary: Service uwsgi is failed

I tried doing a regen-conf with no effect that I could see.

Here are my logs from the regen-conf: https://paste.yunohost.org/raw/higadekuxu

Here is my nginx debug log: hastebin

ljf · December 16, 2020, 10:57am

You use a graphic environment with windows on your ynh server ?

Aleks · December 16, 2020, 1:39pm

Nginx log says

nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (2: No such file or directory)

So apparently somehow maybe you deleted /var/log entirely or something…

pl0 · December 16, 2020, 1:50pm

@ljf correct. i’m not by any means familiar with linux or the command line and find it easier to ‘see’ where my folders are.

@Aleks I must have. is there any way to fix it or do i just need to reinstall?

Aleks · December 16, 2020, 2:06pm

I guess you could try to

mkdir -p /var/log
chown root:root /var/log
chmod 755 /var/log

pl0 · December 16, 2020, 2:24pm

@Aleks i spoke too soon - /var/log does have stuff in it.

  ls -lh --group-directories-first
total 79M
drwxr-xr-x 2 root    root     4.0K Dec 15 22:58 apt
drwxr-x--- 2 root    lp       4.0K Dec 16 00:00 cups
drwxr-s--- 2 mysql   adm      4.0K Dec 15 22:56 mysql
drwxr-xr-x 2 pihole  pihole   4.0K Dec 15 22:30 pihole
drwxrwxr-t 2 root    postgres 4.0K Dec 15 22:30 postgresql
drwx------ 2 root    root     4.0K Dec 15 22:30 private
drwxr-s--- 2 tomcat  adm      4.0K Dec 16 07:35 tomcat9
drwxr-xr-x 2 root    root     4.0K Dec 15 22:30 unattended-upgrades
drwxr-x--- 3 root    root     4.0K Dec 15 22:39 yunohost
-rw-r--r-- 1 root    root     1.1K Dec 15 23:03 alternatives.log
-rw-r----- 1 root    adm      1.5M Dec 16 09:22 auth.log
-rw------- 1 root    root     6.8K Dec 15 22:33 boot.log
-rw-rw---- 1 root    utmp     1.6M Dec 16 09:22 btmp
-rw-r----- 1 root    adm       36M Dec 16 09:22 daemon.log
-rw-r----- 1 root    adm       94K Dec 16 09:15 debug
-rw-r--r-- 1 root    root      70K Dec 15 22:59 dpkg.log
-rw-r----- 1 root    adm      237K Dec 16 07:09 kern.log
-rw-rw-r-- 1 root    utmp     286K Dec 16 09:20 lastlog
-rw-r----- 1 root    adm       90K Dec 16 09:22 mail.info
-rw-r----- 1 root    adm       90K Dec 16 09:22 mail.log
-rw-r----- 1 root    adm       20K Dec 16 09:19 mail.warn
-rw-r----- 1 root    adm      232K Dec 16 09:18 messages
-rw------- 1 root    root     3.3K Dec 16 07:16 php7.3-fpm.log
-rw-r--r-- 1 pihole  pihole   3.1K Dec 16 08:05 pihole-FTL.log
-rw-r--r-- 1 pihole  pihole   4.6K Dec 16 00:00 pihole-FTL.log.1
-rw-r--r-- 1 dnsmasq root     1.9M Dec 16 09:22 pihole.log
-rw-r--r-- 1 dnsmasq root     714K Dec 16 00:00 pihole.log.1
-rw-r--r-- 1 sddm    sddm        0 Dec 15 22:57 sddm.log
-rw-r----- 1 root    adm       31M Dec 16 09:22 syslog
-rw-r----- 1 root    adm      6.0M Dec 16 00:00 syslog.1
-rw-r----- 1 root    adm       87K Dec 16 09:18 user.log
-rw-rw-r-- 1 root    utmp     6.4K Dec 16 09:20 wtmp
-rw-r--r-- 1 root    root      45K Dec 15 22:37 Xorg.0.log
-rw-r--r-- 1 root    root      38K Dec 15 22:32 Xorg.0.log.old

Aleks · December 16, 2020, 3:03pm

Then sounds like the nginx folder disappeared for some reason …

mkdir -p /var/log/nginx
chown root:adm /var/log/nginx
chmod 755 /var/log/nginx

pl0 · December 16, 2020, 3:49pm

That worked to get nginx started and me back into the YNH portal.

I’m seeing a lot of things like these in the logs for the services that are failing though:

-- The process' exit code is 'exited' and its exit status is 1.
Dec 16 10:45:56 systemd[1]: funkwhale-worker.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--

More logs here:

uwsgi logs: https://paste.yunohost.org/uhonoyikob
Redis logs: https://paste.yunohost.org/usumefejam
fail2ban logs: https://paste.yunohost.org/roqaqaxoha
Matrix-synapse logs: https://paste.yunohost.org/pakugixexi
Funkwhale-worker logs: https://paste.yunohost.org/izezugibix

Aleks · December 16, 2020, 4:01pm

Dec 16 10:36:39 redis-server[9808]: >>> 'logfile /var/log/redis/redis-server.log'
Dec 16 10:36:39 redis-server[9808]: Can't open the log file: No such file or directory

You probably need to recreate /var/log/redis … i don’t know what the permissions are supposed to be.

Dec 16 10:37:07 fail2ban-server[9910]:  No file(s) found for glob /var/log/fail2ban.log

You need to touch /var/log/fail2ban.log

FileNotFoundError: [Errno 2] No such file or directory: '/var/log/matrix-synapse/homeserver.log'

You probably need to recreate /var/log/matrix-synapse … i don’t know what the permissions are supposed to be.

 FileNotFoundError: [Errno 2] No such file or directory: '/var/log/funkwhale/worker.log'

You probably need to recreate /var/log/funkwhale … i don’t know what the permissions are supposed to be.

pl0 · December 16, 2020, 5:20pm

@Aleks, thank you! This helped! I’ve got everything started now except

uwsgi

Start
Description
LSB: Start/stop uWSGI server instance(s)

https://paste.yunohost.org/hodakuteno

uwsgi.service - LSB: Start/stop uWSGI server instance(s)
   Loaded: loaded (/etc/init.d/uwsgi; generated)
   Active: failed (Result: exit-code) since Wed 2020-12-16 12:04:54 EST; 10min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1435 ExecStart=/etc/init.d/uwsgi start (code=exited, status=2)

If I’m doing this right, i see that something’s running?

ps aux | grep uwsgi
servera+  2875  0.0  0.0   6084   828 pts/1    S+   12:19   0:00 grep uwsgi

Aleks · December 16, 2020, 5:42pm

No really … when you run ps | grep, the grep command usually shows up in the result of itself … because grep is started before ps, hence ps shows the grep … (yeah)

I don’t know much about what’s going on for uwsgi … this service has a tendency to not tell shit about what’s going on :

uwsgi[11378]: Starting app server(s): uwsgi -> ! ! failed!

Though i suspect uwsgi is not the real app/service ? Could you know what app needs uwsgi ?

pl0 · December 16, 2020, 7:34pm

@Aleks doing some digging around I think I see that Nextcloud (which I have installed) uses this - https://wiki.archlinux.org/index.php/Nextcloud

I noticed that the background cron job function in NextCloud was non-functional, so I reinstalled the app. This fixed the cron job, though I notice that webdav is broken.

I have no idea if anything else is using this, or how to check, sorry.

pl0 · December 16, 2020, 11:35pm

I’m apparently missing more than I thought - directories that I certainly never deleted are missing even after reinstalling applications like Nextcloud and OnlyOffice. The OS seems to be struggling too. I had no idea that running out of disk space and trying to free it up could cause so many issues. I don’t know what else to do, so I’m going to reinstall Debian / YNH from scratch.

system · December 31, 2020, 11:35pm

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.