My YunoHost server
Hardware: physical server
YunoHost version: 4.0.1
I have access to my server : Through SSH and direct access via keyboard / screen.
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no
Description of my issue
I copied a bunch of data to my YNH server and filled up the drive. The volume got stuck and i had to manually unmount it, fsck it, and delete a bunch of stuff (the data I copied over, rm -rf /var/ltmp/) in order to be able to log into my window manager again.
Some YNH services now start, but many of the important ones aren’t running:
summary: Service fail2ban is failed
summary: Service funkwhale-worker is failed
summary: Service haste is failed
summary: Service nginx is failed
summary: Service php7.3-fpm is failed
summary: Service redis-server is failed
summary: Service rspamd is failed
summary: Service uwsgi is failed
I tried doing a regen-conf with no effect that I could see.
Here are my logs from the regen-conf: https://paste.yunohost.org/raw/higadekuxu
Here is my nginx debug log: https://paste.yunohost.org/tuzeqipiva.sql
You use a graphic environment with windows on your ynh server ?
Nginx log says
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (2: No such file or directory)
So apparently somehow maybe you deleted
/var/log entirely or something…
@ljf correct. i’m not by any means familiar with linux or the command line and find it easier to ‘see’ where my folders are.
@Aleks I must have. is there any way to fix it or do i just need to reinstall?
I guess you could try to
mkdir -p /var/log
chown root:root /var/log
chmod 755 /var/log
@Aleks i spoke too soon - /var/log does have stuff in it.
ls -lh --group-directories-first
drwxr-xr-x 2 root root 4.0K Dec 15 22:58 apt
drwxr-x--- 2 root lp 4.0K Dec 16 00:00 cups
drwxr-s--- 2 mysql adm 4.0K Dec 15 22:56 mysql
drwxr-xr-x 2 pihole pihole 4.0K Dec 15 22:30 pihole
drwxrwxr-t 2 root postgres 4.0K Dec 15 22:30 postgresql
drwx------ 2 root root 4.0K Dec 15 22:30 private
drwxr-s--- 2 tomcat adm 4.0K Dec 16 07:35 tomcat9
drwxr-xr-x 2 root root 4.0K Dec 15 22:30 unattended-upgrades
drwxr-x--- 3 root root 4.0K Dec 15 22:39 yunohost
-rw-r--r-- 1 root root 1.1K Dec 15 23:03 alternatives.log
-rw-r----- 1 root adm 1.5M Dec 16 09:22 auth.log
-rw------- 1 root root 6.8K Dec 15 22:33 boot.log
-rw-rw---- 1 root utmp 1.6M Dec 16 09:22 btmp
-rw-r----- 1 root adm 36M Dec 16 09:22 daemon.log
-rw-r----- 1 root adm 94K Dec 16 09:15 debug
-rw-r--r-- 1 root root 70K Dec 15 22:59 dpkg.log
-rw-r----- 1 root adm 237K Dec 16 07:09 kern.log
-rw-rw-r-- 1 root utmp 286K Dec 16 09:20 lastlog
-rw-r----- 1 root adm 90K Dec 16 09:22 mail.info
-rw-r----- 1 root adm 90K Dec 16 09:22 mail.log
-rw-r----- 1 root adm 20K Dec 16 09:19 mail.warn
-rw-r----- 1 root adm 232K Dec 16 09:18 messages
-rw------- 1 root root 3.3K Dec 16 07:16 php7.3-fpm.log
-rw-r--r-- 1 pihole pihole 3.1K Dec 16 08:05 pihole-FTL.log
-rw-r--r-- 1 pihole pihole 4.6K Dec 16 00:00 pihole-FTL.log.1
-rw-r--r-- 1 dnsmasq root 1.9M Dec 16 09:22 pihole.log
-rw-r--r-- 1 dnsmasq root 714K Dec 16 00:00 pihole.log.1
-rw-r--r-- 1 sddm sddm 0 Dec 15 22:57 sddm.log
-rw-r----- 1 root adm 31M Dec 16 09:22 syslog
-rw-r----- 1 root adm 6.0M Dec 16 00:00 syslog.1
-rw-r----- 1 root adm 87K Dec 16 09:18 user.log
-rw-rw-r-- 1 root utmp 6.4K Dec 16 09:20 wtmp
-rw-r--r-- 1 root root 45K Dec 15 22:37 Xorg.0.log
-rw-r--r-- 1 root root 38K Dec 15 22:32 Xorg.0.log.old
Then sounds like the nginx folder disappeared for some reason …
mkdir -p /var/log/nginx
chown root:adm /var/log/nginx
chmod 755 /var/log/nginx
That worked to get nginx started and me back into the YNH portal.
I’m seeing a lot of things like these in the logs for the services that are failing though:
-- The process' exit code is 'exited' and its exit status is 1.
Dec 16 10:45:56 systemd: funkwhale-worker.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
More logs here:
uwsgi logs: https://paste.yunohost.org/uhonoyikob
Redis logs: https://paste.yunohost.org/usumefejam
fail2ban logs: https://paste.yunohost.org/roqaqaxoha
Matrix-synapse logs: https://paste.yunohost.org/pakugixexi
Funkwhale-worker logs: https://paste.yunohost.org/izezugibix
Dec 16 10:36:39 redis-server: >>> 'logfile /var/log/redis/redis-server.log'
Dec 16 10:36:39 redis-server: Can't open the log file: No such file or directory
You probably need to recreate /var/log/redis … i don’t know what the permissions are supposed to be.
Dec 16 10:37:07 fail2ban-server: No file(s) found for glob /var/log/fail2ban.log
You need to
FileNotFoundError: [Errno 2] No such file or directory: '/var/log/matrix-synapse/homeserver.log'
You probably need to recreate /var/log/matrix-synapse … i don’t know what the permissions are supposed to be.
FileNotFoundError: [Errno 2] No such file or directory: '/var/log/funkwhale/worker.log'
You probably need to recreate /var/log/funkwhale … i don’t know what the permissions are supposed to be.
@Aleks, thank you! This helped! I’ve got everything started now except
LSB: Start/stop uWSGI server instance(s)
uwsgi.service - LSB: Start/stop uWSGI server instance(s)
Loaded: loaded (/etc/init.d/uwsgi; generated)
Active: failed (Result: exit-code) since Wed 2020-12-16 12:04:54 EST; 10min ago
Process: 1435 ExecStart=/etc/init.d/uwsgi start (code=exited, status=2)
If I’m doing this right, i see that something’s running?
ps aux | grep uwsgi
servera+ 2875 0.0 0.0 6084 828 pts/1 S+ 12:19 0:00 grep uwsgi
No really … when you run
ps | grep, the grep command usually shows up in the result of itself … because
grep is started before
ps shows the
grep … (yeah)
I don’t know much about what’s going on for uwsgi … this service has a tendency to not tell shit about what’s going on :
uwsgi: Starting app server(s): uwsgi -> ! ! failed!
Though i suspect uwsgi is not the real app/service ? Could you know what app needs uwsgi ?
@Aleks doing some digging around I think I see that Nextcloud (which I have installed) uses this - https://wiki.archlinux.org/index.php/Nextcloud
I noticed that the background cron job function in NextCloud was non-functional, so I reinstalled the app. This fixed the cron job, though I notice that webdav is broken.
I have no idea if anything else is using this, or how to check, sorry.
I’m apparently missing more than I thought - directories that I certainly never deleted are missing even after reinstalling applications like Nextcloud and OnlyOffice. The OS seems to be struggling too. I had no idea that running out of disk space and trying to free it up could cause so many issues. I don’t know what else to do, so I’m going to reinstall Debian / YNH from scratch.
This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.