Hardware: Old laptop or computer YunoHost version: 4.3.5 I have access to my server : Through SSH and webadmin. I also have direct physical access to the server if needed. Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no
Description of my issue
My Nextcloud server is inaccessible. When I attempt to visit it in my browser, I’m met with this error message:
Internal Server Error
The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.
If I look in the server log, I see four messages. They’re quite long, but the relevant parts of the messages appear to be, in order:
“Could not boot files_trashbin: Could not resolve trashManager! Class trashManager does not exist”
“Could not boot files_versions: Could not resolve OCA\Files_Versions\Versions\IVersionManager! Class can not be instantiated”
“Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] Connection refused”
“An exception occurred while executing a query: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away”
Additionally, if I attempt to navigate to my WordPress installation, I’m met with a page that reports “Error establishing a database connection”.
Following the instructions on this StackOverflow question, I increased the value of max_allowed_packet in /etc/mysql/my.cnf and /etc/mysql/mariadb.cnf to 500M (from 50M), and have run sudo systemctl restart mysql.
No change. Both Nextcloud and WordPress still display the same error pages when I attempt to visit them.
How old is your laptop or computer, and what are the general specifications?
How high is the load on the system?
About how many users are on the system at a time?
Did you try, just to be sure, an fsck on the filesystem?
I ask because I helped multiple people running Yunohost with (at least) Nextcloud, Wordpress and Matrix installed, on various hardware, but never encountered the ongoing problems that unfortunately plague your installation.
Sure thing. I’ve replaced my YunoHost domain in these log files with “example.com”.
lines 1325-1347/1347 (END)
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit mariadb.service has exited.
--
-- The process' exit code is 'killed' and its exit status is 6.
Jan 08 13:51:26 example.com systemd[1]: mariadb.service: Failed with result 'signal'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit mariadb.service has entered the 'failed' state with result 'signal'.
Jan 08 13:51:26 example.com systemd[1]: Failed to start MariaDB 10.3.31 database server.
-- Subject: A start job for unit mariadb.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit mariadb.service has finished with a failure.
--
-- The job identifier is 1366946 and the job result is failed.
Jan 08 13:51:26 example.com sudo[31080]: admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/bin/journalctl -xe
Jan 08 13:51:26 example.com sudo[31080]: pam_unix(sudo:session): session opened for user root by admin(uid=0)
Here’s the results of sudo journalctl -u mariadb:
Jan 08 13:24:14 example.com mysqld[15771]: 2022-01-08 13:24:14 0 [Note] InnoDB: Starting final batch to recover 71 pages from redo log.
Jan 08 13:24:15 example.com mysqld[15771]: 2022-01-08 13:24:15 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
Jan 08 13:24:15 example.com mysqld[15771]: 2022-01-08 13:24:15 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
Jan 08 13:24:15 example.com mysqld[15771]: 2022-01-08 13:24:15 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Jan 08 13:24:15 example.com mysqld[15771]: 2022-01-08 13:24:15 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ..
Jan 08 13:24:15 example.com mysqld[15771]: 2022-01-08 13:24:15 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Jan 08 13:24:15 example.com mysqld[15771]: 2022-01-08 13:24:15 0 [Note] InnoDB: Waiting for purge to start
Jan 08 13:24:15 example.com mysqld[15771]: 2022-01-08 13:24:15 0 [Note] InnoDB: 10.3.31 started; log sequence number 36781582820; transaction id 326687783
Jan 08 13:24:15 example.com mysqld[15771]: 2022-01-08 13:24:15 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Jan 08 13:24:15 example.com mysqld[15771]: 2022-01-08 13:24:15 0 [Note] Plugin 'FEEDBACK' is disabled.
Jan 08 13:24:15 example.com mysqld[15771]: 2022-01-08 13:24:15 0 [Note] Recovering after a crash using tc.log
Jan 08 13:24:15 example.com mysqld[15771]: 2022-01-08 13:24:15 0 [Note] Starting crash recovery...
Jan 08 13:24:15 example.com mysqld[15771]: 2022-01-08 13:24:15 0 [Note] Crash recovery finished.
Jan 08 13:24:15 example.com mysqld[15771]: 2022-01-08 13:24:15 0 [Note] InnoDB: Buffer pool(s) load completed at 220108 13:24:15
Jan 08 13:24:18 example.com mysqld[15771]: 2022-01-08 13:24:18 2 [Warning] InnoDB: 16384 bytes should have been read. Only 4096 bytes read. Retrying for the remaining
Jan 08 13:24:21 example.com mysqld[15771]: 2022-01-08 13:24:21 2 [Warning] InnoDB: Retry attempts for reading partial data failed.
Jan 08 13:24:21 example.com mysqld[15771]: 2022-01-08 13:24:21 2 [ERROR] InnoDB: Tried to read 16384 bytes at offset 491520, but was only able to read 4096
Jan 08 13:24:21 example.com mysqld[15771]: 2022-01-08 13:24:21 2 [ERROR] InnoDB: Operating system error number 5 in a file operation.
Jan 08 13:24:21 example.com mysqld[15771]: 2022-01-08 13:24:21 2 [ERROR] InnoDB: Error number 5 means 'Input/output error'
Jan 08 13:24:21 example.com mysqld[15771]: 2022-01-08 13:24:21 2 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/
Jan 08 13:24:21 example.com mysqld[15771]: 2022-01-08 13:24:21 2 [ERROR] InnoDB: File (unknown): 'read' returned OS error 205. Cannot continue operation
Jan 08 13:24:21 example.com mysqld[15771]: 220108 13:24:21 [ERROR] mysqld got signal 6 ;
Jan 08 13:24:21 example.com mysqld[15771]: This could be because you hit a bug. It is also possible that this binary
Jan 08 13:24:21 example.com mysqld[15771]: or one of the libraries it was linked against is corrupt, improperly built,
Jan 08 13:24:21 example.com mysqld[15771]: or misconfigured. This error can also be caused by malfunctioning hardware.
Jan 08 13:24:21 example.com mysqld[15771]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Jan 08 13:24:21 example.com mysqld[15771]: We will try our best to scrape up some info that will hopefully help
Jan 08 13:24:21 example.com mysqld[15771]: diagnose the problem, but since we have already crashed,
Jan 08 13:24:21 example.com mysqld[15771]: something is definitely wrong and this may fail.
Jan 08 13:24:21 example.com mysqld[15771]: Server version: 10.3.31-MariaDB-0+deb10u1
Jan 08 13:24:21 example.com mysqld[15771]: key_buffer_size=16384
Jan 08 13:24:21 example.com mysqld[15771]: read_buffer_size=262144
Jan 08 13:24:21 example.com mysqld[15771]: max_used_connections=0
Jan 08 13:24:21 example.com mysqld[15771]: max_threads=153
Jan 08 13:24:21 example.com mysqld[15771]: thread_count=5
Jan 08 13:24:21 example.com mysqld[15771]: It is possible that mysqld could use up to
Jan 08 13:24:21 example.com mysqld[15771]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 355956 K bytes of memory
Jan 08 13:24:21 example.com mysqld[15771]: Hope that's ok; if not, decrease some variables in the equation.
Jan 08 13:24:21 example.com mysqld[15771]: Thread pointer: 0x7f9240000c08
Jan 08 13:24:21 example.com mysqld[15771]: Attempting backtrace. You can use the following information to find out
Jan 08 13:24:21 example.com mysqld[15771]: where mysqld died. If you see no messages after this, something went
Jan 08 13:24:21 example.com mysqld[15771]: terribly wrong...
Jan 08 13:24:21 example.com mysqld[15771]: stack_bottom = 0x7f925affcd58 thread_stack 0x20000
Jan 08 13:24:21 example.com mysqld[15771]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563afd317cee]
A similar check for mysql notes in the journal turned up nothing:
$ sudo journalctl -u mysql
-- Logs begin at Sat 2022-01-08 13:24:14 PST, end at Sat 2022-01-08 13:53:45 PST. --
-- No entries --
I’m using an old PC that’s got an Intel Core i3-2120 @ 3.30 GHz processor, and 4GB of 1333 MHz RAM. There’s usually only 1-2 users on at a time.
At your suggestion, I ran fsck on the main drive. It turned up nothing:
How long has your server been running, how much time have you invested in getting everything set up as it is?
I am not so much a fan of starting from in the face of trouble instead of finding a root cause, but it seems you are having problems that should not arise normally.
If you have some spare hardware, you might copy your current installation there (‘off line’, so to speak) and see if the problems persist. Back up system and apps via the admin interface, start with a clean installation on your spare computer, and run the ‘restore from back up’ instead of post-install.
Keep your current server online while testing. If the problem does not arise on the temporary server, you can decide whether you want to compare configurations and versions versus reinstalling your main server. The comparison will of course show the files you know you changed, maybe there’s more. Reinstalling the server you can then do with hardly any trouble or downtime by restoring a current back up on the temporary server, and make that your main server for a couple of days before doing the trick the other way around.
I am unfortunately not fluent enough in MySQL and PHP to recognize the possible causes of the errors mentioned in the log. SearX helped me on the way finding a post by someone who resolved your current problem, with the trashbinclass. Does it help you?
It looks like I fsck'd the wrong drive. The root drive is failing.
This is also a hell of a time to realize that I don’t have backups. I thought I’d have some automatic backups from recent system upgrades, but apparently not!
