Nextcloud: Failing upgrade and backup attempts (php8.2-fpm.service) - Application broken/inacessible

[wasserkrug]

/ Nextcloud: Failing upgrade and backup attempts (php8.2-fpm.service)

My YunoHost server

Hardware: VPS (Strato)
YunoHost version: 11.2.10.3
I have access to my server : Through SSH & Webadmin
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : not to my knowledge
If your request is related to an app, specify its name and version: Nextcloud 28.0.1~ynh1

Description of my issue: Situation now

After a backup attempt, Nextcloud appears in the list of applications, but I cannot view the app details in the webadmin (it simply stays blank). The URL under which Nextcloud was accessible now points to the SSO page. Nextcloud does not appear as application tile.

Background

Yesterday, I tried to do some simple yunohost-maintenance and wanted to update the system. But now, ultimately my Nextcloud app is broken and I’ve run out of ideas how to fix it and would be very thankful for some help. How it came about:

1. I also had this exact problem and could fix it with the help provided in the topic - thanks a lot! After restarting the server, the problem did no longer appear in the diagnosis.

2. I tried updating my Nextcloud from 28.0.1~ynh1 to 28.0.1~ynh2. There seemed to be some problem with php8.2-fpm.service; there was a cycle where the PHP 8.2 FastCGI Process Manager was repeatedly started and stopped. This is the log.

3. After the failed attempt to update the Nextcloud app, I got the Internal Server Error message as described in this thread whenever trying to access Nextcloud. Nextcloud was no longer working, also my desktop and phone apps could not sync with Nextcloud any more.
   a. yunohost app upgrade nextcloud --force also did not help in my case.

4. This morning, I manually uninstalled Nextcloud and attempted to recover the backup I had saved from before the first update attempt. This also did not work. I no longer have the Internal Server Error, but Nextcloud does not even exist in the user interface any more (as described above under Situation now). Here is the log from the backup attempt. And here is the accompanying log " Provisioning/deprovisioning/updating a resource".

Additional information which might help: Both yesterday and today, this specific error was listed from journalctl -xe:

systemd[80739]: Failed to create /user.slice/user-50465.slice/user@50465.service/init.scope control group: Permission denied
systemd[80739]: Failed to allocate manager object: Permission denied

Do any of you have any idea on how to proceed in this case? I would be very thankful for any and all help! Best regards and thank you very much!

[wasserkrug]

Getting my server to respond to me again

So, in the middle of today my server then seemed to be completely down, and many attempts at regaining access via SSH didn’t even establish a connection, I felt a little bit like in a lottery.

One wild goose chase and a lot of sweat later, I am glad to have found something that approximates a solution.

For anyone running into the same issue I did, after many frustrating hours where my VPS was barely responsive and seemed to constantly even shut me out, here is how I got it back on track:

After rebooting the server the hard way via my provider, the command systemctl --failed showed me that there were two things apparently failing:

  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
● systemd-udev-trigger.service loaded failed failed Coldplug All udev Devices
● user@50465.service           loaded failed failed User Manager for UID 50465

which was probably also the reason why before, I couldn’t even enter a reboot command or switch to the root user or anything of the like.

With systemctl status I could inquire some more, but was not learning anything useful. The first magic bullet command:

sudo systemctl reset-failed

After this command, my Yunohost-server started responding to me again without massive and weird delays. Yunohost was functional (even if Nextcloud still wasn’t).

Restoring Nextcloud to functionality

1. I decided to remove the Nextcould app again. Removing worked without problems.
2. I restored my original backup. This time, restoring the backup worked without problems, but: I was back seeing a familiar error: The Internal Server Error as described above and in many other threads.
3. Using the yunohost-webadmin-interface, I experimented putting Nextcloud in maintenance mode. I opened Nextcloud in the user interface, and in fact, instead of an ugly Internal Server Error message I was greeted by a beautiful cloud on blue background informing me of maintenance mode. So, Nextcloud seemed to be somehow intact?
4. I started familiarising myself with the occ method of passing commands to Nextcloud via SSH. Seemed scary at first, but simply typing the overview command helped me in reading through and understanding what might help:
   sudo -u nextcloud php8.2 --define apc.enable_cli=1 /var/www/nextcloud/occ
5. Discovered by chance the second magic bullet command:

cd /var/www/nextcloud/
sudo -u nextcloud php8.2 --define apc.enable_cli=1 occ maintenance:repair

That’s all it took! Nextcloud was functional again! Well, more or less, because I started running into the issues in this thread.

Ongoing problem with user@50465.service

Because of that, I also tweaked the php.ini configuration files a bit and restarted php8.2-fpm.service a couple of times. The first few times, I kept getting the same issue immediately after the service restart:

systemctl --failed would yield the following:

  UNIT               LOAD   ACTIVE SUB    DESCRIPTION
● user@50465.service loaded failed failed User Manager for UID 50465

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
1 loaded units listed.

systemctl reset-failed would fix it again, and then it was gone.

This repeated for a couple of times. Now, the last 3 service restarts, it disappeared. I don’t know why precisely, but the error is hopefully gone for good.

So far, from me. My setup is still shaky but I hope it will hold the next couple of days. I have no idea though when and how it would be safe for me to try updating my Nextcloud app again? If any of the issues re-emerge, I will reply here again. In the meantime I hope no one else runs into the same issues I did, and if you do, I hope that this thread will be a good help to you!

Best regards and a joyous week.