502 Bad Gateway

Strathy · July 23, 2017, 3:16am

Hey everyone.

I’m running:

yunohost
2.6.4
yunohost-admin
2.6.1
moulinette
2.6.1
ssowat
2.6.8

On a Raspberry Pi.

After my last update, I’m getting 502 Bad Gateway on Nextcloud and Rainloop. I did some reading but I’m not sure what is causing this issue for me. Here is what I find:

In Updates, NextCloud shows as needing an update, but when I try to do it, it fails. This is what I get:

app_upgrade_failed
unable to upgrade apps: [Errno 61] No app to upgrade
Unable to upgrade nextcloud

exit 1

echo ‘Backup failed, the upgrade process was aborted.’

ynh_die ‘Backup failed, the upgrade process was aborted.’
Backup failed, the upgrade process was aborted.

‘[’ 22 -eq 0 ‘]’
There is nothing to save
Unable to back up the app ‘nextcloud’
mysqldump: Got error: 2002: “Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (2 “No such file or directory”)” when trying to connect
Data dir will not saved, because backup_core_only is set.
–ignore-hooks is deprecated and will be removed in thefuture. Please use --ignore-system instead.

sudo yunohost backup create --ignore-hooks --apps nextcloud --name nextcloud-before-upgrade1

When I look at services, several are inactive including php5-fpm. When I try to start it I get:

Unable to start service ‘php5-fpm’
Unable to execute command ‘service php5-fpm start’

The log for php5-fpm is this:

[18-Jul-2017 17:00:12] NOTICE: [pool nextcloud] child 11244 exited with code 0 after 65191.191625 seconds from start
[18-Jul-2017 17:00:12] NOTICE: [pool nextcloud] child 25476 started
[18-Jul-2017 17:00:42] NOTICE: [pool nextcloud] child 11248 exited with code 0 after 65210.022702 seconds from start
[18-Jul-2017 17:00:42] NOTICE: [pool nextcloud] child 25483 started
[18-Jul-2017 17:01:12] NOTICE: [pool nextcloud] child 11261 exited with code 0 after 65221.125242 seconds from start
[18-Jul-2017 17:01:12] NOTICE: [pool nextcloud] child 25485 started
[18-Jul-2017 17:01:42] NOTICE: [pool nextcloud] child 11267 exited with code 0 after 65191.107147 seconds from start
[18-Jul-2017 17:01:42] NOTICE: [pool nextcloud] child 25499 started
[18-Jul-2017 17:02:12] NOTICE: [pool nextcloud] child 11265 exited with code 0 after 65251.071320 seconds from start
[18-Jul-2017 17:02:12] NOTICE: [pool nextcloud] child 25509 started
[19-Jul-2017 11:05:14] NOTICE: [pool nextcloud] child 25476 exited with code 0 after 65101.990348 seconds from start
[19-Jul-2017 11:05:14] NOTICE: [pool nextcloud] child 8502 started
[19-Jul-2017 11:05:17] NOTICE: [pool nextcloud] child 25483 exited with code 0 after 65075.211428 seconds from start
[19-Jul-2017 11:05:17] NOTICE: [pool nextcloud] child 8504 started
[19-Jul-2017 11:05:43] NOTICE: [pool nextcloud] child 25485 exited with code 0 after 65071.000614 seconds from start
[19-Jul-2017 11:05:43] NOTICE: [pool nextcloud] child 8510 started
[19-Jul-2017 11:06:13] NOTICE: [pool nextcloud] child 25499 exited with code 0 after 65070.772024 seconds from start
[19-Jul-2017 11:06:13] NOTICE: [pool nextcloud] child 8525 started
[19-Jul-2017 11:06:43] NOTICE: [pool nextcloud] child 25509 exited with code 0 after 65070.789258 seconds from start
[19-Jul-2017 11:06:43] NOTICE: [pool nextcloud] child 8538 started
[20-Jul-2017 05:13:14] NOTICE: [pool nextcloud] child 8502 exited with code 0 after 65279.545002 seconds from start
[20-Jul-2017 05:13:14] NOTICE: [pool nextcloud] child 23054 started
[20-Jul-2017 05:13:44] NOTICE: [pool nextcloud] child 8504 exited with code 0 after 65306.394016 seconds from start
[20-Jul-2017 05:13:44] NOTICE: [pool nextcloud] child 23060 started
[20-Jul-2017 05:13:54] NOTICE: [pool nextcloud] child 8510 exited with code 0 after 65291.226461 seconds from start
[20-Jul-2017 05:13:54] NOTICE: [pool nextcloud] child 23062 started
[20-Jul-2017 05:14:14] NOTICE: [pool nextcloud] child 8525 exited with code 0 after 65280.909594 seconds from start
[20-Jul-2017 05:14:14] NOTICE: [pool nextcloud] child 23064 started
[20-Jul-2017 05:14:44] NOTICE: [pool nextcloud] child 8538 exited with code 0 after 65281.215398 seconds from start
[20-Jul-2017 05:14:44] NOTICE: [pool nextcloud] child 23066 started
[20-Jul-2017 23:20:20] NOTICE: [pool nextcloud] child 23060 exited with code 0 after 65195.944954 seconds from start
[20-Jul-2017 23:20:20] NOTICE: [pool nextcloud] child 5585 started
[20-Jul-2017 23:20:45] NOTICE: [pool nextcloud] child 23054 exited with code 0 after 65251.591490 seconds from start
[20-Jul-2017 23:20:45] NOTICE: [pool nextcloud] child 5587 started
[20-Jul-2017 23:21:15] NOTICE: [pool nextcloud] child 23064 exited with code 0 after 65221.563024 seconds from start
[20-Jul-2017 23:21:15] NOTICE: [pool nextcloud] child 5591 started
[20-Jul-2017 23:21:46] NOTICE: [pool nextcloud] child 23062 exited with code 0 after 65271.252828 seconds from start
[20-Jul-2017 23:21:46] NOTICE: [pool nextcloud] child 5597 started
[20-Jul-2017 23:22:15] NOTICE: [pool nextcloud] child 23066 exited with code 0 after 65251.230253 seconds from start
[20-Jul-2017 23:22:15] NOTICE: [pool nextcloud] child 5602 started
[21-Jul-2017 17:27:47] NOTICE: [pool nextcloud] child 5587 exited with code 0 after 65221.820303 seconds from start
[21-Jul-2017 17:27:47] NOTICE: [pool nextcloud] child 20595 started
[21-Jul-2017 17:28:17] NOTICE: [pool nextcloud] child 5585 exited with code 0 after 65277.614359 seconds from start
[21-Jul-2017 17:28:17] NOTICE: [pool nextcloud] child 20597 started
[21-Jul-2017 17:28:47] NOTICE: [pool nextcloud] child 5591 exited with code 0 after 65251.678594 seconds from start
[21-Jul-2017 17:28:47] NOTICE: [pool nextcloud] child 20604 started
[21-Jul-2017 17:28:56] NOTICE: [pool nextcloud] child 5597 exited with code 0 after 65230.274968 seconds from start
[21-Jul-2017 17:28:56] NOTICE: [pool nextcloud] child 20606 started
[21-Jul-2017 17:29:17] NOTICE: [pool nextcloud] child 5602 exited with code 0 after 65221.723489 seconds from start
[21-Jul-2017 17:29:17] NOTICE: [pool nextcloud] child 20608 started

Sorry for posting so much stuff … any help would be appreciated!

Strathy

Strathy · July 26, 2017, 3:07pm

Also getting the following email every half hour:

Cron root@YunoHost [ -x /usr/lib/php5/sessionclean ] && /usr/lib/php5/sessionclean

Illegal instruction

Anybody have anything on this at all? Right now my Yunohost is not usable for me.

Aleks · July 26, 2017, 3:12pm

Edit: ugh, nevermind my idea, I was thinking about another issue (about backup restoration) but that’s actually unrelated Sorry about the noise

Hi there,

sorry I missed your previous thread,

can you try to do the following (if you did not install archivemount)(Yes I know the command looks very mysterious ) :

cp /bin/false /usr/bin/archivemount

and retry the nextcloud upgrade ?

Strathy · July 26, 2017, 3:41pm

Thanks! That did not work either. This is the log output:

https://paste.yunohost.org/huweguxufa

Aleks · July 26, 2017, 3:45pm

Eh okay (c.f. my previous message, my proposition was actually unterlated to your issue, I was mistaking with backup restoration)

But your log shows an interesting message :

2017-07-26 15:38:11,399 WARNING  yunohost.hook &lt;lambda&gt; - [794.33] mysqldump: Got error: 2002: "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")" when trying to connect

Can you try to check that mysql is up with

service mysql status

Strathy · July 27, 2017, 1:23pm

OK - mysql was inactive. I tried starting it and nothing happened. No logs either. Here is what I see:

Aleks · July 27, 2017, 1:26pm

Hm okay that’s weird

Can you log in SSH ? Maybe doing a service mysql status will show more logs… Also maybe doing a dpkg --list | grep mysql should show if mysql is still installed. (Or maybe dpkg --list | grep mariadb ?

Strathy · July 27, 2017, 1:33pm

Here are the results of those three ideas:

Aleks · July 27, 2017, 2:06pm

Okay, donc deja la bonne nouvelle c’est que mysql est toujours là

Ensuite, normalement le “vrai” fichier de log est caché dans un truc comme /var/lib/mysql/tonhostname.err.

Du coup est-ce que tu peux montrer le resultat de tail -n 100 /var/lib/mysql/tonhostname.err (il faut remplacer tonhostname par le nom de la machine Dans ton cas ça a l’air d’etre YunoHost)

Strathy · July 27, 2017, 2:29pm

Here are the results:

https://paste.yunohost.org/lidiziniga.vbs

Aleks · July 27, 2017, 2:33pm

Eh okay, that looks like Mysql DB corruption I think Did you server shutdown suddently recently (e.g. electrical power cut or something) ?

Strathy · July 27, 2017, 2:35pm

It happens. I’m out in the country and our power supply can go out quite suddenly sometimes. Is there a fix?

Aleks · July 27, 2017, 2:55pm

Okay,

in fact it does not look exactly like other corruptions I have seen …

Well, I don’t know what this exactly does, but you can try the following. Open /etc/mysql/my.cnf (e.g. with nano /etc/mysql/my.cnf) and add this line in the [mysqld] block :

innodb_force_recovery = 1

(as described here)

Then try restarting mysql again.

Strathy · August 2, 2017, 2:58am

The above did work and I was able to start mysql. However, I still could not get php5-fpm to start. None of the things I tried from Google searches would allow it to start. So I finally uninstalled NextCloud and restarted php5-fpm and boom it works again. So now I can access Rainloop again.

I have tried to install NextCloud again, but it fails and kills the php5-fpm again … although I can easily restart it.

I wish I could fix this but my knowledge is just not where it needs to be to do much more. At this point I will have to go back to Evernote as I just can’t trust YunoHost with important files.

The mail server seems to be working well still … so I will continue to use that part for now.

Strathy · August 2, 2017, 3:14am

Further update. Still getting emails every half hour:

Cron root@YunoHost [ -x /usr/lib/php5/sessionclean ] && /usr/lib/php5/sessionclean

Illegal instruction