Nginx cannot start after update

Support
, ,
1

Hello everyone / Bonsoir à tous

My YunoHost server

Hardware: VPS bought online
YunoHost version: 4.3.6.2 on Debian 10
I have access to my server : Through SSH
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : yes
If yes, please explain:
I enabled 2fa for ssh according to this guide : How to Use Two-Factor Authentication with SSH on Debian 9 | Linode

Description of my issue

Earlier this evening I started an update via the Admin GUI. A few system packages needed updates (if i remember correctly they were: yunohost-admin, moulinette and ssowat) No applications needed updating. I came back 1 or 2 hours after and noticed the server is unreachable via https. I tried a reboot with no success. I connected via SSH and noticed that nginx is not running and fails to start because of a timeout.

Here is what I tried:

  • restarting the service:
$ sudo yunohost service restart nginx

Job for nginx.service failed because a timeout was exceeded.
See "systemctl status nginx.service" and "journalctl -xe" for details.
Warning: Could not execute the command 'systemctl restart nginx'
  • checking the logs:
$ sudo journalctl -u nginx

Jan 26 22:39:18 mydomain.com systemd[1]: Starting A high performance web server and a reverse proxy server...
Jan 26 22:40:48 mydomain.com systemd[1]: nginx.service: Start-pre operation timed out. Terminating.
Jan 26 22:40:48 mydomain.com systemd[1]: nginx.service: Control process exited, code=killed, status=15/TERM
Jan 26 22:40:48 mydomain.com systemd[1]: nginx.service: Failed with result 'timeout'.
Jan 26 22:40:48 mydomain.com systemd[1]: Failed to start A high performance web server and a reverse proxy server.
  • Tested the nginx config:
$ sudo nginx -t

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
  • retried the update from cli, but everything is up to date :
$ sudo yunohost tools update system

Info: Fetching available upgrades for system packages...
Info: Nothing to do. Everything is already up-to-date.
apps: 
system:
  • I tried to see if something else was not installed correctly, but everything seems fine :
$ sudo apt install --fix-broken

Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

I’m at a loss as to how to continue. Does anyone have any ideas ?

Thanks in advance and have a nice evening / Merci d’avance et bonne soirée

2

Does it give any other information when you run the non-yunohost service command?

$ sudo service nginx start # or restart, of course, depending on current status
``

When I run either of them, there is no mention of 'start-pre' operations. You could have a look in the init-file at `/etc/init.d/nginx' if anything could be related (I don't have anything 'pre' there either). 

[quote="pigeontech, post:1, topic:18623"]
* I tried to see if something else was not installed correctly, but everything seems fine :

$ sudo apt install --fix-broken

[/quote]

You could try `sudo apt get reinstall nginx`.

Does diagnosis on the command line give any suggestions?
1 Like
3

Hi wbk,

Thanks for your reply.

Here’s the output of the non-yunohost command:

$ sudo service nginx start
Job for nginx.service failed because a timeout was exceeded.
See "systemctl status nginx.service" and "journalctl -xe" for details.

Here is the service status info:

$ systemctl status nginx.service
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: timeout) since Sat 2022-01-29 16:58:23 CET; 3min 45s ago
     Docs: man:nginx(8)
  Process: 20219 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=killed, signal=TERM)

Here is what I find in journalctl:

$ sudo journalctl -xe | grep nginx
Jan 29 16:56:51 mydomain.com sudo[20195]:    admin : TTY=pts/0 ; PWD=/home/admin ; USER=root ; COMMAND=/usr/sbin/service nginx start
-- Subject: A start job for unit nginx.service has begun execution
-- A start job for unit nginx.service has begun execution.
Jan 29 16:58:23 mydomain.com systemd[1]: nginx.service: Start-pre operation timed out. Terminating.
Jan 29 16:58:23 mydomain.com systemd[1]: nginx.service: Control process exited, code=killed, status=15/TERM
-- An ExecStartPre= process belonging to unit nginx.service has exited.
Jan 29 16:58:23 mydomain.com systemd[1]: nginx.service: Failed with result 'timeout'.
-- The unit nginx.service has entered the 'failed' state with result 'timeout'.
-- Subject: A start job for unit nginx.service has failed
-- A start job for unit nginx.service has finished with a failure.

Diagnosis complains about ports and domains not reachable and the service not running:

[ERROR] Port 80 is not reachable from outside.
  - Exposing this port is needed for web features (service nginx)
  - To fix this issue, you most probably need to configure port forwarding on your internet router as described in https://yunohost.org/isp_box_config

[ERROR] Port 443 is not reachable from outside.
  - Exposing this port is needed for web features (service nginx)
  - To fix this issue, you most probably need to configure port forwarding on your internet router as described in https://yunohost.org/isp_box_config

=================================
Web (web)
=================================

[ERROR] Domain mydomain.com appears unreachable through HTTP from outside the local network.
  - Connection error: could not connect to the requested domain, it's very likely unreachable.

=================================
Services status check (services)
=================================

[ERROR] Service nginx is failed :(
  - You can try to restart the service, and if it doesn't work, have a look at the service logs in the webadmin (from the command line, you can do this with 'yunohost service restart nginx' and 'yunohost service log nginx').

I reinstalled nginx via sudo apt-get reinstall nginx and tried to start the service, but the same timeout occurs.

4

Hi,
We have the same issue on an ARN VPS. We are currently searching a solution.

Do you have find a solution ?

5

Try running this command

/usr/sbin/nginx -s stop && /usr/sbin/nginx

6

Hi,

No, I didn’t find a solution. I did a clean reinstall.

7

salut,
Suite à une intervention sur les serveurs d’ARN, il semble que je sois confronté au même problème…

Différentes commandes testées :

systemctl restart nginx

'See "systemctl status nginx.service" and "journalctl -xe" for details.
Warning: Could not execute the command 'systemctl restart nginx'
Error: Could not restart the service 'nginx'

Recent service logs:-- Logs begin at Sun 2022-02-06 10:28:29 UTC, end at Sun 2022-02-06 18:30:04 UTC. --
Feb 06 18:13:07 systemd[1]: Starting A high performance web server and a reverse proxy server...
Feb 06 18:14:37 systemd[1]: nginx.service: Start-pre operation timed out. Terminating.
Feb 06 18:14:37 systemd[1]: nginx.service: Control process exited, code=killed, status=15/TERM
Feb 06 18:14:37 systemd[1]: nginx.service: Failed with result 'timeout'.
Feb 06 18:14:37 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Feb 06 18:26:12 systemd[1]: Starting A high performance web server and a reverse proxy server...
Feb 06 18:27:42 systemd[1]: nginx.service: Start-pre operation timed out. Terminating.
Feb 06 18:27:42 systemd[1]: nginx.service: Control process exited, code=killed, status=15/TERM
Feb 06 18:27:42 systemd[1]: nginx.service: Failed with result 'timeout'.
Feb 06 18:27:42 systemd[1]: Failed to start A high performance web server and a reverse proxy server.'

yunohost tools regen-conf --with-diff --dry-run nginx

Pas de réponse…


3.

df -h


```> Filesystem      Size  Used Avail Use% Mounted on
> udev            984M     0  984M   0% /dev
> tmpfs           200M   21M  179M  11% /run
> /dev/vda1        98G   48G   46G  51% /
> tmpfs           998M  132K  998M   1% /dev/shm
> tmpfs           5.0M     0  5.0M   0% /run/lock
> tmpfs           998M     0  998M   0% /sys/fs/cgroup```

4.

free -m

>               total        used        free      shared  buff/cache   available
> Mem:           1995        1226         205          22         562         589
> Swap:          1023        1023           0

systemctl restart nginx
  1. Job for nginx.service failed because a timeout was exceeded. See “systemctl status nginx.service” and “journalctl -xe” for details.
  2. sysstat is already the newest version (12.0.3-2)
  3. root@fsu72:/# sudo iostat Linux 4.19.0-18-amd64 (fsu72.fr) 02/06/2022 x86_64 (1 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 6.62 0.00 1.80 1.41 0.03 90.14 Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn vda 5.86 41.44 74.12 1848189 3305644
  4. 19:58
    le ‘90.14’ est en rouge.
  1. systemctl status nginx.service
systemctl status nginx.service

● nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Sun 2022-02-06 18:57:08 UTC; 3min 31s ago Docs: man:nginx(8) Process: 27326 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=killed, signal Feb 06 18:55:38 fsu72.fr systemd[1]: Starting A high performance web server and a reverse proxy server… Feb 06 18:57:08 fsu72.fr systemd[1]: nginx.service: Start-pre operation timed out. Terminating. Feb 06 18:57:08 fsu72.fr systemd[1]: nginx.service: Control process exited, code=killed, status=15/TERM Feb 06 18:57:08 fsu72.fr systemd[1]: nginx.service: Failed with result ‘timeout’. Feb 06 18:57:08 fsu72.fr systemd[1]: Failed to start A high performance web server and a reverse proxy se

cat /root/test

test

journalctl -u ngnix


> -- Logs begin at Sun 2022-02-06 10:28:29 UTC, end at Sun 2022-02-06 19:09:32 UTC. -- Feb 06 18:13:07 [fsu72.fr](http://fsu72.fr/) systemd[1]: Starting A high performance web server and a reverse proxy server... Feb 06 18:14:37 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Start-pre operation timed out. Terminating. Feb 06 18:14:37 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Control process exited, code=killed, status=15/TERM Feb 06 18:14:37 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Failed with result 'timeout'. Feb 06 18:14:37 [fsu72.fr](http://fsu72.fr/) systemd[1]: Failed to start A high performance web server and a reverse proxy se Feb 06 18:26:12 [fsu72.fr](http://fsu72.fr/) systemd[1]: Starting A high performance web server and a reverse proxy server... Feb 06 18:27:42 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Start-pre operation timed out. Terminating. Feb 06 18:27:42 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Control process exited, code=killed, status=15/TERM Feb 06 18:27:42 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Failed with result 'timeout'. Feb 06 18:27:42 [fsu72.fr](http://fsu72.fr/) systemd[1]: Failed to start A high performance web server and a reverse proxy se Feb 06 18:30:46 [fsu72.fr](http://fsu72.fr/) systemd[1]: Starting A high performance web server and a reverse proxy server... Feb 06 18:32:17 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Start-pre operation timed out. Terminating. Feb 06 18:32:17 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Control process exited, code=killed, status=15/TERM Feb 06 18:32:17 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Failed with result 'timeout'. Feb 06 18:32:17 [fsu72.fr](http://fsu72.fr/) systemd[1]: Failed to start A high performance web server and a reverse proxy se Feb 06 18:55:32 [fsu72.fr](http://fsu72.fr/) systemd[1]: Starting A high performance web server and a reverse proxy server... Feb 06 18:55:38 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Control process exited, code=killed, status=15/TERM Feb 06 18:55:38 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Succeeded. Feb 06 18:55:38 [fsu72.fr](http://fsu72.fr/) systemd[1]: Stopped A high performance web server and a reverse proxy server. Feb 06 18:55:38 [fsu72.fr](http://fsu72.fr/) systemd[1]: Starting A high performance web server and a reverse proxy server... Feb 06 18:57:08 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Start-pre operation timed out. Terminating. Feb 06 18:57:08 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Control process exited, code=killed, status=15/TERM Feb 06 18:57:08 [fsu72.fr](http://fsu72.fr/) systemd[1]: nginx.service: Failed with result 'timeout'. Feb 06 18:57:08 [fsu72.fr](http://fsu72.fr/) systemd[1]: Failed to start A high performance web server and a reverse proxy se

9.

ps aux | grep nginx



>root     28501  0.0  0.0   6144   884 pts/0    S+   19:10   0:00 grep nginx

10.

usr/sbin/nginx -t -q -g ‘daemon on; master_process on;’

>Pas de réponse.

11.

fuser -k 80/tcp
fuser -k 443/tcp


>Pas de réponse

12.

dpkg -l | grep apache


>Pas de réponse

13.

systemctl start nginx


> 1. Job for nginx.service failed because a timeout was exceeded. See "systemctl status nginx.service" and "journalctl -xe" for details.

14.

yunohost app list


> apps: 
>   0: 
>     description: Zero Knowledge realtime collaborative editor
>     domain_path: cryptpad.fsu72.fr/
>     id: cryptpad
>     name: CryptPad
>     version: 4.12.0~ynh1
>   1: 
>     description: Online editor providing collaborative editing in real-time
>     domain_path: fsu72.fr/pad
>     id: etherpad_mypads
>     name: Etherpad
>     version: 1.8.16~ynh2
>   2: 
>     description: Next-generation forum made simple
>     domain_path: forum.fsu72.fr/
>     id: flarum
>     name: Forum
>     version: 1.2.0~ynh1
>   3: 
>     description: URL Shortener
>     domain_path: fsu72.fr/lstu
>     id: lstu
>     name: Lstu
>     version: 0.23-0~ynh1
>   4: 
>     description: Self hosting files and sharing anonymous application
>     domain_path: lufi.fsu72.fr/
>     id: lufi
>     name: Lufi
>     version: 0.05.16~ynh1
>   5: 
>     description: Online storage, file sharing platform and various other applications
>     domain_path: fsu72.fr/nextcloud
>     id: nextcloud
>     name: Nextcloud
>     version: 22.2.3~ynh1
>   6: 
>     description: Create and edit documents collaboratively
>     domain_path: onlyoffice.fsu72.fr/
>     id: onlyoffice
>     name: OnlyOffice
>     version: 6.4.0~ynh2
>   7: 
>     description: Service for planning an appointment or making a decision quickly and easily
>     domain_path: date.fsu72.fr/
>     id: opensondage
>     name: Date
>     version: 1.1.17~ynh1
>   8: 
>     description: Trello-like kanban
>     domain_path: fsu72.fr/wekan
>     id: wekan
>     name: Wekan
>     version: 5.93~ynh1

15.

netstat -tulpn | grep 80

> tcp        0      0 127.0.0.1:8024          0.0.0.0:*               LISTEN      2140/python3        
> tcp        0      0 0.0.0.0:8095            0.0.0.0:*               LISTEN      455/node            
> tcp        0      0 127.0.0.1:8096          0.0.0.0:*               LISTEN      1616/lufi           
> tcp        0      0 127.0.0.1:8097          0.0.0.0:*               LISTEN      2144/python3        
> tcp        0      0 127.0.0.1:8098          0.0.0.0:*               LISTEN      1607/lstu           
> tcp6       0      0 :::8000                 :::*                    LISTEN      1027/docservice     
> udp6       0      0 fe80::a800:ff:feee::123 :::*                                416/ntpd

/usr/sbin/nginx -t -g 'master_process on;'

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

yunohost settings get security.nginx.compatibility

intermediate

ps aux | grep nginx

root 31744 0.0 0.0 6144 888 pts/0 R+ 19:49 0:00 grep nginx

systemctl start nginx

Job for nginx.service failed because a timeout was exceeded.
See “systemctl status nginx.service” and “journalctl -xe” for details.

tail -n150 /var/log/nginx/error.log

2022/02/06 07:03:09 [info] 4905#4905: Using 32768KiB of shared memory for nchan in /etc/nginx/nginx.conf:63
2022/02/06 18:14:37 [info] 23663#23663: Using 32768KiB of shared memory for nchan in /etc/nginx/nginx.conf:63
2022/02/06 18:27:42 [info] 24605#24605: Using 32768KiB of shared memory for nchan in /etc/nginx/nginx.conf:63
2022/02/06 19:11:47 [info] 28647#28647: Using 32768KiB of shared memory for nchan in /etc/nginx/nginx.conf:63
2022/02/06 19:34:55 [info] 30675#30675: Using 32768KiB of shared memory for nchan in /etc/nginx/nginx.conf:63
2022/02/06 19:42:05 [info] 31230#31230: Using 32768KiB of shared memory for nchan in /etc/nginx/nginx.conf:63

systemctl restart nginx

Job for nginx.service failed because a timeout was exceeded.
See “systemctl status nginx.service” and “journalctl -xe” for details.

systemctl status nginx.service

● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: failed (Result: timeout) since Mon 2022-02-07 11:45:05 UTC; 2min 55s ago
Docs: man:nginx(8)
Process: 4194 ExecStartPre=/bin/true (code=exited, status=0/SUCCESS)
Process: 4195 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=killed

Feb 07 11:43:35 fsu72.fr systemd[1]: Starting A high performance web server and a reverse proxy
Feb 07 11:45:05 fsu72.fr systemd[1]: nginx.service: Start-pre operation timed out. Terminating.
Feb 07 11:45:05 fsu72.fr systemd[1]: nginx.service: Control process exited, code=killed, status=
Feb 07 11:45:05 fsu72.fr systemd[1]: nginx.service: Failed with result ‘timeout’.
Feb 07 11:45:05 fsu72.fr systemd[1]: Failed to start A high performance web server and a reverse

À cette heure, je suis sans solution !!!

8

A priori, tu peux tout de même tenter de lancer un backup (avec les commandes yunohost) et tenter une restauration de ton serveur. Le fait que nginx soit éteint n’empèche pas le backup en général.

Si tu veux je te fais un 2ème VPS pour faciliter la transition.

9

Salut,

je suis navré si mon message a pu paraître courroucé… C’était plus un appel à l’aide pour pouvoir trouver une solution qui profite tant à d’autres qu’à moi face au problème rencontré.

J’ai lancé une commande yunohost backup create.

Ok pour un autre VPS si c’est possible. Merci encore pour la disponibilité, le temps passé et la qualité des services rendu par ARN.

Merci ljf et ARN.

10

Salut,

Démarrage d’un autre VPS…

Via SSH :
connexion en admin@domaine.tld

Sudo su pour passer en root ! (indispensable)

  1. Lancement d’un backup : yunohost backup create
  2. Création de 2 fichiers :
  • 20220207-131219.tar
  • 20220207-131219.info.json
  1. Transfert des 2 fichiers backup du 1er serveur vers le 2e : (info ici)
scp /home/yunohost.backup/archives/20220207-131219.info.json admin@xx.xxx.xxx.xxx:/home/yunohost.backup/archives/

scp /home/yunohost.backup/archives/20220207-131219.tar admin@xx.xxx.xxx.xxx:/home/yunohost.backup/archives/
  1. Via l’interface admin : paramétrage des réglages de base du 2d VPS :
  • création du nom de domaine
  • changement du mot de passe
  • réglage des dns (via Gandi.net) et création du certificat Let’s Encrypt
  1. Lancement du backup

Le début se déroule sans problème… Puis, problème avec Nginx…
Avant la réinstallation de la 1ère application, un problème lié à Nginx a été indiqué (identique au problème sur le précédent serveur…) puis un message d’erreur est apparu en fin de l’install de chaque application… Enfin, en cliquant sur OK pour acter la non installation…

Aucune application installée… Mais, plus d’accès via la console visuelle de yunohost…

  1. Retour à l’accès SSH…
sudo yunohost service restart nginx

● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: timeout) since Tue 2022-02-08 13:25:43 UTC; 45s ago
     Docs: man:nginx(8)
  Process: 30193 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=kille

Feb 08 13:24:12 fsu72.fr systemd[1]: Starting A high performance web server and a reverse proxy 
Feb 08 13:25:43 fsu72.fr systemd[1]: nginx.service: Start-pre operation timed out. Terminating.
Feb 08 13:25:43 fsu72.fr systemd[1]: nginx.service: Control process exited, code=killed, status=
Feb 08 13:25:43 fsu72.fr systemd[1]: nginx.service: Failed with result 'timeout'.
Feb 08 13:25:43 fsu72.fr systemd[1]: Failed to start A high performance web server and a reverse
  1. Installation de iostat
sudo apt install sysstat

iostat

Linux 4.19.0-18-amd64 (fsu72.fr)        02/08/2022      _x86_64_        (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.96    0.00    4.84    8.89    0.06   80.25

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
vda              54.24      3032.09      6146.41   41361817   83845284

Voilà… Retour à la case départ… Si ça peut aider à cerner le problème et contribuer à trouver la solution pour le résoudre. :exploding_head:

11

Zblerg, I think if I was to try to debug this, I would maybe use strace but that’s a bit technical … (something like strace -P $(pidof nginx) while nginx is starting … Or maybe try to manually start nginx, maybe with a debug option if one exists

12

I read about this error and some guys talked about ‘nginx systemd service file’

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.