Certficate Renewal Fail? -63 days validity after upgrading to Buster?

Support
1

My YunoHost server

Hardware: computer
YunoHost version:

  • yunohost version: 4.0.3 (stable)
    - yunohost-admin version: 4.0.3 (stable)
    - moulinette version: 4.0.3 (stable)
    - ssowat version: 4.0.3+202007291517 (stable)

I have access to my server : ssh / keyboard
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no
If yes, please explain:

Description of my issue

Upgraded to Buster 2 days ago. 14 hours ago my Diagnosis was showing lots of issues so I checked and suddenly my LetsEncrypt certificate is -63 days valid.

I had issues in diagnosis about NGINX conf files were modified by me (they weren’t) so I forced regen’ed them.

I then tried to update it with admin@arkadi:~$ sudo yunohost domain cert-install arkadi.one --force but it failed.

2020-08-12 10:07:33,719: ERROR - Wrote file to /tmp/acme-challenge-public/y1YGohdDDr4OzMqw1AKrol8vTyfIrkwRJ9K6iIrIuqs, but couldn't download http://maindomain.tld/.well-known/acme-challenge/y1YGohdDDr4OzMqw1AKrol8vTyfIrkwRJ9K6iIrIuqs: Error:
Url: http://maindomain.tld/.well-known/acme-challenge/y1YGohdDDr4OzMqw1AKrol8vTyfIrkwRJ9K6iIrIuqs
Data: None
Response Code: None
Response: <urlopen error [Errno 110] Connection timed out>
2020-08-12 10:07:33,777: ERROR - Certificate renewing for maindomain.tld failed !

https://paste.yunohost.org/raw/kexurawopo

I tried the cert-install both from webadmin and from terminal.

Diagnosis was complaining about yunohost-firewall before so I ran `sudo yunohost service status’ to check on things.

glances: 
  configuration: unknown
  description: LSB: Starts and daemonize Glances server
  last_state_change: 2020-08-12 13:54:25
  start_on_boot: enabled
  status: exited

php7.0-fpm: 
  configuration: unknown
  description: The PHP 7.0 FastCGI Process Manager
  last_state_change: 2020-08-12 13:54:42
  start_on_boot: enabled
  status: failed

postfix: 
    configuration: unknown
   description: Used to send and receive e-mails
  last_state_change: 2020-08-12 13:55:17
  start_on_boot: enabled
  status: unknown
   
 yunohost-firewall: 
    configuration: unknown
    description: Manages open and close connection ports to services
    last_state_change: 2020-08-12 09:44:24
    start_on_boot: enabled
    status: unknown

Status unknown/failed on these things probably not good.

Any ideas? Lots of weird things happening.

2

Poking around a lot and it feels like my migration to Buster is incomplete. I don’t remember seeing any problems when running the migration though.

  1. Sources still list Stretch

    ~$ sudo apt update
    Hit:1 Index of /debian-security stretch/updates InRelease
    Hit:2 Index of /php/ stretch InRelease
    Ign:3 Index of /debian stretch InRelease
    Hit:4 Index of /debian stretch-updates InRelease
    Hit:5 Index of /debian stretch Release
    Hit:7 Index of /debian/ stretch InRelease
    Hit:8 Index of /torproject.org buster InRelease

  2. Says migration to Buster still pending

$ sudo yunohost tools migrations list
migrations:
0:
description: Upgrade the system to Debian Buster and YunoHost 4.x
disclaimer: None
id: 0015_migrate_to_buster
mode: manual
name: migrate_to_buster
number: 15
state: pending
1:
description: Migrate php7.0-fpm ‘pool’ conf files to php7.3
disclaimer: None
id: 0016_php70_to_php73_pools
mode: auto
name: php70_to_php73_pools
number: 16
state: pending
2:
description: Migrate databases from PostgreSQL 9.6 to 11
disclaimer: None
id: 0017_postgresql_9p6_to_11
mode: auto
name: postgresql_9p6_to_11
number: 17
state: pending
3:
description: Migrate old network traffic rules to the new nftable system
disclaimer: None
id: 0018_xtable_to_nftable
mode: auto
name: xtable_to_nftable
number: 18
state: pending

  1. Says I’m on Buster but Yunohost doesn’t seem to know that

    $ lsb_release -a
    No LSB modules are available.
    Distributor ID: Debian
    Description: Debian GNU/Linux 10 (buster)
    Release: 10
    Codename: buster

3

Ok I tried to troubleshoot this on my own.

  1. I changed /etc/apt/sources.list to a new one that only has buster sources

    $ cat /etc/apt/sources.list
    #DEBIAN
    deb Index of /debian buster main contrib
    deb-src Index of /debian buster main contrib

    deb Index of /debian-security buster/updates main contrib non-free
    deb-src Index of /debian-security buster/updates main contrib non-free

    deb Index of /debian buster-updates main contrib non-free
    deb-src Index of /debian buster-updates main contrib non-free

    yunohost
    deb Index of /debian/ buster stable

  2. I ran ‘apt update && apt upgrade’. Fails because of ‘php7.0-fpm.service’.

    $ sudo apt upgrade
    Reading package lists… Done
    Building dependency tree
    Reading state information… Done
    Calculating upgrade… Done
    The following packages will be upgraded:
    libicu63 python3-httplib2
    2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    1 not fully installed or removed.
    Need to get 8,329 kB of archives.
    After this operation, 103 kB of additional disk space will be used.
    Do you want to continue? [Y/n] y
    Get:1 Index of /debian buster/main amd64 libicu63 amd64 63.1-6+deb10u1 [8,300 kB]
    Get:2 Index of /debian buster/main amd64 python3-httplib2 all 0.11.3-2 [29.2 kB]
    Fetched 8,329 kB in 8s (1,080 kB/s)
    Reading changelogs… Done
    (Reading database … 80029 files and directories currently installed.)
    Preparing to unpack …/libicu63_63.1-6+deb10u1_amd64.deb …
    Unpacking libicu63:amd64 (63.1-6+deb10u1) over (63.1-6+0~20190318101012.1+stretch~1.gbp07d7b6) …
    Setting up libicu63:amd64 (63.1-6+deb10u1) …
    (Reading database … 80029 files and directories currently installed.)
    Preparing to unpack …/python3-httplib2_0.11.3-2_all.deb …
    Unpacking python3-httplib2 (0.11.3-2) over (0.11.3-1+0~20190212170638.3+buster~1.gbp2efb8a) …
    Setting up python3-httplib2 (0.11.3-2) …
    Setting up php7.0-fpm (7.0.33-30+0~20200807.37+debian10~1.gbp047872) …
    Job for php7.0-fpm.service failed because the control process exited with error code.
    See “systemctl status php7.0-fpm.service” and “journalctl -xe” for details.
    invoke-rc.d: initscript php7.0-fpm, action “restart” failed.
    ● php7.0-fpm.service - The PHP 7.0 FastCGI Process Manager
    Loaded: loaded (/lib/systemd/system/php7.0-fpm.service; enabled; vendor preset: enabled)
    Active: failed (Result: exit-code) since Wed 2020-08-12 15:56:33 CST; 16ms ago
    Docs: man:php-fpm7.0(8)
    Process: 25941 ExecStart=/usr/sbin/php-fpm7.0 --nodaemonize --fpm-config /etc/php/7.0/fpm/php-fpm.conf (code=exited, status=78)
    Process: 25942 ExecStopPost=/usr/lib/php/php-fpm-socket-helper remove /run/php/php-fpm.sock /etc/php/7.0/fpm/pool.d/www.conf 70 (code=exited, status=0/SUCCESS)
    Main PID: 25941 (code=exited, status=78)

    Aug 12 15:56:33 arkadi.one systemd[1]: Starting The PHP 7.0 FastCGI Process Manager…
    Aug 12 15:56:33 arkadi.one php-fpm7.0[25941]: [12-Aug-2020 15:56:33] NOTICE: PHP message: PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib/php/20151012/zip.so’ - /usr/lib/php/20151012/zip.so: cannot open shared object file: No such file or directory in Unknown on line 0
    Aug 12 15:56:33 arkadi.one php-fpm7.0[25941]: [12-Aug-2020 15:56:33] NOTICE: PHP message: PHP Warning: Module ‘apc’ already loaded in Unknown on line 0
    Aug 12 15:56:33 arkadi.one php-fpm7.0[25941]: [12-Aug-2020 15:56:33] ERROR: [pool leed] the chdir path ‘/var/www/leed’ does not exist or is not a directory
    Aug 12 15:56:33 arkadi.one php-fpm7.0[25941]: [12-Aug-2020 15:56:33] ERROR: failed to post process the configuration
    Aug 12 15:56:33 arkadi.one php-fpm7.0[25941]: [12-Aug-2020 15:56:33] ERROR: FPM initialization failed
    Aug 12 15:56:33 arkadi.one systemd[1]: php7.0-fpm.service: Main process exited, code=exited, status=78/CONFIG
    Aug 12 15:56:33 arkadi.one systemd[1]: php7.0-fpm.service: Failed with result ‘exit-code’.
    Aug 12 15:56:33 arkadi.one systemd[1]: Failed to start The PHP 7.0 FastCGI Process Manager.
    dpkg: error processing package php7.0-fpm (–configure):
    installed php7.0-fpm package post-installation script subprocess returned error exit status 1
    Processing triggers for libc-bin (2.28-10) …
    Errors were encountered while processing:
    php7.0-fpm
    E: Sub-process /usr/bin/dpkg returned an error code (1)

Seems my problem is that not fully changed over to buster, and php70 and php7.3 are conflicting?

My other idea is to skip migration 15 and try the others.

4

Uuuuh wokay … so as suggested in the other topic, yes, first let’s skip migration 15 to allow to run the others …

(I’m not sure to understand why your sources.list was still in stretch considering your system and yunohost was on buster … but anyway …)

Regarding the certificate thing, it would help if you could share the results of the diagnosis (there’s a button to share on the diagnosis view in the webadmin)

Not sure about why php7.0 is broken, maybe the migration php7.0->7.3 will help but i’m not so optimist

1 Like
5

Yeah I’m not very optimistic, but I’ll try to work this out.

I’m prepping another computer to make a new server install as a plan B if this fails.

  1. Diagnosis from Webadmin: https://paste.yunohost.org/raw/eruqezikuk

  2. Used Webadmin to skip migration 15.

    2020-08-12 18:24:13,308: WARNING - Skipping migration 0015_migrate_to_buster…
    (migration 16)
    2020-08-12 18:24:20,591: INFO - Running migration 0016_php70_to_php73_pools…
    2020-08-12 18:24:20,843: DEBUG - Running ‘systemctl restart php7.3-fpm’
    2020-08-12 18:24:21,104: WARNING - Could not execute the command ‘systemctl restart php7.3-fpm’
    2020-08-12 18:24:21,129: DEBUG - Running ‘systemctl enable php7.3-fpm --quiet’
    2020-08-12 18:24:23,272: DEBUG - Running ‘systemctl reload nginx’
    2020-08-12 18:24:23,451: SUCCESS - Migration 0016_php70_to_php73_pools completed

    (migration 17 +18)
    2020-08-12 18:24:23,533: INFO - Running migration 0017_postgresql_9p6_to_11…
    2020-08-12 18:24:23,589: WARNING - PostgreSQL was not installed on your system. Nothing to do.
    2020-08-12 18:24:23,590: SUCCESS - Migration 0017_postgresql_9p6_to_11 completed
    2020-08-12 18:24:23,643: INFO - Running migration 0018_xtable_to_nftable…
    2020-08-12 18:24:25,372: DEBUG - No default hook for action ‘post_iptable_rules’ in /usr/share/yunohost/hooks/
    2020-08-12 18:24:25,403: DEBUG - Executing command ‘sh -c YNH_INTERFACE=api YNH_CWD=/etc/yunohost/hooks.d/post_iptable_rules YNH_STDINFO=/tmp/tmpMNJ1Av/stdinfo YNH_STDRETURN=/tmp/tmpcK89fh/stdreturn BASH_XTRACEFD=7 /bin/bash -x “./50-monitorix” False True 7>&1’…
    2020-08-12 18:24:25,404: DEBUG - About to run the command ‘[‘sh’, ‘-c’, ‘YNH_INTERFACE=api YNH_CWD=/etc/yunohost/hooks.d/post_iptable_rules YNH_STDINFO=/tmp/tmpMNJ1Av/stdinfo YNH_STDRETURN=/tmp/tmpcK89fh/stdreturn BASH_XTRACEFD=7 /bin/bash -x “./50-monitorix” False True 7>&1’]’
    2020-08-12 18:24:25,449: DEBUG - ++ systemctl status monitorix.service
    2020-08-12 18:24:25,450: DEBUG - ++ grep Active
    2020-08-12 18:24:25,450: DEBUG - ++ cut -d ‘)’ -f1
    2020-08-12 18:24:25,451: DEBUG - ++ cut ‘-d(’ -f2
    2020-08-12 18:24:25,451: DEBUG - + [[ running = \r\u\n\n\i\n\g ]]
    2020-08-12 18:24:25,452: DEBUG - + systemctl stop monitorix.service
    2020-08-12 18:24:25,653: DEBUG - + sleep 1
    2020-08-12 18:24:26,656: DEBUG - + pkill -f ‘monitorix-httpd listening on’
    2020-08-12 18:24:26,657: DEBUG - + systemctl start monitorix.service
    2020-08-12 18:24:26,983: DEBUG - Running ‘systemctl reload fail2ban’
    2020-08-12 18:24:27,376: SUCCESS - Firewall reloaded
    2020-08-12 18:24:27,425: DEBUG - Running ‘systemctl restart fail2ban’
    2020-08-12 18:24:28,490: SUCCESS - Service ‘fail2ban’ restarted
    2020-08-12 18:24:28,492: SUCCESS - Migration 0018_xtable_to_nftable completed

  3. Diagnosis log after running migrations looks worse. Now services: php7.0-fpm, php7.3fpm, uwsgi, and writefreely failed. Hmm.

https://paste.yunohost.org/raw/cufayoseqo

6

Alrighty then can you share the logs for php7.3-fpm (in Services > php7.3-fpm, you should find a button to share the logs)

7

php7.3-fpm log (on pastebin, webadmin taking a long time to post a log)

Thank you for your help in troubleshooting this. Hope we can defeat this php7.3-fpm monster :smiley:

8

So the relevant error message is : [pool leed] the chdir path '/var/www/leed' does not exist or is not a directory

Does that mean anything to you ? Is leed supposed to be installed on your system ? Or did you uninstall it ?

9

Oh Leed is an app I uninstalled sometime after ‘installing Buster’. I guess it didn’t uninstall itself very well :expressionless:

I tried reinstalling it to make the system happy but can’t do that because php7.3-fpm not happy right now.

10

Alrighty so removing the file should get rid of the issue :

rm /etc/php/7.3/fpm/pool.d/leed.conf

(I think)

Then try to restart the service (from the webadmin, Services > php7.3-fpm > Restart)

1 Like
11

Hallelujah! It worked.

php7.3-fpm is now running.

Both of my SSL certs have now been updated.

I also did rm /etc/php/7.0/fpm/pool.d/leed.conf and got php7.0 running as well.

12

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.