Killed Yunohost with "Pi-hole -r" (php7.3-fpm, dnmasq)

My YunoHost server

Hardware: Old laptop or computer
YunoHost version: x.x.x
I have access to my server : Through SSH | through the webadmin | direct access via keyboard
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no
If yes, please explain:

Description of my issue…

php7.3-fpm and dnsmasq are broken(?) after uninstalling Pihole
Uninstall log: https://paste.yunohost.org/raw/juwanujaci

Dnsmasq has also disappeared?

$ sudo yunohost service status
Error: Failed to get status information via dbus for service dnsmasq, systemctl didn't recognize this service ('NoSuchUnit').

What happened today…

  • Tried to fix Wireguard & Pihole today by starting fresh.
  • Removed Wireguard & Pihole. Everything OK!
  • Installed Pihole latest available. Everything OK!
  • Tried to increase the DNS cache size by running pihole -r
  • This caused many problems
  • So I was going to uninstall and reinstall Pihole but that killed php7.3-fpm. and dnsmasq

Goal…
Fix php7.3-fpm
Fix dnmasq
Reinstall pihole

Note, I do have daily backups of yunohost system (yunohost backup create --system).
Would rolling back to that help?

Holy bejeezus.

I see what pihole -r did.

/var/www/html$ ls
index.nginx-debian.html

It wiped the whole /var/www/html folder. :expressionless: :expressionless: :expressionless:

Do you remember what is supposed to be there? For me it shows:

# ls /var/www/html/
index.nginx-debian.html  maintenance.etherpad_mypads.html

And actually, etherpad is not installed anymore :thinking:

– edit – I just accidentally repaired some problem by apt reinstall yunohost yunohost-admin moulinette, while checking whether it would have any adverse effects for someone. Give it a try!

There were folders for each app. I know nextcloud had a folder there. My blog was there (I have a backup of those files).

I think nginx has lots of folders in there for each domain. Nginx is a mysterious thing that I don’t know much about.

All the tiles in the Yunohost SSO that link to apps don’t work now because the folders are gone in /var/www/html

Are those not in /etc/nginx/conf.d?

I tested a couple of my apps (startled, because I just ran the reinstall, and have a /var/www/html as empty as you…), but all of them work. Nextcloud does, Wordpress, Element, redirects, either on a seperate domain or a subdirectory. Did you try the apt reinstall?

  1. Sadly this didn’t work.
    apt reinstall yunohost yunohost-admin moulinette
Setting up php7.3-fpm (7.3.33-1+0~20211119.91+debian10~1.gbp618351) ...
locale: Cannot set LC_ALL to default locale: No such file or directory
Job for php7.3-fpm.service failed because the control process exited with error code.
See "systemctl status php7.3-fpm.service" and "journalctl -xe" for details.
invoke-rc.d: initscript php7.3-fpm, action "restart" failed.
● php7.3-fpm.service - The PHP 7.3 FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php7.3-fpm.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2022-01-09 22:02:46 CST; 21ms ago
     Docs: man:php-fpm7.3(8)
  Process: 19124 ExecStart=/usr/sbin/php-fpm7.3 --nodaemonize --fpm-config /etc/php/7.3/fpm/php-fpm.conf (code=exited, status=78)
  Process: 19126 ExecStopPost=/usr/lib/php/php-fpm-socket-helper remove /run/php/php-fpm.sock /etc/php/7.3/fpm/pool.d/www.conf 73 (code=e
xited, status=0/SUCCESS)
 Main PID: 19124 (code=exited, status=78)

Jan 09 22:02:46 arkadi.one systemd[1]: Starting The PHP 7.3 FastCGI Process Manager...
Jan 09 22:02:46 arkadi.one php-fpm7.3[19124]: [09-Jan-2022 22:02:46] NOTICE: PHP message: PHP Warning:  PHP Startup: Unable to load dynam
ic library 'smbclient.so' (tried: /usr/lib/php/20180731/smbclient.so (/usr/lib/php/20180731/smbclient.so: cannot open shared object file:
 No such file or directory), /usr/lib/php/20180731/smbclient.so.so (/usr/lib/php/20180731/smbclient.so.so: cannot open shared object file
: No such file or directory)) in Unknown on line 0
Jan 09 22:02:46 arkadi.one php-fpm7.3[19124]: [09-Jan-2022 22:02:46] ERROR: [pool my_webapp] cannot get uid for user 'my_webapp'
Jan 09 22:02:46 arkadi.one php-fpm7.3[19124]: [09-Jan-2022 22:02:46] ERROR: FPM initialization failed
Jan 09 22:02:46 arkadi.one systemd[1]: php7.3-fpm.service: Main process exited, code=exited, status=78/CONFIG
Jan 09 22:02:46 arkadi.one systemd[1]: php7.3-fpm.service: Failed with result 'exit-code'.
Jan 09 22:02:46 arkadi.one systemd[1]: Failed to start The PHP 7.3 FastCGI Process Manager.
dpkg: error processing package php7.3-fpm (--configure):
 installed php7.3-fpm package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 php7.3-fpm
E: Sub-process /usr/bin/dpkg returned an error code (1)
  1. You’re right about /etc/nginx/conf.d. I have all my NGINX files in there.
    I tried all my apps though domain.tld/nextcloud for example and it give me a “502 bad gateway” error.

  2. I noticed in the pihole uninstall script it did this…

2022-01-09 21:19:58,072: DEBUG - The following packages will be REMOVED:
2022-01-09 21:19:58,073: DEBUG -   cmake* cmake-data* idn2* libgmp-dev* libgmpxx4ldbl* libidn11-dev*
2022-01-09 21:19:58,073: DEBUG -   libjsoncpp1* libncurses-dev* libreadline-dev* librhash0* libuv1*
2022-01-09 21:19:58,074: DEBUG -   linux-image-4.19.0-17-amd64* nettle-dev* php7.3-sqlite3* pihole-ynh-deps*

Maybe I need to reinstall php7.3-sqlite3?

  1. Related to #3. I wonder if I can just reinstall this PHP thing.

$ sudo journalctl -xe
Jan 09 22:09:00 arkadi.one sessionclean[19180]: PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/pdo_mysq
Jan 09 22:09:00 arkadi.one sessionclean[19180]: PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/pdo_sqli
Jan 09 22:09:00 arkadi.one sessionclean[19180]: PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/redis.so
Jan 09 22:09:00 arkadi.one sessionclean[19180]: PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/smbclien
Jan 09 22:09:00 arkadi.one sessionclean[19180]: PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/sqlite3.
Jan 09 22:09:01 arkadi.one sessionclean[19180]: PHP Warning: PHP Startup: Unable to load dynamic library ‘/usr/lib/php/20151012/zip.so’
Jan 09 22:09:01 arkadi.one sessionclean[19180]: PHP Warning: Module ‘apc’ already loaded in Unknown on line 0
Jan 09 22:09:01 arkadi.one systemd[1]: phpsessionclean.service: Succeeded.
– Subject: Unit succeeded

that one also rings some alarm, but is OK if uname -a shows another kernel.

and

are worth looking into. Unable to set default locale usually only gives warnings, not errors directly though.
Having a specifically dated version in /usr/lib/php/20180731/ might point to something. Any idea which app needs it?

Worth a try, maybe have a look at apt show pihole-ynh-deps, maybe there is a package there that is also a dependeny for php-fpm.

(I’m probably offline this afternoon, good luck!)

Apps are in /var/www (usually), not /var/www/html

1 Like
  1. That kernel is not in use, so that’s ok.
$ sudo uname -a
Linux arkadi.one 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 GNU/Linux
  1. Hmm. I’ll have to figure out another way to find this package.
$ sudo apt show pihole-ynh-deps
N: Unable to locate package pihole-ynh-deps
N: Unable to locate package pihole-ynh-deps
E: No packages found
  1. That locale problem has been bugging me for a long time. I’ve tried many tutorials online but can’t figure out how to get rid of it.

  2. Not sure what app needs that php smbclient.so file. I would uninstall some unessential apps to help debug it, but the state the system is in, I’m not sure the other app removals will go any better.

$ sudo apt install php7.3-sqlite3
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package php7.3-sqlite3 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package 'php7.3-sqlite3' has no installation candidate

Not sure what my next move is here. I’ll have to get some sleep and see if anybody has any suggestions.

Hmm… points 1 and 3 are ok then.

I’d expected it to be an available package, because it is being removed. Maybe the package is created on the fly during install, for easy reference during uninstall.

4, smbclient.so, I do have this directory at /usr/lib/php/20180731/, but no smbclient.so. There are directories for the past eight years or so, but none of them contain a smbclient.so.

hmmm… What about $ sudo update-alternatives --display php, which versions are available? Mine says 7.4 would be best (though it is at 7.3 at the moment, when something PHP got upset after a Nextcloud upgrade).

You can set it to another version with update-alternatives --config php. I’d try 7.4 first, and see if it solves the problem, and then try possible other available versions. It won’t break anything (not more than it is now :-P) and you can set it back to what it was afterwards.

  1. I googled a bit and found that Nextcloud seems to have a reference to smbclient.so
    PHP Startup: Unable to load dynamic library 'smbclient.so' - ℹ️ Support - Nextcloud community

They suggest installing smbclient. I tried and failed.
dpkg: error processing package php7.3-fpm (--configure): installed php7.3-fpm package post-installation script subprocess returned error exit status 1

While doing this I noticed that sury repository has bad gpg key. Never noticed that before.

Err:2 https://packages.sury.org/php stretch InRelease
  The following signatures were invalid: EXPKEYSIG B188E2B695BD4743 DEB.SURY.ORG Automatic Signing Key <deb@sury.org>

Also noticed there are somethings that need to be upgraded but doesn’t tell me what.

smbclient is already the newest version (2:4.9.5+dfsg-5+deb10u2).
smbclient set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.
1 not fully installed or removed.

OK fixed the signing of the sury key and found things that could be upgraded. Does this look good?

$ sudo apt-key del B188E2B695BD4743
OK
$ sudo curl -sSL -o /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
$ apt list --upgradeable
Listing... Done
libdns-export162/oldoldstable 1:9.10.3.dfsg.P4-12.3+deb9u10 amd64 [upgradable from: 1:9.10.3.dfsg.P4-12.3+deb9u6]
libisc-export160/oldoldstable 1:9.10.3.dfsg.P4-12.3+deb9u10 amd64 [upgradable from: 1:9.10.3.dfsg.P4-12.3+deb9u6]
libssl1.0.2/oldoldstable 1.0.2u-1~deb9u6 amd64 [upgradable from: 1.0.2u-1~deb9u1]
linux-compiler-gcc-6-x86/oldoldstable 4.9.290-1 amd64 [upgradable from: 4.9.228-1]
linux-kbuild-4.9/oldoldstable 4.9.290-1 amd64 [upgradable from: 4.9.228-1]
php-gd/stretch 2:8.0+91+0~20211231.40+debian9~1.gbpe2a6c5 all [upgradable from: 2:7.4+76+0~20200511.26+debian10~1.gbpc9beb6]
php-mbstring/stretch 2:8.0+91+0~20211231.40+debian9~1.gbpe2a6c5 all [upgradable from: 2:7.4+76+0~20200511.26+debian10~1.gbpc9beb6]
php-xml/stretch 2:8.0+91+0~20211231.40+debian9~1.gbpe2a6c5 all [upgradable from: 2:7.4+76+0~20200511.26+debian10~1.gbpc9beb6]
python3-toml/stable 0.10.0-1+yunohost all [upgradable from: 0.10.0-1]
unscd/stable 0.53-1+yunohost amd64 [upgradable from: 0.53-1+b1]

Says oldoldstable, debian 9??? I hesitate to upgrade some of these.

  • About finding alternatives to PHP. This is what it tells me.
 $ sudo update-alternatives --display php
php - manual mode
  link best version is /usr/bin/php7.4
  link currently points to /usr/bin/php7.0
  link php is /usr/bin/php
  slave php.1.gz is /usr/share/man/man1/php.1.gz
/usr/bin/php7.0 - priority 70
  slave php.1.gz: /usr/share/man/man1/php7.0.1.gz
/usr/bin/php7.3 - priority 73
  slave php.1.gz: /usr/share/man/man1/php7.3.1.gz
/usr/bin/php7.4 - priority 74
  slave php.1.gz: /usr/share/man/man1/php7.4.1.gz

PS thanks for giving me some suggestions here.

(open to read the rest of my story, sury and updates, before I noticed the ominous 'currently points to 7.0' )

I had the invalid keys on Sury quite a while back, having picked up the new key back then, I expect to have the newer packages. Oldstable is Buster/Debian 10, time flies :stuck_out_tongue:

To take one example, I have two versions of libdns-export installed, 162 like you and 1104:

# apt search libdns-export
Sorting... Done
Full Text Search... Done
libdns-export1104/oldstable,now 1:9.11.5.P4+dfsg-5.1+deb10u6 amd64 [installed,automatic]
  Exported DNS Shared Library

libdns-export162/now 1:9.10.3.dfsg.P4-12.3+deb9u7 amd64 [installed,local]
  Exported DNS Shared Library

Just to be sure, your Debian is 10, is it not?

# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:        10
Codename:       buster

You could try upgrading one at a time by apt install package-name. It will set the package from ‘installed, automatic’ to ‘installed, manual’ (taking it out of the list of autoremove packages), and upgrade only that package.

For myself, I would apt upgrade or even better, use the Yunohost upgrade command, yunohost tools update system/yunohost tools upgrade system.

Ah! I should have read your whole post with more attention before starting my story above. Nextcloud (18+?) needs at least PHP 7.3!

Yes, I’m on Debian 10.
I tried yunohost tools upgrade system. There are lots of upgrades there.

$ sudo yunohost tools upgrade system
Error: You cannot do this right now because dpkg/APT (the system package managers) seems to be in a broken state... You can try to solve this issue by connecting through SSH and running `sudo apt install --fix-broken` and/or `sudo dpkg --configure -a`.
admin@arkadi:~$ sudo apt install --fix-broken

I tried both apt install --fix-broken and dpkg --configure -a and php7.3-fpm stops both of those processes with the same error :pensive:

About the install failing I see this line:
Jan 10 07:27:06 arkadi.one php-fpm7.3[6493]: [10-Jan-2022 07:27:06] ERROR: [pool my_webapp] cannot get uid for user 'my_webapp'
This is complaining about this app I have installed, GitHub - YunoHost-Apps/my_webapp_ynh: Custom Web app with SFTP access I’m not sure how to do anything about it, but it seems to bother php.

I also noticed it wants me to upgrade apps that I haven’t even had installed for over a year: Leed, writefreely, keeweb, monitorix. It’s like Pihole literally took my system back in time?

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 10 (buster)
Release:	10
Codename:	buster

Ok at least I’m still on Debian 10.

Also I had the latest version of Nextcloud installed before this all happened, and now it says I’m back on version 18? :cold_sweat:

current_version: 18.0.2~ynh1
    id: nextcloud
    label: Nextcloud
    new_version: 22.2.3~ynh1

The more we debug here, the more confused I get.

Full output of yunohost tools upgrade

$ sudo yunohost tools update
Info: Fetching available upgrades for system packages...
Info: Updating application catalog...
Success! The application catalog has been updated!
Warning: Failed to get label for app keeweb ?
Warning: Failed to get label for app writefreely ?
Warning: Failed to get label for app leed ?
Warning: Failed to get label for app monitorix ?
apps: 
  0: 
    current_version: 1.7.7~ynh3
    id: keeweb
    label: None
    new_version: 1.18.7~ynh1
  1: 
    current_version: 0.12.0~ynh1
    id: writefreely
    label: None
    new_version: 0.13.1~ynh1
  2: 
    current_version: 1.0~ynh2
    id: transmission
    label: Transmission
    new_version: 1.0~ynh4
  3: 
    current_version: 1.8.3~ynh9
    id: leed
    label: None
    new_version: 1.9.0~ynh3
  4: 
    current_version: 0.16.0~ynh1
    id: searx
    label: Searx
    new_version: 1.0.0~ynh5
  5: 
    current_version: 18.0.2~ynh1
    id: nextcloud
    label: Nextcloud
    new_version: 22.2.3~ynh1
  6: 
    current_version: 0.96.0~ynh3
    id: calibreweb
    label: Calibre-web
    new_version: 0.96.14~ynh1
system: 
  0: 
    current_version: 1:9.10.3.dfsg.P4-12.3+deb9u6
    name: libdns-export162
    new_version: 1:9.10.3.dfsg.P4-12.3+deb9u10
  1: 
    current_version: 1:9.10.3.dfsg.P4-12.3+deb9u6
    name: libisc-export160
    new_version: 1:9.10.3.dfsg.P4-12.3+deb9u10
  2: 
    current_version: 1.0.2u-1~deb9u1
    name: libssl1.0.2
    new_version: 1.0.2u-1~deb9u6
  3: 
    current_version: 4.9.228-1
    name: linux-compiler-gcc-6-x86
    new_version: 4.9.290-1
  4: 
    current_version: 4.9.228-1
    name: linux-kbuild-4.9
    new_version: 4.9.290-1
  5: 
    current_version: 2:7.4+76+0~20200511.26+debian10~1.gbpc9beb6
    name: php-gd
    new_version: 2:8.0+91+0~20211231.40+debian9~1.gbpe2a6c5
  6: 
    current_version: 2:7.4+76+0~20200511.26+debian10~1.gbpc9beb6
    name: php-mbstring
    new_version: 2:8.0+91+0~20211231.40+debian9~1.gbpe2a6c5
  7: 
    current_version: 2:7.4+76+0~20200511.26+debian10~1.gbpc9beb6
    name: php-xml
    new_version: 2:8.0+91+0~20211231.40+debian9~1.gbpe2a6c5
  8: 
    current_version: 0.10.0-1
    name: python3-toml
    new_version: 0.10.0-1+yunohost
  9: 
    current_version: 0.53-1+b1
    name: unscd
    new_version: 0.53-1+yunohost

It could be that uninstalling took your system back in time, if it crashed somewhere and accidentally restored a back up of Yunohost system from long ago. Seems far fetched, at first sight. It is in line with the symptoms though, with PHP at 7.0 etc.

Do you still have the full log of the Pihole uninstall? How far back do the backups go?

You could try yunohost tools upgrade system to only upgrade system packages and leave the apps as they are, for now.

Full log.

I tried only updating the system but can’t get past the php7.3-fpm error.

Honestly, I feel the battle is kinda lost here. I am basically thinking about doing a full re-install.

I posted an issue on pihole_ynh github basically for maintainers to maybe take a look at the uninstall script. I’m not sure how it could go that badly. This is an edge case, but quite a catastrophic one.

For a full reinstall, I have all my data backed up. It will just be time consuming. I am only now trying to figure out do I install debian 10 or debian 11. I guess safest bet is debian 10.

I’m not sure what else to try. I can’t get past that php broken install, and even if I do. It looks like my system is compromised with old stuff.

You already ran update-alternatives to set PHP to 7.3 instead of 7.0? If not, first see if that helps!

I asked because I wondered whether the uninstall could have caused a restore of such an old state. Not your own backups, but the system created backups in Yunohost.

Probably not related, but I see this line in the log,

That is about the Wireguard module for the kernel being uninstalled. The newer kernel was available in /boot at that time, but it seems the old kernel was still running. It should not give a problem though.

1 Like

I haven’t done this. I’m reading about this now. It looks like I should do this…

sudo update-alternatives --config php
select php7.3

Then I’ll try to run some updates and things again. As you already mentioned, I don’t have much to lose at this time. I’ll just make sure my data is backup up as well as I thought.

I probably won’t do a full reinstall for a few days, so I still have a few days to mess around with this craziness.

I also saw this and wasn’t sure what was happening there. I actually uninstalled Wireguard before installing, and messing up this pihole installation.

Exactly!

In that case, it would seem the system was already messed up before uninstalling Pihole.

Could it be there is a problem with your hard disk? Try an fsck on all partitions to be sure (for the root partition you’ll have to schedule it with sudo touch /forcefsck).

Browsing though your prior posts, did you have Pihole installed for a long time? Some speculation:

  • install pihole in 2020 (Yunohost creates some backups)
  • install all kinds of software
  • something goes not right
  • uninstall Wireguard
  • something goes not right
  • uninstall Pihole; the little errors so far trigger a partial roll-back; the script thinks it is rolling back the installation instead of the uninstallation, and restores the backup from 2020

It is just my fantasy, not based on any knowledge of the actual situation or looking into the scripts involved!

I did this.
sudo update-alternatives --config php
select php7.3

Then didn’t know what to do, so I tried
sudo apt upgrade --fix-missing
Still failed.

You’re pretty much right on about how I got to this point:

  • installed pihole in 2020 (Yunohost creates some backups)
  • install all kinds of software
  • something goes not right with yunohost dnsmasq and or pihole/wireguard updates circa winter 2021
  • pihole and wireguard don’t work
  • uninstall Wireguard
  • uninstall, reinstall pihole
  • something goes not right (I ran pihole -r)
  • uninstall Pihole to fix problem; the little errors so far trigger a partial roll-back; the script thinks it is - rolling back the installation instead of the uninstallation, and restores the backup from 2020

So, basically a huge mess.

1 Like

Oh noes!

While having backups available, I guess it is not worth it anymore to try to figure out how exactly this mess got formed. How do you intend to restore your server?