Upgrade to 2.7 broke YunoHost

I tried to use the Updates section of the web interface to upgrade Yunohost.

I hit the Update button under the Applications section first. That process took a while before it failed with Unexpected Error. Then I tried to upgrade the packages on the server by clicking on the Update button under the Packages section. This failed with the same error again.
I tried the same process again in vain. Then my server stopped serving the web-interface.

Upon inspection on the server, I realized that the web-server (nginx) had gone down. I tried starting it, but it would not come online - it stayed in active (exited) state with no proper logging in journalctl.

I tried sudo apt install -f and got surprised by the following :

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  dns-root-data dnsmasq dnsmasq-base dovecot-antispam etckeeper fail2ban
  glances haveged hddtemp ldap-utils libaio1 libapparmor1
  libdbd-ldap-perl libdbd-mysql-perl libgmime-2.6-0 libhavege1
  libluajit-5.1-2 libluajit-5.1-common libmail-spf-perl libmemcached11
  libmemcachedutil2 libmilter1.0.1 libnet-dns-perl libnet-ip-perl
  libnetaddr-ip-perl libnetfilter-conntrack3 libnss-ldapd
  libnss-myhostname libodbc1 libopendbx1 libopendbx1-sqlite3 libopendkim9
  libpam-ldapd libperl5.20 librbl1 libreadline5 libslp1
  libterm-readkey-perl libunbound2 libvbr2 lua-bitop lua-event lua-expat
  lua-filesystem lua-json lua-ldap lua-lpeg lua-rex-pcre lua-sec
  lua-socket lua-zlib lua5.1 mariadb-client-10.0 mariadb-client-core-10.0
  mariadb-common mariadb-server mariadb-server-10.0
  mariadb-server-core-10.0 metronome moulinette netcat-openbsd
  nginx-common nscd nslcd nslcd-utils opendkim-tools php-gettext php5-fpm
  php5-ldap php5-mysql postfix-policyd-spf-perl python-bottle
  python-dnspython python-gevent python-gevent-websocket python-gnupg
  python-greenlet python-ldap python-miniupnpc python-pyinotify
  python3-apt python3-bottle python3-crypto python3-jinja2
  python3-markupsafe python3-psutil python3-pyasn1 python3-pysnmp4
  rmilter rspamd slapd sudo-ldap unattended-upgrades unzip
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  ssowat yunohost yunohost-admin
0 upgraded, 0 newly installed, 3 to remove and 1 not upgraded.
After this operation, 18.0 MB disk space will be freed.
Do you want to continue? [Y/n]

I thought it was normal before an upgrade and continued but unfortunately, it only removed the packages and didn’t upgrade them.

(Reading database ... 99029 files and directories currently installed.)
Removing yunohost-admin (2.6.2) ...
dpkg: warning: while removing yunohost-admin, directory '/usr/share/yunohost/admin' not empty so not removed
Removing yunohost (2.6.5) ...
Removing ssowat (2.6.8) ...

So, I tried to install the packages manually myself :

sudo apt update && sudo apt upgrade --yes
sudo apt install --yes ssowat yunohost yunohost-admin

and I was thrown the following :

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 ssowat : Depends: nginx-extras (>= 1.6.2) but it is not going to be installed
 yunohost : Depends: nginx-extras (>= 1.6.2) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

OK. So, then I tried to install nginx-extras only to be thrown some more errors :

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 nginx-extras : Depends: perlapi-5.14.2 but it is not installable
                Depends: libgd2-noxpm (>= 2.0.36~rc1~dfsg) but it is not installable or
                         libgd2-xpm (>= 2.0.36~rc1~dfsg) but it is not installable
                Depends: libperl5.14 (>= 5.14.2) but it is not installable
                Recommends: passenger (>= 5.1.8) but it is not going to be installed
                Recommends: passenger (< 5.1.9) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

I looked around the web and found that perlapi-5.14.2 is provided by the perl-base package and my perl-base is already the newest version. I couldn’t find the base package for libgd2-noxmp or libgd2-xmp.

If it makes any sense, I have these lines flooding my journalctl :

modl[29743]: modl.INFO: s:33:"2006 : MySQL server has gone away"; [] []
modl[29743]: modl.INFO: delete from sessionx where active = 0 and start < :timestamp {"timestamp":"2017-09-11 01:05:07"} []

I thought it would be something with the mysql.service, but the error keeps coming even after I started the mysql.service (it was inactive (dead) during my investigation).
And I don’t know what package / service modl refers to so that I could restart it to see if it can pick up the restarted mysql.service.

My server is completely down ATM since yunohost: command not found.
Does someone have some idea as to how I can resolve this and get my server up and running?

Can you specify your Debian version ? It will help us to understand why the dependances are not resolved…

lsb_release --short --description says Debian GNU/Linux 8.9 (jessie)

Could I please get some help here @genma? I’ve got a server that has been down for 5 days because of this issue. :confused:

Backup the system, maybe make a cloned image. It would be good if you can backup all the YunoHost data manually eg. certificates ,apps,mysql,mails etc.
And try to install the the YunoHost server again but DO NOT CONTINUE THE POST INSTALL and restart the server.(Try it at your own risk, you can loose all your data).

Is that the only option I have left?

I was doing just that actually. I was migrating servers and I hadn’t even gotten all the applications running when this happened. When 2.7 came out, I thought, “why not get the latest version?” and boom - the whole system goes tumbling down. I am so disappointed. :disappointed: :disappointed_relieved:

Sad to here this. I’ve done the migration from 2.5 to 2.6 and 2.7 (at each realease version) without problem.
Actually, I don’t see other option than doing a backup of your data (mysql tables and /home/yunohost.apps folder and subfolder) and trying to reinstall 2.7 version.

Are you familiar with command line? Maybe you can use command line to try to check yunohost log (in /var/log/) and give us some /more informations?

I already did look there the first day itself. I couldn’t identify anything wrong there. But I’ll put the last 10 lines (each) of the log files in the direcory for your reference here :

root@shinenelson:/var/log/yunohost# tail yunohost-api.log.1
2017-09-11 02:56:40,375 SUCCESS  yunohost.app success - [27776.1030] The SSOwat configuration has been generated
2017-09-11 02:56:40,376 SUCCESS  yunohost.app success - [27776.1030] Upgrade complete
2017-09-11 02:56:40,376 SUCCESS  yunohost.tools success - [27776.1030] The system has been upgraded
2017-09-11 02:56:40,528 INFO     geventwebsocket.handler log_request - 127.0.0.1 - - [2017-09-11 02:56:40] "PUT /upgrade?ignore_packages HTTP/1.1" 200 12465 472.414048
2017-09-11 03:00:39,887 INFO     moulinette.actionsmap process - processing action [27776.1031]: yunohost.tools.upgrade
2017-09-11 03:00:42,112 INFO     yunohost.tools tools_upgrade - [27776.1031] Critical packages (yunohost, moulinette, yunohost-admin, ssowat) will be upgraded later
2017-09-11 03:00:42,173 INFO     yunohost.tools tools_upgrade - [27776.1031] Upgrading packages...
2017-09-11 03:14:19,396 INFO     yunohost.tools tools_upgrade - [27776.1031] Done
2017-09-11 03:14:19,397 SUCCESS  yunohost.tools success - [27776.1031] The system has been upgraded
2017-09-11 03:14:19,587 INFO     geventwebsocket.handler log_request - 127.0.0.1 - - [2017-09-11 03:14:19] "PUT /upgrade?ignore_apps HTTP/1.1" 200 12507 819.743973
root@shinenelson:/var/log/yunohost# tail yunohost-cli.log
2017-09-11 02:55:44,156 INFO     moulinette.actionsmap process - processing action [4155.1]: yunohost.app.setting
2017-09-11 02:56:19,112 INFO     moulinette.actionsmap process - processing action [4409.1]: yunohost.user.list
2017-09-11 02:56:20,623 INFO     moulinette.actionsmap process - processing action [4458.1]: yunohost.app.setting
2017-09-11 02:56:21,348 INFO     moulinette.actionsmap process - processing action [4466.1]: yunohost.app.setting
2017-09-11 02:56:23,883 INFO     moulinette.actionsmap process - processing action [4545.1]: yunohost.app.setting
2017-09-11 02:56:27,384 INFO     moulinette.actionsmap process - processing action [4578.1]: yunohost.app.setting
2017-09-11 02:56:28,095 INFO     moulinette.actionsmap process - processing action [4585.1]: yunohost.app.setting
2017-09-11 02:56:28,799 INFO     moulinette.actionsmap process - processing action [4592.1]: yunohost.app.setting
2017-09-11 02:56:29,552 INFO     moulinette.actionsmap process - processing action [4597.1]: yunohost.app.setting
2017-09-11 02:56:30,261 INFO     moulinette.actionsmap process - processing action [4607.1]: yunohost.app.setting

I don’t think I can do that on the existing machine anyway, since yunohost would still not be installable - that is the primary problem here. I’ll probably have to start from scratch - installing the OS and everything. But that would preset another problem - where would I back my existing data up to? But that’s not the real problem here. My most important problem is my mail server going down. I cannot afford that at any cost. :disappointed_relieved:

Can you check other log ? (nginx, postfix and other logs)? Maybe you (we, by we I Mean the Yunohost community) can find something… I hope so.

That’s highly unlikely. postfix is still active (running). I can also confirm that dovecot is also active (running) - both since systemctl says so and also because mails gets delivered correctly to my inbox.

But nginx - not so much. It is inactive (dead) since the upgrade. That’s why I’m having this issue. And I isolated the issue to the nginx-extras in the preliminary investigation itself. ssowat, yunohost and yunohost-admin won’t install because they depend on nginx-extras and nginx-extras won’t install because it depends on perlapi-5.14.2, libgd2-noxpm or libgd2-xpm and libperl5.14. This is the main problem from the start.

I did try purging nginx, nginx-common and nginx-extras and re-installing; so, I lost whatever logs I had in the process. But the re-install didn’t fix anything.

It’s too tricky for me… I ask for help on social network, hoping some people will come here and find how to help.

Edit : call done on twitter with a link for this topic.

1 Like

Can you also post the tweet link here too, so that I can keep track of it?

https://twitter.com/genma/status/909052147941167104 & https://framapiaf.org/@genma/2638778/ (2nd is on a mastodon instance)

1 Like

I finally fixed my server :metal: :nerd: :sunglasses: The issue was with the source of passenger which was causing the issue. It was from this source that the nginx binary was being downloaded.

What didn’t catch my eye earlier was the keyword wheezy from the output I shared in my first post. I took so long to note that. Then I went on a grep frenzy trying to find wheezy :joy:

I knew the issue was with the nginx binaries that were being downloaded. Then it all came to me and struck me all at once. I just have to look for the source from nginx was being downloaded.

I searched /etc/apt/sources.list.d/ and couldn’t find a nginx.list. I grepd for nginx in this directory and came up with nothing :disappointed:

I found passenger, but didn’t expect the nginx binaries to come from there. Then I tried to purge nginx completely and try to re-install. And surprisingly, I couldn’t even install nginx.

sudo apt install nginx
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 nginx : Depends: nginx-extras (>= 1:1.12.1-8.5.1.8~wheezy1) but it is not going to be installed
         Depends: nginx-extras (< 1:1.12.1-8.5.1.8~wheezy1.1~) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

But this is when I noticed wheezy.

greping --recursively for --regexp 'wheezy' gave me one and only one result - /etc/apt/sources.list.d/passenger.list. I found the culprit - it was this passenger.list.

I edited the source list file to point to the jessie repos, apt updated and then apt install --yes ssowat yunohost yunohost-admin and voila! everything was back to normal. Of course, I had to install more packages, viz,

  libruby2.1 nginx-common nginx-extras passenger passenger-dev
  passenger-doc ruby-rack ruby2.1 rubygems-integration ssowat yunohost
  yunohost-admin

I also had zombie nginx processes running from during the upgrade; but those were not something a sudo pkill --full nginx --signal SIGKILL couldn’t kill.

In the end, my server was up and running :relieved:

Now, the question to the Yunohost @admins would be : did you accidentally forget to update /etc/apt/sources.list.d/passenger.list to deb https://oss-binaries.phusionpassenger.com/apt/passenger $(lsb_release --short --code) main when you upgraded Yunohost to version 2.7 or was that something the application upgrade from the web console broke?

I was able to fix my server because I had some experience with servers, command line, terminals, nginx, passenger, etc. But this might not be the case with a normal user. Can you please get this part checked and verified so that it doesn’t break for other normal users?

1 Like

The problem is still there in 2.7.10. I am still getting the same error. Any advice on fixing it will be very helpful.

I manage to fix it up manually running update/upgrade commands on the ssh

sudo apt-get update
and
sudo apt-get upgrade

Looks like it doesn’t update from GUI but works from the prompt. Somebody may want to have a look at it.

@abhinavjain ,

Firstly : backup and copy these backup files on another device than your server.

I’m just a simple user but the easier way to upgrade should be to connect you to web-admin’s panel in order to apply updates (instance & applications).

With a terminal, it should be :

sudo apt-get update
sudo apt-get dist-upgrade
then
sudo yunohost app update #lists of apps avaliable update
sudo yunohost app upgrade apps1 --verbose upgrade apps1 with verbose to post in case of problem during the upgrade
sudo yunohost app upgrade apps2 --verbose upgrade apps2 with verbose to post in case of problem during the upgrade
… or
sudo yunohost app upgrade --verbose upgrade all apps with verbose to post in cases of problems during the upgrades

May it could helps you.

ppr

Surprisingly, every time I make a major version upgrade to my distribution, yunohost, nginx and passenger seems to break my server.

I had a similar issue this time around with v3.0.0 as well. Again, the same culprit - nginx-extras, and this time it wasn’t the distribution’s code that broke it. Somehow, the passenger repository lists nginx-common for version 1.14, but doesn’t have the corresponding nginx-extras. I had to manually downgrade nginx-common to version 1.10 so that I could install nginx-extras which was a hard dependency for the yunohost package.

The migration script didn’t run as smooth as I thought it would. It broke because of dependency conflicts of multiple packages (mainly due to some of them being on hold when their dependencies weren’t). However, I took some extra time in figuring those stuff out and manually fixing those upgrade issues.

After an initial go through, after around 250 packages were nicely upgraded, I ran the migration script. It ran through and exited with the message Migration Complete!.
However, upon inspecting the migration log, I found that the script complained at various points including a conf_regen hook for nginx.

After a lot of debugging, it was just this package - nginx-extras that was broken. ( Aside : why does passenger always manage to screw us up? :thinking: ) After the downgrade of nginx-common, I was able to install nginx-extras, but in the process it removed ssowat, yunohost and yunohost-admin.

After managing to re-install them back too. My webadmin configurations seems to have disappeared since I’m getting the postinstall page when I navigate to my webadmin page.

Is it possible to get those configurations back since I wouldn’t want to have to regenerate all of those (and that’s a lot of configurations that I have made over the year) all over again?

The good news is that everything is still intact - my websites, apps, and even my SSO. It is just the webadmin that is not in it’s place. Any pointers to debug this would be appreciated!

Hmmm I’m not sure to understand 100% what happened, but if you are confident that stuff like your ldap database and so on are still there, you may try to tell yunohost that the postinstall was already done with :

touch /etc/yunohost/installed

Hopefully that should fix the issues

Wow :heart_eyes: The solution was so easy.

I was sure about this because I logged into a user’s account and got the SSO page.

All that was missing was to tell webadmin that everything was good - this was probably the only missing piece.
And it worked like a charm!
After touching the file, I got to the webadmin login page. I also skimmed through all of the top-level admin settings just to be sure and everything looks to be intact from the outside. My server is back online “completely in one piece” now.

Thanks for the tip @Aleks! :heart:

1 Like