Alpha-testing phase for YunoHost on Debian Stretch

Aleks · March 9, 2018, 1:30pm

Dear all,

following some work and development in YunoHost, we are now reading to open an alpha-testing phase for YunoHost’s support and migration to Debian Stretch !

We need your feedback to test and debug this delicate operation, such that people with not much technical knowledge will be able to enjoy a migration as smooth and transparent as possible !

Keep in mind that alpha means that bugs are to be expected ! The system will be unstable ! There’s a high chance that we will make strong structural changes in the various piece of software / configuration. Therefore, do not perform the following steps just because you absolutely want to have a YunoHost on Stretch in production !

The following test cases describe specific items we would like to get feedback on (in particular #1 and #2 for now). Feel free to pick one and tell us what you found !

Test case #1 - Install on a fresh system

Goal: test YunoHost on a newly installed system (Debian Stretch)
Difficulty: ~easy (if you are already familiar with YunoHost install process)
Steps:

Setup a machine with Debian Stretch. Either on a VPS, an ARM board or a regular computer ;
Connect as root on the machine ;
Launch the install of YunoHost by running (be careful with the syntax) :

bash <(wget -q -O- https://install.yunohost.org/stretch)

(If wget is not installed on the system, install it with apt-get install wget)

What to test: test that the install works as expected. After that, try to do stuff that you would personally do with a real-life instance, e.g. browse the webadmin, create users, install and play with apps, install certificates, send/receive emails, chat on XMPP, …

Test case #2 - Migrate a recently installed system from Jessie to Stretch

Goal: test YunoHost’s “automatic” migration from a recently installed Jessie to Stretch
Difficulty: ~medium (bit more technical that test case #1)
Steps:

Setup a machine with a YunoHost on Debian Jessie, using the current install instructions available on https://yunohost.org/#/install.
Do a few things that you would do on a real instance : e.g. create a user, install a few apps…
Run this script ( Do not run it on a production server !!! ). This will switch your instance to the unstable branch containing the Stretch migration :

bash <(wget -q -O- https://install.yunohost.org/switchToUnstable)

In the command line, run yunohost tools migrations migrate and read what it tells you.

What to test: try to be attentive to what’s happening during the migration. Report anything you find odd in term of UX… After the migration is done, try to make sure that everything worked. In particular, fail2ban should be running, your previously installed apps should still be accessible, and yunohost --version should tell you that you’re in 3.x.

Test case #3 - Migrate a prod-like instance from Jessie to Stretch

Goal: test YunoHost’s “automatic” migration on a “real-life instance”
Difficulty: ~hard (requires some heavy technical manipulations)
Steps:

Start with a “real-life” production instance that you have. For instance, a Raspberry Pi which got initially installed with YunoHost 2.2 with a few custom apps and/or manual modifications.
Make a clone of it … E.g. if this is an ARM board, make a clone of the SD card. For a regular computer or a VPS, it’s a bit trickier, but maybe achievable with rsync… [To be completed]
Follow same steps as test case #2 on your clone.

What to test: same as test case #2.

Aleks · March 10, 2018, 5:21pm

Josue · March 11, 2018, 11:37am

Hello,

I have made a test of migration with some apps. I found some small issue, but I don’t have tested everything.

Firstly, one apps-deps package has been removed, I don’t really know why, but probably because it has some php dependence. In this part of log we could see that horde-ynh-deps is removed.

1681302 DEBUG Running apt command :
DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=none apt-get --fix-broken --show-upgraded --assume-yes -o Dpkg::Options::="--force-confnew" -o Dpkg::Options::="--force-confmiss" -o Dpkg::Options::="--force-confdef" dist-upgrade
Lecture des listes de paquets…
Construction de l'arbre des dépendances…
Lecture des informations d'état…
Calcul de la mise à jour…
Les paquets suivants ont été installés automatiquement et ne sont plus nécessaires :
 comerr-dev g++-4.9 gobjc-4.9 imagemagick-common krb5-multidev
 libalgorithm-c3-perl libarchive-extract-perl libbind9-90 libclass-c3-perl
 libclass-c3-xs-perl libcpan-meta-perl libdata-optlist-perl
 libdata-section-perl libdns100 libfftw3-double3 libgssrpc4 libintl-perl
 libintl-xs-perl libisc95 libisccc90 libisccfg90 libjasper1 libjs-jquery
 libkadm5clnt-mit11 libkadm5srv-mit11 libkdb5-8 libllvm3.5
 liblog-message-perl liblog-message-simple-perl liblqr-1-0 liblwres90
 libmodule-build-perl libmodule-load-conditional-perl
 libmodule-pluggable-perl libmodule-runtime-perl libmodule-signature-perl
 libmro-compat-perl libobjc-4.9-dev libpackage-constants-perl
 libparams-classify-perl libparams-util-perl libperl4-corelibs-perl
 libpgm-5.1-0 libpth20 librrd4 libslp1 libsodium13 libsoftware-license-perl
 libsub-exporter-perl libsub-install-perl libterm-ui-perl
 libtext-template-perl libtidy-0.99-0 libuuid-perl libwebp5 libwebpdemux1
 libwebpmux1 libxapian22 libzmq3 linux-image-3.16.0-4-amd64 php5-tidy
 python-bs4 python-cffi python-colorama python-distlib python-html5lib
 python-lxml python-ndg-httpsclient python-ply python-pycparser
 python-support python-webencodings python3-chardet python3-colorama
 python3-distlib python3-html5lib python3-requests python3-urllib3
 python3-webencodings ttf-dejavu-core
Veuillez utiliser « sudo apt autoremove » pour les supprimer.
Les paquets suivants seront ENLEVÉS :
 horde-ynh-deps libmagickcore-6.q16-2 libmagickwand-6.q16-2 php5-imagick
Les NOUVEAUX paquets suivants seront installés :
 python3-pyinotify python3-systemd
Les paquets suivants ont été conservés :
 imagemagick-common moulinette ssowat yunohost yunohost-admin
Les paquets suivants seront mis à jour :
 fail2ban

If we reinstall this package after the migration, we don’t have any dependence problem with this package on stretch.

I found also that fail2ban didn’t start after the migration. I need to type yunohost service regen-conf -f to fix this issue.

I have also just a question : Is the postgresql migration already implemented ?

All logs of the migration is available here.

Taekiro · March 11, 2018, 3:30pm

Hi,

I have tested the case #2, and I ran into some issue. I installed Debian Jessie, Yunohost and 4 apps (NetData, NextCloud (on a subdomain), rainloop, and transmission) no problem. I ran the switchToUnstable script, then the yunohost migrations migrate --accept-disclaimer command and it gave me the following issue :

The following packages have unmet dependencies:
yunohost : Depends: unscd but it is not going to be installed
Depends: rspamd (>= 1.6.0) but 1.3.5-1 is to be installed
Recommends: php-gd
Recommends: php-curl
Recommends: php-mcrypt
E: Unable to correct problems, you have held broken packages.

yunohost --version gave me :

yunohost:
repo: now
version: 2.7.10+2018.03.10.00~stretchmigration
yunohost-admin:
repo: now
version: 2.7.7+2018.02.11.22.47
moulinette:
repo: now
version: 2.7.7+2018.02.11.22.40
ssowat:
repo: now
version: 2.7.7+2018.02.11.22.42

apt full-upgrade did upgrade moulinette, ssowat and yunohost-admin but not yunohost. I upgraded rspamd through the instructions given on their website (apt install rspamd gave me rspamd is already the newest version (1.3.5-1).) and then I was able to upgrade yunohost to version 3.x . It gave me the following :

Warning: The configuration file ‘/etc/nslcd.conf’ has been manually modified and will not be updated
Warning: The configuration file ‘/etc/postfix/master.cf’ has been manually modified and will not be updated
Success! The configuration has been updated for service 'rspamd’
Success! The configuration has been updated for service 'dnsmasq’
Warning: The configuration file ‘/etc/fail2ban/filter.d/yunohost.conf’ has been manually removed and will not be created
Warning: The configuration file ‘/etc/mysql/my.cnf’ has been manually modified and will not be updated
Success! The configuration has been updated for service 'dovecot’
Warning: Job for fail2ban.service failed because the control process exited with error code.
Warning: See “systemctl status fail2ban.service” and “journalctl -xe” for details.
Error: Script execution failed: /usr/share/yunohost/hooks/conf_regen/52-fail2ban
Launching migrations…
Warning: Running migration 4 php5_to_php7_pools…
Restarting YunoHost firewall…
Setting up php-curl (1:7.0+49) …
Processing triggers for php7.0-fpm (7.0.27-0+deb9u1) …

I restarted the computer, was able to connect to the sso and the web admin but fail2ban was marked as inactive. So I tried the yunohost service regen-conf -f command, which worked for fail2ban but now I have 502 errors when I try to access NextCloud or Rainloop, and NetData doesn’t load properly the page (transmission seems to work fine). I also tried to install a new app which seems to work fine.

Hope that help. I can retry from the start with new instructions if needed.

PS. I just realised reading @Josue logs that rspamd is supposed to be upgraded from the http://vinaigrette.yunohost.org repo, no idea why it didn’t do that.

ppr · March 11, 2018, 5:59pm

Hi @Aleks ,

A quick feedback about a test case#1 on a Raspberry Pi 3 based on NOOBS Lite Stretch :

sudo apt update && sudo apt dist-upgrade && sudo apt autoremove && sudo apt autoclean
sudo rpi-update #because without it installation failed on Raspberry Pi 1B
sudo passwd root
reboot
logon as root
bash <(wget -q -O- https://install.yunohost.org/stretch) : OK

logon as root
yunohost tools postintall
alpha-test-case1-rpi3.nohost.me #To delete as soon as possible
Create a single user : OK

Install Wallabag2:

Install Wallabag2 by the panel not tested CLI : OK

Install Custom Webapp :

Install Custom Webapp (public+datat base) by the panel not tested CLI : NOK
#pacman reload page F5 installation failed
CLI : raspi-config => Menu 5 - P2 : Enable SSH
Install Custom Webapp (public+datat base) by the panel not tested CLI : OK after enable SSH

reboot

Install LE certificate for alpha-test-case1-rpi3.nohost.me :

Install Let’s Encrypt certificate : OK by the panel not tested CLI

Install NextCloud

Install NextCloud : NOK
/etc/cron.d/nextcloud wasn’t deleted because it doesn’t exist.
php7.0-fpm.service is not active, cannot reload.
#pacman reload page F5 installation failed

ppr

Aleks · March 12, 2018, 12:16pm

Hey,

thanks everyone for the precious and prompt feedback ! I’ll have a look when I can at the different issues found…

@Taekiro : yes indeed, the whole rspamd thing was supposed to be handled automatically Not sure why it did not want to run the upgrade (did it do a big upgrade before you encountered the The following packages have unmet dependencies ?)

Nope, not yet … it’s still an open ticket in the stretch roadmap

Taekiro · March 12, 2018, 5:45pm

It did do a big upgrade before encountering the unmet dependencies message. Here are the last lines :

W: APT had planned for dpkg to do more than it reported back (752 vs 756).
Affected packages: fail2ban:amd64
Warning: Starting the yunohost package upgrade … The migration will end, but the actual upgrade will happen right after. After the operation is complete, you might have to re-log on the webadmin.
Canceled hold on yunohost.
Canceled hold on yunohost-admin.
Canceled hold on moulinette.
Canceled hold on ssowat.
Warning: Activating upgrade of yunohost packages, to be ran right after this command ends.
root@*********************:~# Reading package lists…
Building dependency tree…
Reading state information…
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
yunohost : Depends: unscd but it is not going to be installed
Depends: rspamd (>= 1.6.0) but 1.3.5-1 is to be installed
Recommends: php-gd
Recommends: php-curl
Recommends: php-mcrypt
E: Unable to correct problems, you have held broken packages.

Shiro · March 15, 2018, 10:36am

Same kind of error for NextCloud (case #1, no problem with user creation or certificates, terminal).
Here is the “verbose” log, just in case: https://cloud.fluffy.kiwi/index.php/s/qffxGs6lPztKnbV .

Shiro · March 15, 2018, 10:43am

Aaaaand, installing this package https://packages.debian.org/sid/php-zip solves the issue (but now, error 500 at login).

Aleks · March 15, 2018, 10:44am

Does installing php-ldap helps ?

Shiro · March 15, 2018, 10:45am

Sorry, it does not

php-ldap is already the newest version (1:7.0+49).

edit :
if it helps: php log
[15-Mar-2018 10:46:34] WARNING: [pool nextcloud] child 688 said into stderr: "ERROR: Unable to set php_value 'always_populate_raw_post_data'"

Shiro · March 15, 2018, 11:02am

If I re-install Nextcloud in the root folder (test.mochi.academy instead of test.mochi.academt/cloud/) it works

Aleks · March 16, 2018, 9:20pm

Hey,

thanks a lot again for the feedback !

I made a few changes / upgrades to fix a few reported issue, namely :

The whole fail2ban should now be fixed (among with other things i forgot)
The The following packages have unmet dependencies should be fixed (it was an issue with the switchToUpgrade script)

The following issues are not yet fixed / understood though :

Nextcloud install (c.f. some discussion happening in https://github.com/YunoHost-Apps/nextcloud_ynh/pull/96 )
The reason why horde-ynh-deps (for instance) gets uninstalled is still not understood

thomas · March 25, 2018, 7:59am

I’m late to the party but I just made a fresh install on Stretch and so far no problems at all !
I use Nextcloud, Wallabag and FreshRss and everything is ok.

The only thing i tried that didn’t work was the Linux Dash package. The page loads ok but stays blank with “Linux Dash” on top.

Don’t know if it helps but sometime it’s good to know that everything is working

Salamandar · March 25, 2018, 5:55pm

There’s a problem with the Dovecot/NGinx installation.

root@raspberrypi:~# bash ./stretch 
[INFO] Running upgrade_system
[INFO] Running install_script_dependencies
[INFO] Running create_custom_config
[INFO] Running confirm_installation
[INFO] Running setup_package_source
[INFO] Running apt_update
[INFO] Running register_debconf
[INFO] Running workaround_avahi_installation
[INFO] User avahi already exists (with uid uid=108(avahi) gid=112(avahi) groups=112(avahi)), skipping avahi workaround
[INFO] Running install_yunohost_packages
[FAIL] Installation of Yunohost packages failed
[INFO] Installation logs are available in /var/log/yunohost-installation_20180325_173931.log
root@raspberrypi:~# apt install -f
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
7 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
[master d4ed5b6] saving uncommitted changes in /etc prior to apt run
 4 files changed, 3 insertions(+), 3 deletions(-)
 delete mode 120000 systemd/system/dovecot.service
 delete mode 120000 systemd/system/dovecot.socket
Setting up dovecot-core (1:2.2.27-3+deb9u2) ...
Job for dovecot.service failed because the control process exited with error code.
See "systemctl status dovecot.service" and "journalctl -xe" for details.
invoke-rc.d: initscript dovecot, action "start" failed.
● dovecot.service - Dovecot IMAP/POP3 email server
   Loaded: loaded (/lib/systemd/system/dovecot.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2018-03-25 17:46:44 UTC; 28ms ago
     Docs: man:dovecot(1)
           http://wiki2.dovecot.org/
  Process: 28685 ExecStart=/usr/sbin/dovecot (code=exited, status=89)
 Main PID: 22366 (code=exited, status=0/SUCCESS)

Mar 25 17:46:44 raspberrypi systemd[1]: Starting Dovecot IMAP/POP3 email server...
Mar 25 17:46:44 raspberrypi dovecot[28685]: Error: socket() failed: Address family not supported by protocol
Mar 25 17:46:44 raspberrypi dovecot[28685]: Error: service(imap-login): listen(::, 143) failed: Address family not supported by protocol
Mar 25 17:46:44 raspberrypi dovecot[28685]: Fatal: Failed to start listeners
Mar 25 17:46:44 raspberrypi systemd[1]: dovecot.service: Control process exited, code=exited status=89
Mar 25 17:46:44 raspberrypi systemd[1]: Failed to start Dovecot IMAP/POP3 email server.
Mar 25 17:46:44 raspberrypi systemd[1]: dovecot.service: Unit entered failed state.
Mar 25 17:46:44 raspberrypi systemd[1]: dovecot.service: Failed with result 'exit-code'.
dpkg: error processing package dovecot-core (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of dovecot-managesieved:
 dovecot-managesieved depends on dovecot-core (= 1:2.2.27-3+deb9u2); however:
  Package dovecot-core is not configured yet.

dpkg: error processing package dovecot-managesieved (--configure):
 dependency problems - leaving unconfigured
Setting up nginx-extras (1.10.3-1+deb9u1) ...
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xe" for details.
invoke-rc.d: initscript nginx, action "start" failed.
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2018-03-25 17:46:45 UTC; 29ms ago
     Docs: man:nginx(8)
  Process: 28717 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)

Mar 25 17:46:44 raspberrypi systemd[1]: Starting A high performance web server and a reverse proxy server...
Mar 25 17:46:45 raspberrypi nginx[28717]: nginx: [emerg] socket() [::]:80 failed (97: Address family not supported by protocol)
Mar 25 17:46:45 raspberrypi nginx[28717]: nginx: configuration file /etc/nginx/nginx.conf test failed
Mar 25 17:46:45 raspberrypi systemd[1]: nginx.service: Control process exited, code=exited status=1
Mar 25 17:46:45 raspberrypi systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Mar 25 17:46:45 raspberrypi systemd[1]: nginx.service: Unit entered failed state.
Mar 25 17:46:45 raspberrypi systemd[1]: nginx.service: Failed with result 'exit-code'.
dpkg: error processing package nginx-extras (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of dovecot-antispam:
 dovecot-antispam depends on dovecot-abi-2.2.abiv27; however:
  Package dovecot-abi-2.2.abiv27 is not installed.
  Package dovecot-core which provides dovecot-abi-2.2.abiv27 is not configured yet.

dpkg: error processing package dovecot-antispam (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ssowat:
 ssowat depends on nginx-extras (>= 1.6.2); however:
  Package nginx-extras is not configured yet.

dpkg: error processing package ssowat (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of yunohost:
 yunohost depends on ssowat (>= 2.7.1); however:
  Package ssowat is not configured yet.
 yunohost depends on dovecot-managesieved; however:
  Package dovecot-managesieved is not configured yet.
 yunohost depends on dovecot-antispam; however:
  Package dovecot-antispam is not configured yet.
 yunohost depends on nginx-extras (>= 1.6.2); however:
  Package nginx-extras is not configured yet.

dpkg: error processing package yunohost (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of yunohost-admin:
 yunohost-admin depends on yunohost (>= 2.7.1); however:
  Package yunohost is not configured yet.

dpkg: error processing package yunohost-admin (--configure):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 dovecot-core
 dovecot-managesieved
 nginx-extras
 dovecot-antispam
 ssowat
 yunohost
 yunohost-admin
E: Sub-process /usr/bin/dpkg returned an error code (1)

The log file shows pretty much the same errors as apt-get.

I’m on a fresh Raspbian Stretch install on a Pi3b+.
Yesterday I did a “dirty” install of the stretch script after the “stable” install (that didn’t run properly of course), I did not get this error.

That seems to be related to IPv6 not being available now. /proc/sys/net/ipv6 does not exist.

Aleks · March 25, 2018, 6:04pm

Uh yeah that’s really weird … did you tinker anything before installing it ? As far as I know, IPv6 should be available on RPi images :s …

Salamandar · March 25, 2018, 7:54pm

I stumbled on some weird things, more related to Raspberry Pi handling the kernel update that to Yunohost.
I did reinstall Raspbian but forgot to wipe the boot partition (root partition on usb disk). The boot partition still contained linux414 images from rpi-update. But the root partition contained linux49 headers and the modules were on the wrong directory.

That said you’ll understand why ipv6 wasn’t available…

I don’t understand how they can expect things to work when installing 4.14 manually but still providing 4.9 in the repository…

EDIT: Worse : with uname returning 4.14, rpi-update didn’t detect the headers/modules were in the wrong place and didn’t update anything as it should have.

Aleks · March 25, 2018, 8:01pm

Uh okay, that indeed doesn’t sound related to YunoHost itself … But when flashing your RPi image, why are you not flashing the whole device instead of just one partition ? o.O

Salamandar · March 25, 2018, 8:04pm

Because I moved the root partition to an USB SSD.
Originally I flashed the SDCard then rsync-ed the root partition to the ssd.
When reinstalling I re-did the rsync but did not re-create the boot partition. Fail. ^^

EDIT : I confirm that re-installing correctly fixes everything. Sorry for the false alert !

thomas · March 30, 2018, 4:28pm

Problem on las update: when trying to go in admin section, the first run wizard appears and it’s impossible to go through it.

When I run: yunohost tools postinstall in CLI i give my domain and my actual admin password and all I got is “Invalid password”

Edit: Thanks to @Aleks it is now solved with the creation of /etc/yunohost/installed:
touch /etc/yunohost/installed