YunoHost 3.7 spooky testing / Call for feedback

Announcements
, ,
1

Hello everyone :jack_o_lantern: !

We just released a new testing version for YunoHost and would be happy to receive feedback before releasing it as a stable version :yum:

This release includes quite important changes in YunoHost’s internal, and therefore requires careful validation to minimize the amount of remaining bugs.

The major changes are:

  • Group and permission mechanism. A new permission mechanism replaces the old one. With this new system, you should be able to create groups of users and allow only specific groups to access apps or features like email or xmpp. You will also be able to set (or unset) the app as public using a special group called visitors. You can manage these groups and permissions in the interface available under the Users category.

  • Some improvements to detect app installs that broke critical part of the system such as nginx or fail2ban. Also improved the handling of app installation failures or upgrades failures: for example, YunoHost will stop attempting to upgrade apps if one app fails to upgrade (otherwise we saw some report of catastrophic chain reactions).

  • Spookier warnings for dangerous app installations (meaning app with bad / questionnable quality or not from the Yunohost’s app catalog)

  • Quite a lot of messages improvements, string cleaning, language rework


  • We are getting rid of etckeeper because it causes too many issues compared to what it brings.

  • For app packagers : support for app manifests in toml, which in the long run should be more flexible and easy to write and manage (and consistent with the app action / panel stuff).

  • For developers: we worked on many tests that will help us to have more confidence in our changes, or identify possible regressions more easily in the future

  • Improved translations for Catalan, Occitan, French, Esperanto, Arabic, German, Spanish, Norwegian BokmĂ„l, Portuguese, Swedish

Thanks so much to all the contributors :heart: ! (advocatux, Aksel K., Aleks, Allan N., amirale qt, Armin P., Bram, ButterflyOfFire, Carles S. A., chema o. r., decentral1se, Emmanuel V., Etienne M., Filip B., Geoff M., htsr, Jibec, Josué, Julien J., Kayou, liberodark, ljf, lucaskev, Lukas D., madtibo, Martin D., Mélanie C., nr 458 h, pitfd, ppr, Quentí, sidddy, troll, tufek yamero, xaloc33, yalh76)

:hammer_and_wrench: Detailed changelog

Major changes

Smaller or pretty technical fix/enh

How to participate to the beta-testing :construction_worker_woman: :construction_worker_man:

:warning: Do not do this on a critical production server!

From the command line, you can launch the following command to switch to testing:

curl https://install.yunohost.org/switchToTesting | bash

(If you are familiar with bash scripting, you might want to read what this does before blindly running the command)

After this command, you should be running YunoHost 3.7.0.

What to test? :space_invader: :telescope:

Testing that the upgrades and corresponding migrations work correctly is already a significant step. After this, you can test and report feedback on the new group and permission interface. Please also validate that the permission you see listed are indeed effective.

In parallel, we also are testing apps to look for regressions.

6 Likes
3

Bonjour,

J’ai regardĂ© vite fait, c’est assez impressionnant !
En revanche voir ses applications (Strut, en ce qui me concerne) disponibles par un simple mondomaine.tld/strut c’est un peu effrayant.

J’ai testĂ© en ligne de commande les permissions et groupes “par dĂ©faut”

yunohost user permission list
permissions:
gitea.main:
allowed: all_users
leed.main:
allowed: all_users
mail.main:
allowed: all_users
nextcloud.main:
allowed: visitors
phpmyadmin.main:
allowed: monpremierutilisateur
rainloop.main:
allowed: all_users
redirect.main:
allowed: visitors
strut.main:
allowed: visitors
wemawema.main:
allowed: all_users
xmpp.main:
allowed: all_users

il y a un truc que je n’ai pas rĂ©ussi Ă  faire : sur phpmyadmin j’ai ajoutĂ© un " User specific permissions", et maintenant je ne sais pas comment faire pour l’enlever.

C’est un peu sommaire comme test, mais chez moi ça a l’air de bien fonctionner : applis et systùme.

1 Like
4

Tested also on my end, where I discovered that my Syncthing app was unprotected. :scream: I was able to fix that swiftly. As far as I saw, the functions work as I expected them to work.

Some UI remarks:

  • The switchToTesting script was initially aborting while requesting to upgrade some packages. An apt update and apt upgrade later, it was fixed.
  • While requesting an upgrade through the webadmin, one of the logs line is cut. It is about password migration for Postgresql.
Screenshots

Annotation%202019-11-01%20114426
Annotation 2019-11-01 114426.png1421×527 23.4 KB

image
image.png1422×813 32.1 KB

  • In the “Groups and permissions” page, collapsing the boxes initially did not work. I had to clear my cache.
  • We may need a visual confirmation that the user’s click on an app name has been acknowledged and is being processed. I was able to add twice Syncthing to my username. The duplicate was gone after refreshing the page. My suggestion would be to instantly close the drop-down menu after clicking on an item.
Screenshot

Annotation%202019-11-01%20121803
Annotation 2019-11-01 121803.png857×242 8.19 KB

Addendum 1:

  • There is also a Madam-Michu-unfriendly string on the app main configuration page: “[object Object] Visitors”
Screenshot

image
image.png1410×660 18.9 KB

Addendum 2:

  • The permission system can be tested with Grav, branch v1.6.16! Give users permission grav.admin or grav.user. :partying_face:
2 Likes
5

Bonjour,
Aprùs installation de la “testing”,
un apt autoremove me propose les suppressions suivantes :
1- etckeeper, ok !

2- iproute : est-ce normal ?

6

Bonjour,

J’ai essayĂ© d’exĂ©cuter le script sur mon Raspberry Pi, mais il semble que curl https://install.yunohost.org/switchToTesting | bash ne fonctionne pas.
Je ne passe pas en testing dans le sources.list.

root@***:~# yunohost --version
yunohost:
repo: now
version: 3.6.4.3
yunohost-admin:
repo: now
version: 3.6.4
moulinette:
repo: now
version: 3.6.4
ssowat:
repo: now
version: 3.6.4

root@***:~# curl https://install.yunohost.org/switchToTesting | bash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 459 100 459 0 0 1670 0 --:–:-- --:–:-- --:–:-- 1675
----
Patching sources.list to enable testing repository

----
----
Running ‘apt-get update’
----
Hit:1 Index of /raspbian stretch InRelease
Hit:2 http://repo.yunohost.org/debian stretch InRelease
Get:3 Index of /debian stretch InRelease [25.4 kB]
Fetched 25.4 kB in 1s (16.8 kB/s)
Reading package lists
 Done
----
Running ‘apt-get dist-upgrade’
----
Reading package lists
 Done
Building dependency tree
Reading state information
 Done
Calculating upgrade
 Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@***:~# cat /etc/apt/sources.list
deb Index of /raspbian stretch main contrib non-free rpi
# Uncomment line below then ‘apt-get update’ to enable ‘apt-get source’
#deb-src Index of /raspbian stretch main contrib non-free rpi

root@***:~# cat /etc/apt/sources.list.d/yunohost.list
deb http://repo.yunohost.org/debian/ stretch stable

Dois-je passer le stable Ă  testing manuellement dans /etc/apt/sources.list.d/yunohost.list ou bien une modification du script sera faite et Ă  re-tester ?

ppr

7

Oui, il y a aussi eu un changement de dĂ©pendance “trivial” (remplacĂ© par iproute2, qui etait deja installĂ© par iproute 
 bref, c’est normal)

Oui tu peux le faire manuellement aussi 
 ce serait interessant de tester pourquoi le script ne fonctionne pas dans ton cas, mais si tu regardes ce que le script fait, il est juste censé remplacer / ajouter testing manuellement a la fin de la ligne.

8

Le script ne pouvait pas faire le travail car j’avais repo au lieu de forge :

root@***:/home/admin# cat /etc/apt/sources.list.d/yunohost.list
#deb http://repo.yunohost.org/debian/ stretch testing
deb Index of /debian/ stretch testing

Je ne sais pas comment c’est arrivĂ© lĂ  
 YunoHost a Ă©tĂ© fraĂźchement installĂ© sur ma Raspberry Pi 3B afin de faire tourner CI Package Check 
 peut-ĂȘtre est-ce lĂ  le soucis.

Maintenant je suis bien en v3.7 Testing. Le script ajoute bien testing en fin de ligne dans le sources.list YunoHost.

Résumé

root@:/home/admin# cat /etc/apt/sources.list.d/yunohost.list
#deb http://repo.yunohost.org/debian/ stretch testing
deb Index of /debian/ stretch testing
root@:/home/admin# apt update && apt dist-upgrade && apt autoremove && apt autoclean && yunohost tools update && yunohost tools upgrade --apps && yunohost tools upgrade --system
Hit:1 Index of /raspbian stretch InRelease
Get:2 Index of /debian stretch InRelease [25.4 kB]
Hit:3 Index of /debian/ stretch InRelease
Fetched 25.4 kB in 1s (17.9 kB/s)
Reading package lists
 Done
Building dependency tree
Reading state information
 Done
4 packages can be upgraded. Run ‘apt list --upgradable’ to see them.
Reading package lists
 Done
Building dependency tree
Reading state information
 Done
Calculating upgrade
 Done
The following packages have been kept back:
moulinette ssowat yunohost yunohost-admin
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Reading package lists
 Done
Building dependency tree
Reading state information
 Done
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Reading package lists
 Done
Building dependency tree
Reading state information
 Done
Info: Fetching available upgrades for system packages

Info: Fetching available upgrades for applications

Success! The application list yunohost has been fetched
apps:
system:
0:
current_version: 3.6.4
name: moulinette
new_version: 3.7.0
1:
current_version: 3.6.4
name: ssowat
new_version: 3.7.0
2:
current_version: 3.6.4.3
name: yunohost
new_version: 3.7.0.1
3:
current_version: 3.6.4
name: yunohost-admin
new_version: 3.7.0
Info: No apps to upgrade
Info: Upgrading packages

Info: Now upgrading ‘special’ (yunohost-related) packages 

Warning: This action will end but the actual special upgrade will continue in background. Please don’t start any other action on your server in the next ~10 minutes (depending on your hardware speed). Once it’s done, you may have to re-log on the webadmin. The upgrade log will be available in Tools > Log (in the webadmin) or through ‘yunohost log list’ (in command line).
root@***:/home/admin# Running scope as unit: run-rb5cf8953f8314a859abd4dd47c7c03fe.scope
Reading package lists

Building dependency tree

Reading state information

Calculating upgrade

The following packages were automatically installed and are no longer required:
etckeeper iproute
Use ‘apt autoremove’ to remove them.
The following packages will be upgraded:
moulinette ssowat yunohost yunohost-admin
4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.2 MB of archives.
After this operation, 625 kB of additional disk space will be used.
Get:1 Index of /debian/ stretch/testing armhf moulinette all 3.7.0 [67.5 kB]
Get:2 Index of /debian/ stretch/testing armhf ssowat all 3.7.0 [744 kB]
Get:3 Index of /debian/ stretch/testing armhf yunohost all 3.7.0.1 [787 kB]
Get:4 Index of /debian/ stretch/testing armhf yunohost-admin all 3.7.0 [8,575 kB]
Fetched 10.2 MB in 1s (6,213 kB/s)
(Reading database 
 61999 files and directories currently installed.)
Preparing to unpack 
/moulinette_3.7.0_all.deb 

Unpacking moulinette (3.7.0) over (3.6.4) 

Setting up moulinette (3.7.0) 

(Reading database 
 62003 files and directories currently installed.)
Preparing to unpack 
/archives/ssowat_3.7.0_all.deb 

Unpacking ssowat (3.7.0) over (3.6.4) 

Setting up ssowat (3.7.0) 

(Reading database 
 62004 files and directories currently installed.)
Preparing to unpack 
/yunohost_3.7.0.1_all.deb 

Unpacking yunohost (3.7.0.1) over (3.6.4.3) 

Setting up yunohost (3.7.0.1) 


Configuration file ‘/etc/dpkg/origins/yunohost’, does not exist on system.
Installing new config file as you requested.
Regenerating configuration, this might take a while

Success! Configuration for category ‘metronome’ updated
Warning: The configuration file ‘/etc/postfix/main.cf’ has been manually modified and will not be updated
Success! Configuration for category ‘dnsmasq’ updated
Success! Configuration for category ‘fail2ban’ updated
Success! Configuration for category ‘dovecot’ updated
Success! Configuration for category ‘slapd’ updated
Launching migrations

Info: Running migration 0011_setup_group_permission

Info: Creating a backup of LDAP database and apps settings prior to the actual migration.
Info: Updating LDAP schema

Info: Updating LDAP database

Info: Creating a group for each user

Info: Migrating permissions from apps settings to LDAP

Info: Migration completed. You are now able to manage usergroups.
Success! Migration 0011_setup_group_permission completed
Info: Running migration 0012_postgresql_password_to_md5_authentication

Success! Migration 0012_postgresql_password_to_md5_authentication completed
Restarting YunoHost firewall

(Reading database 
 62013 files and directories currently installed.)
Preparing to unpack 
/yunohost-admin_3.7.0_all.deb 

Unpacking yunohost-admin (3.7.0) over (3.6.4) 

Processing triggers for systemd (232-25+deb9u12) 

Setting up yunohost-admin (3.7.0) 

Processing triggers for man-db (2.7.6.1-2) 

Done!

YunoHost package upgrade completed !
Press [Enter] to get the command line back

root@***:/home/admin#

ppr

9

My server has just upgraded to the new testing (with unattended_upgrade) strangely yunohost-admin was not in testing mode after that (all other package was)

So i rerun manually

apt update
apt-full-upgrade

I note that my mail users are not recognized:

<xxxx@yyyyy> (expanded from <zzzz@aaaa>): host yyyyyyyy[89.234.141.xxx] said: 550 5.1.1 <xxxx@yyyyy>: Recipient address rejected: User unknown in virtual mailbox table (in reply to RCPT TO command)

It might be a problem of ipv6 connectivity breaking the upgrade (my vpn connexion flap every hour since 3 days
). Or it could be a bug in this testing.

It’s because the migration about group and permission was not executed automatically.

10

In fact the migration fails that’why it’s not done:

“La migration 0011_setup_group_permission a Ă©chouĂ© avec l’exception Le groupe XXXX est inconnu : annulation”

11

So the issue is that i have permission on users that doesn’t exist anymore, but the permission are still here in /etc/yunohost/apps/*/settings.yml (allowed_users) https://github.com/YunoHost/yunohost/blob/stretch-unstable/src/yunohost/data_migrations/0011_setup_group_permission.py#L106

12

We also need to add some loading indicator when we click to delete/add a permission on web admin. On my lime2 it’ hawfully slow, may be due to my connectivity issue) and it’s truelly strange to click and the permission disappear after 20s


13

Euh, ça fait beaucoup de testing :smile:

#cat /etc/apt/sources.list.d/yunohost.list
deb http://forge.yunohost.org/debian/ stretch stable testing testing testing testing
1 Like
14

Bonjour,
J’ai mis l’application redirect dans le groupe Visitors.
Du coup :

  • mondomaine/redirect OK
  • Ă  travers l’interface Yunohost/SSO : KO !
    Est-ce voulu ?
15

Tu veux dire que en Ă©tant logguĂ© tu ne peux plus y accĂ©der ? Ou juste que la tuile n’apparait pas ?

(En tout cas apriori nope, ça a pas l’air voulu)

16

Bonjour @Aleks,
La tuile apparaüt bien, mais la redirection ne s’effectue pas en cliquant dessus. On dirait que ça mouline !
Les autres tuiles/applications fonctionnent bien.
J’essaierai, dĂšs que possible, avec mes autres applis que je passerai en «Visitors» pour tester.
DĂ©solĂ©, j’aurais du ĂȘtre plus explicite.

17

I also had the exact same issue during the upgrade. An old user that was deleted long ago, but still present in the settings.yml.

I had to re-run the ldap migration manually

18

Today, one week after the upgrade, nobody can log into the portal, and I cannot ssh into the server as admin, because of a connection to the ldap server.
I restarted both ldap and nslcd. Connection was possible again after the nslcd restart.

Going back in history I see that nslcd was logging connection error to the ldap server since one week, so since the upgrade to 3.7
I’m not sure how to interpret slapd logs : I had regular logging messages since the upgrade, but with suspicious lines like:
Nov 05 12:45:58 ynh slapd[13074]: connection_read(25): no connection!

Now everything seems normal.
slapd still logs the following new warnings/errors since the migration :

Nov 06 10:54:54 ynh slapd[8146]: <= mdb_equality_candidates: (permission) not indexed
Nov 06 11:09:33 ynh slapd[8146]: slap_global_control: unrecognized control: 1.3.6.1.4.1.4203.666.5.16

For the rest: congrats!

1 Like
19

Ah, maybe we should add a nslcd restart after the migration ? (Those stuff are always quite mysterious 
) But now that you mention it, maybe that’s also what’s behind these weird issue where we see people failing to authenticate using a valid password on the webadmin 


Ah yes maybe we need to add this in indexed fields in the configuration (dunno where exactly but I remember the PR)

This one though, I think it’s unrelated to 3.7, but I don’t know the cause behind it 


20

Je m’ auto-rĂ©ponds !
Avec les autres applis je n’ ai pas le mĂȘme problĂšme.
Je pense que cela peut venir de redirect_ynh, voici mon log à l’installation:

root@domain:~#yunohost app install redirect
Available domains:
-domain.tld
Choose a domain for your redirect (default: domain.tld):
Choose a path for your redirect (default: /redirect):
Redirect destination path (default: http://127.0.0.1): http://someotherdomain.tld
Redirect type [public_302 | public_301 | public_proxy | private_proxy] (default: public_302):
Info: Installing the app ‘redirect’

Warning: /!\ Packagers! This app is still using the skipped/protected/unprotected_uris/regex settings which are now obsolete and deprecated
 Instead, you should use the new helpers ‘ynh_permission_{create,urls,update,delete}’ and the ‘visitors’ group to initialize the public/private access. Check out the documentation at the bottom of Users groups and permissions | Yunohost Documentation to learn how to use the new permission mechanism.
Success! Installation completed
root@domain:~#