YunoHost 3.7 spooky testing / Call for feedback

Hello everyone :jack_o_lantern: !

We just released a new testing version for YunoHost and would be happy to receive feedback before releasing it as a stable version :yum:

This release includes quite important changes in YunoHost’s internal, and therefore requires careful validation to minimize the amount of remaining bugs.

The major changes are:

  • Group and permission mechanism. A new permission mechanism replaces the old one. With this new system, you should be able to create groups of users and allow only specific groups to access apps or features like email or xmpp. You will also be able to set (or unset) the app as public using a special group called visitors. You can manage these groups and permissions in the interface available under the Users category.

  • Some improvements to detect app installs that broke critical part of the system such as nginx or fail2ban. Also improved the handling of app installation failures or upgrades failures: for example, YunoHost will stop attempting to upgrade apps if one app fails to upgrade (otherwise we saw some report of catastrophic chain reactions).

  • Spookier warnings for dangerous app installations (meaning app with bad / questionnable quality or not from the Yunohost’s app catalog)

  • Quite a lot of messages improvements, string cleaning, language rework…

  • We are getting rid of etckeeper because it causes too many issues compared to what it brings.

  • For app packagers : support for app manifests in toml, which in the long run should be more flexible and easy to write and manage (and consistent with the app action / panel stuff).

  • For developers: we worked on many tests that will help us to have more confidence in our changes, or identify possible regressions more easily in the future

  • Improved translations for Catalan, Occitan, French, Esperanto, Arabic, German, Spanish, Norwegian Bokmål, Portuguese, Swedish

Thanks so much to all the contributors :heart: ! (advocatux, Aksel K., Aleks, Allan N., amirale qt, Armin P., Bram, ButterflyOfFire, Carles S. A., chema o. r., decentral1se, Emmanuel V., Etienne M., Filip B., Geoff M., htsr, Jibec, Josué, Julien J., Kayou, liberodark, ljf, lucaskev, Lukas D., madtibo, Martin D., Mélanie C., nr 458 h, pitfd, ppr, Quentí, sidddy, troll, tufek yamero, xaloc33, yalh76)

:hammer_and_wrench: Detailed changelog

Major changes

Smaller or pretty technical fix/enh

How to participate to the beta-testing :construction_worker_woman: :construction_worker_man:

:warning: Do not do this on a critical production server!

From the command line, you can launch the following command to switch to testing:

curl https://install.yunohost.org/switchToTesting | bash

(If you are familiar with bash scripting, you might want to read what this does before blindly running the command)

After this command, you should be running YunoHost 3.7.0.

What to test? :space_invader: :telescope:

Testing that the upgrades and corresponding migrations work correctly is already a significant step. After this, you can test and report feedback on the new group and permission interface. Please also validate that the permission you see listed are indeed effective.

In parallel, we also are testing apps to look for regressions.

4 Likes

Bonjour,

J’ai regardé vite fait, c’est assez impressionnant !
En revanche voir ses applications (Strut, en ce qui me concerne) disponibles par un simple mondomaine.tld/strut c’est un peu effrayant.

J’ai testé en ligne de commande les permissions et groupes “par défaut”

#yunohost user permission list
permissions:
gitea.main:
allowed: all_users
leed.main:
allowed: all_users
mail.main:
allowed: all_users
nextcloud.main:
allowed: visitors
phpmyadmin.main:
allowed: monpremierutilisateur
rainloop.main:
allowed: all_users
redirect.main:
allowed: visitors
strut.main:
allowed: visitors
wemawema.main:
allowed: all_users
xmpp.main:
allowed: all_users

il y a un truc que je n’ai pas réussi à faire : sur phpmyadmin j’ai ajouté un " User specific permissions", et maintenant je ne sais pas comment faire pour l’enlever.

C’est un peu sommaire comme test, mais chez moi ça a l’air de bien fonctionner : applis et système.

1 Like

Tested also on my end, where I discovered that my Syncthing app was unprotected. :scream: I was able to fix that swiftly. As far as I saw, the functions work as I expected them to work.

Some UI remarks:

  • The switchToTesting script was initially aborting while requesting to upgrade some packages. An apt update and apt upgrade later, it was fixed.
  • While requesting an upgrade through the webadmin, one of the logs line is cut. It is about password migration for Postgresql.
Screenshots


  • In the “Groups and permissions” page, collapsing the boxes initially did not work. I had to clear my cache.
  • We may need a visual confirmation that the user’s click on an app name has been acknowledged and is being processed. I was able to add twice Syncthing to my username. The duplicate was gone after refreshing the page. My suggestion would be to instantly close the drop-down menu after clicking on an item.
Screenshot

Addendum 1:

  • There is also a Madam-Michu-unfriendly string on the app main configuration page: “[object Object] Visitors”
Screenshot

Addendum 2:

  • The permission system can be tested with Grav, branch v1.6.16! Give users permission grav.admin or grav.user. :partying_face:
2 Likes

Bonjour,
Après installation de la “testing”,
un apt autoremove me propose les suppressions suivantes :
1- etckeeper, ok !

2- iproute : est-ce normal ?

Bonjour,

J’ai essayé d’exécuter le script sur mon Raspberry Pi, mais il semble que curl https://install.yunohost.org/switchToTesting | bash ne fonctionne pas.
Je ne passe pas en testing dans le sources.list.

root@***:~# yunohost --version
yunohost:
repo: now
version: 3.6.4.3
yunohost-admin:
repo: now
version: 3.6.4
moulinette:
repo: now
version: 3.6.4
ssowat:
repo: now
version: 3.6.4

root@***:~# curl https://install.yunohost.org/switchToTesting | bash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 459 100 459 0 0 1670 0 --:–:-- --:–:-- --:–:-- 1675
----
Patching sources.list to enable testing repository…
----
----
Running ‘apt-get update’
----
Hit:1 http://raspbian.raspberrypi.org/raspbian stretch InRelease
Hit:2 http://repo.yunohost.org/debian stretch InRelease
Get:3 http://archive.raspberrypi.org/debian stretch InRelease [25.4 kB]
Fetched 25.4 kB in 1s (16.8 kB/s)
Reading package lists… Done
----
Running ‘apt-get dist-upgrade’
----
Reading package lists… Done
Building dependency tree
Reading state information… Done
Calculating upgrade… Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@***:~# cat /etc/apt/sources.list
deb http://raspbian.raspberrypi.org/raspbian/ stretch main contrib non-free rpi
# Uncomment line below then ‘apt-get update’ to enable ‘apt-get source’
#deb-src http://raspbian.raspberrypi.org/raspbian/ stretch main contrib non-free rpi

root@***:~# cat /etc/apt/sources.list.d/yunohost.list
deb http://repo.yunohost.org/debian/ stretch stable

Dois-je passer le stable à testing manuellement dans /etc/apt/sources.list.d/yunohost.list ou bien une modification du script sera faite et à re-tester ?

ppr

Oui, il y a aussi eu un changement de dépendance “trivial” (remplacé par iproute2, qui etait deja installé par iproute … bref, c’est normal)

Oui tu peux le faire manuellement aussi … ce serait interessant de tester pourquoi le script ne fonctionne pas dans ton cas, mais si tu regardes ce que le script fait, il est juste censé remplacer / ajouter testing manuellement a la fin de la ligne.

Le script ne pouvait pas faire le travail car j’avais repo au lieu de forge :

root@***:/home/admin# cat /etc/apt/sources.list.d/yunohost.list
#deb http://repo.yunohost.org/debian/ stretch testing
deb http://forge.yunohost.org/debian/ stretch testing

Je ne sais pas comment c’est arrivé là … YunoHost a été fraîchement installé sur ma Raspberry Pi 3B afin de faire tourner CI Package Check … peut-être est-ce là le soucis.

Maintenant je suis bien en v3.7 Testing. Le script ajoute bien testing en fin de ligne dans le sources.list YunoHost.

Résumé

root@***:/home/admin# cat /etc/apt/sources.list.d/yunohost.list
#deb http://repo.yunohost.org/debian/ stretch testing
deb http://forge.yunohost.org/debian/ stretch testing
root@***:/home/admin# apt update && apt dist-upgrade && apt autoremove && apt autoclean && yunohost tools update && yunohost tools upgrade --apps && yunohost tools upgrade --system
Hit:1 http://raspbian.raspberrypi.org/raspbian stretch InRelease
Get:2 http://archive.raspberrypi.org/debian stretch InRelease [25.4 kB]
Hit:3 http://forge.yunohost.org/debian stretch InRelease
Fetched 25.4 kB in 1s (17.9 kB/s)
Reading package lists… Done
Building dependency tree
Reading state information… Done
4 packages can be upgraded. Run ‘apt list --upgradable’ to see them.
Reading package lists… Done
Building dependency tree
Reading state information… Done
Calculating upgrade… Done
The following packages have been kept back:
moulinette ssowat yunohost yunohost-admin
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Reading package lists… Done
Building dependency tree
Reading state information… Done
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
Reading package lists… Done
Building dependency tree
Reading state information… Done
Info: Fetching available upgrades for system packages…
Info: Fetching available upgrades for applications…
Success! The application list yunohost has been fetched
apps:
system:
0:
current_version: 3.6.4
name: moulinette
new_version: 3.7.0
1:
current_version: 3.6.4
name: ssowat
new_version: 3.7.0
2:
current_version: 3.6.4.3
name: yunohost
new_version: 3.7.0.1
3:
current_version: 3.6.4
name: yunohost-admin
new_version: 3.7.0
Info: No apps to upgrade
Info: Upgrading packages…
Info: Now upgrading ‘special’ (yunohost-related) packages …
Warning: This action will end but the actual special upgrade will continue in background. Please don’t start any other action on your server in the next ~10 minutes (depending on your hardware speed). Once it’s done, you may have to re-log on the webadmin. The upgrade log will be available in Tools > Log (in the webadmin) or through ‘yunohost log list’ (in command line).
root@***:/home/admin# Running scope as unit: run-rb5cf8953f8314a859abd4dd47c7c03fe.scope
Reading package lists…
Building dependency tree…
Reading state information…
Calculating upgrade…
The following packages were automatically installed and are no longer required:
etckeeper iproute
Use ‘apt autoremove’ to remove them.
The following packages will be upgraded:
moulinette ssowat yunohost yunohost-admin
4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.2 MB of archives.
After this operation, 625 kB of additional disk space will be used.
Get:1 http://forge.yunohost.org/debian stretch/testing armhf moulinette all 3.7.0 [67.5 kB]
Get:2 http://forge.yunohost.org/debian stretch/testing armhf ssowat all 3.7.0 [744 kB]
Get:3 http://forge.yunohost.org/debian stretch/testing armhf yunohost all 3.7.0.1 [787 kB]
Get:4 http://forge.yunohost.org/debian stretch/testing armhf yunohost-admin all 3.7.0 [8,575 kB]
Fetched 10.2 MB in 1s (6,213 kB/s)
(Reading database … 61999 files and directories currently installed.)
Preparing to unpack …/moulinette_3.7.0_all.deb …
Unpacking moulinette (3.7.0) over (3.6.4) …
Setting up moulinette (3.7.0) …
(Reading database … 62003 files and directories currently installed.)
Preparing to unpack …/archives/ssowat_3.7.0_all.deb …
Unpacking ssowat (3.7.0) over (3.6.4) …
Setting up ssowat (3.7.0) …
(Reading database … 62004 files and directories currently installed.)
Preparing to unpack …/yunohost_3.7.0.1_all.deb …
Unpacking yunohost (3.7.0.1) over (3.6.4.3) …
Setting up yunohost (3.7.0.1) …

Configuration file ‘/etc/dpkg/origins/yunohost’, does not exist on system.
Installing new config file as you requested.
Regenerating configuration, this might take a while…
Success! Configuration for category ‘metronome’ updated
Warning: The configuration file ‘/etc/postfix/main.cf’ has been manually modified and will not be updated
Success! Configuration for category ‘dnsmasq’ updated
Success! Configuration for category ‘fail2ban’ updated
Success! Configuration for category ‘dovecot’ updated
Success! Configuration for category ‘slapd’ updated
Launching migrations…
Info: Running migration 0011_setup_group_permission…
Info: Creating a backup of LDAP database and apps settings prior to the actual migration.
Info: Updating LDAP schema…
Info: Updating LDAP database…
Info: Creating a group for each user…
Info: Migrating permissions from apps settings to LDAP…
Info: Migration completed. You are now able to manage usergroups.
Success! Migration 0011_setup_group_permission completed
Info: Running migration 0012_postgresql_password_to_md5_authentication…
Success! Migration 0012_postgresql_password_to_md5_authentication completed
Restarting YunoHost firewall…
(Reading database … 62013 files and directories currently installed.)
Preparing to unpack …/yunohost-admin_3.7.0_all.deb …
Unpacking yunohost-admin (3.7.0) over (3.6.4) …
Processing triggers for systemd (232-25+deb9u12) …
Setting up yunohost-admin (3.7.0) …
Processing triggers for man-db (2.7.6.1-2) …
Done!

YunoHost package upgrade completed !
Press [Enter] to get the command line back

root@***:/home/admin#

ppr

My server has just upgraded to the new testing (with unattended_upgrade) strangely yunohost-admin was not in testing mode after that (all other package was)

So i rerun manually

apt update
apt-full-upgrade

I note that my mail users are not recognized:

<xxxx@yyyyy> (expanded from <zzzz@aaaa>): host yyyyyyyy[89.234.141.xxx] said: 550 5.1.1 <xxxx@yyyyy>: Recipient address rejected: User unknown in virtual mailbox table (in reply to RCPT TO command) 

It might be a problem of ipv6 connectivity breaking the upgrade (my vpn connexion flap every hour since 3 days…). Or it could be a bug in this testing.

It’s because the migration about group and permission was not executed automatically.

In fact the migration fails that’why it’s not done:

“La migration 0011_setup_group_permission a échoué avec l’exception Le groupe XXXX est inconnu : annulation”

So the issue is that i have permission on users that doesn’t exist anymore, but the permission are still here in /etc/yunohost/apps/*/settings.yml (allowed_users) https://github.com/YunoHost/yunohost/blob/stretch-unstable/src/yunohost/data_migrations/0011_setup_group_permission.py#L106

We also need to add some loading indicator when we click to delete/add a permission on web admin. On my lime2 it’ hawfully slow, may be due to my connectivity issue) and it’s truelly strange to click and the permission disappear after 20s…

Euh, ça fait beaucoup de testing :smile:

#cat /etc/apt/sources.list.d/yunohost.list
deb http://forge.yunohost.org/debian/ stretch stable testing testing testing testing
1 Like

Bonjour,
J’ai mis l’application redirect dans le groupe Visitors.
Du coup :

  • mondomaine/redirect OK
  • à travers l’interface Yunohost/SSO : KO !
    Est-ce voulu ?

Tu veux dire que en étant loggué tu ne peux plus y accéder ? Ou juste que la tuile n’apparait pas ?

(En tout cas apriori nope, ça a pas l’air voulu)

Bonjour @Aleks,
La tuile apparaît bien, mais la redirection ne s’effectue pas en cliquant dessus. On dirait que ça mouline !
Les autres tuiles/applications fonctionnent bien.
J’essaierai, dès que possible, avec mes autres applis que je passerai en «Visitors» pour tester.
Désolé, j’aurais du être plus explicite.

I also had the exact same issue during the upgrade. An old user that was deleted long ago, but still present in the settings.yml.

I had to re-run the ldap migration manually

Today, one week after the upgrade, nobody can log into the portal, and I cannot ssh into the server as admin, because of a connection to the ldap server.
I restarted both ldap and nslcd. Connection was possible again after the nslcd restart.

Going back in history I see that nslcd was logging connection error to the ldap server since one week, so since the upgrade to 3.7
I’m not sure how to interpret slapd logs : I had regular logging messages since the upgrade, but with suspicious lines like:
Nov 05 12:45:58 ynh slapd[13074]: connection_read(25): no connection!

Now everything seems normal.
slapd still logs the following new warnings/errors since the migration :

Nov 06 10:54:54 ynh slapd[8146]: <= mdb_equality_candidates: (permission) not indexed
Nov 06 11:09:33 ynh slapd[8146]: slap_global_control: unrecognized control: 1.3.6.1.4.1.4203.666.5.16

For the rest: congrats!

1 Like

Ah, maybe we should add a nslcd restart after the migration ? (Those stuff are always quite mysterious …) But now that you mention it, maybe that’s also what’s behind these weird issue where we see people failing to authenticate using a valid password on the webadmin …

Ah yes maybe we need to add this in indexed fields in the configuration (dunno where exactly but I remember the PR)

This one though, I think it’s unrelated to 3.7, but I don’t know the cause behind it …

Je m’ auto-réponds !
Avec les autres applis je n’ ai pas le même problème.
Je pense que cela peut venir de redirect_ynh, voici mon log à l’installation:

root@domain:~#yunohost app install redirect
Available domains:
-domain.tld
Choose a domain for your redirect (default: domain.tld):
Choose a path for your redirect (default: /redirect):
Redirect destination path (default: http://127.0.0.1): http://someotherdomain.tld
Redirect type [public_302 | public_301 | public_proxy | private_proxy] (default: public_302):
Info: Installing the app ‘redirect’…
Warning: /!\ Packagers! This app is still using the skipped/protected/unprotected_uris/regex settings which are now obsolete and deprecated… Instead, you should use the new helpers ‘ynh_permission_{create,urls,update,delete}’ and the ‘visitors’ group to initialize the public/private access. Check out the documentation at the bottom of yunohost.org/groups_and_permissions to learn how to use the new permission mechanism.
Success! Installation completed
root@domain:~#