Buster and beyond (better update/testing, based on upstreaming, debian integrated infrastrucure)

Contribute Room Core dev
1

A sincere question, looking at the work needed that goes into keeping yunohost up to date with new debian releases, and to maintain and test everything separately.

Couldn’t it streamline the workflow and ease things quite a bit, if yunohost could make more use of the recent server management infrastructure that has become available as part of the standard debian releases?

The freedombox .deb package, for example, is part of debian and its derivates. It covers the basic setup, configuration, and maintenance of a webserver stack, as well as several other things for many platforms and devices already.
It implements a (python) web portal to configure other packages, and even some system administration with cockpit. And, it has a “backports channel” to quickly ship related UI and feature updates to “stable” users.

What if,

  • The yunohost frontend and admin system can be installed on top of (later maybe depending on) the freedombox package, and thus make use of freedombox features and developments.
  • Without any deeper integration, the current yunohost web frontend might basically be installable like any other a web application on freedombox, including the ability to configure the yunohost frontend as the default homepage.
  • The yunohost packages (scripts) could be made installable and testable directly on debian/sid+freedombox+yunohost systems.

Do you see usefull things to use, otherwise unfeasable features included for free, collaboration benefits, and maybe yunohost improving or including something to debian and it’s derivatives at some point?

2

Do you have anything specific in mind ?

I don’t really understand the whole thing about installing the yunohost-admin on top of freedombox and what it has to do with new debian releases … that just sounds like “why don’t you install the nginx module on top of apache2 since they both are a webserver ?” …?

The work needed to support new debian release is because there are big and small technical details to adjust so that everything runs smoothly, and I doubt there’s a magic silver bullet that’s gonna make it so that you don’t need a hard-working human testing and fixing stuff in the end …?

3

sounds like “why don’t you install the nginx module on top of apache2 since they both are a webserver ?”

All right, you know the subject. :blush: The webserver choice really is a central point. And I think a good choice to have.

I think yunohost really pioneered in making self-hosting really much more accesible, including support to install (and package) more web applications in general.

The possibility to install web-apps means yunohost will likely never be able to become an official debian package, but debian self-hosting has by now evolved to a point that’s promising for collaboration. (It seems we’re at about the right time for devs to sincerely evaluate that, see below.)

I’m affraid I also don’t see a magic silver bullet. Nevertheless, the amount of small technical details to adjust and smooth out, might still be reduced or focused on more specific things, by doing the work in conjunction with others and sharing parts of code and testing, and benefiting from features contributed by others. (More eyes and shoulders.)

Do you have anything specific in mind ?

Some random points I had in mind.

  • If the ynh_* web-app packaging helpers could work for nginx and apache etc., they could be shared.
  • The freedombox devs are planning to support installing Nextcloud even though it is not part of debian proper. So they are prepared to put a considerable amount of work into Nextcloud, which may be better invested in collaborating on releasing a shared helper lib with yunohost.
  • Freedombox devs also need to put lot of work into adding support to configure a mailserver, which yunohost sports already. Also here a feature complete solution may also be best achievable and maintainable in the long run based on a collaborative effort that spreads into debian proper.
  • Freedombox is developing NAS (network filesystem) features that are very nice for homeservers.
  • A sledgehammer first start may be to allow installing yunohost and freedombox packages side by side (nginx and apache, maybe two IPs), then go from there seeing if calling the other’s python helpers works, etc.
2 Likes
4

Sorry, I had overlooked that.

I was first thinking of yunohost packages, if they can install on debian sid (with some helper package installed) it would be much easier to test them with the next debian release.

But if yunohost itself can use and contribute to the same helper package, maybe it can also be easier to test and upgrade to the next debian release with less lag (continuous testing)?

5

I don’t know what we’re talking about exactly here … There’s :

  • how to make the transition to newer release easier
  • then suddenly talking about how we could have shared pieces of software with Freedombox
  • and a bunch of specific details about Freedombox’s roadmap

which makes it sounds like you have a bigger idea in mind … which may be cool, but it’s confusing and would help if you can clarify what you have in mind exactly (or maybe I’m overinterpretating idk)

I guess, yes, but the point is, you don’t know in advance what are going to be the changes in future debian release, and because everything in a server distribution is pretty tight-coupled, it’s very hard to make anything “future proof”. Installing a yunohost app require that yunohost is installed in the first place. And for yunohost to be installed, you need to fix whatever needs to be fixed. Then yes, we have CI for apps, but first you gotta make yunohost work.

So no trick is going to allow you to “test things without having to put your hands in the dirt and fix stuff” (and I’m always confused why so many people always want to put so much thinking and effort into “how we gonna work around the issue” instead of facing the root issues which is often not that bad…)

Ultimately the number one root issue for slowness of moving to the newest Debian release is : number of humans actually working on it. It’s not even that it’s complicated per se, it’s just that we need people to actually put their hands in the dirt and run tests and implement fixes and ask questions and just make it work.

I don’t know if you are a dev / sysadmin or anything but I’m really skeptical about this kind of idea. This just sounds like “it would be nice if the rocket we’re designing could work with two kind of propellant interchangeably”. Yes, that’s super nice on paper. But if you know anything about rockets, you know that everything in the rocket is tight coupled to the propellant used. Same with automatizing system administration in the context of Yunohost or Freedombox or any similar project. “It would be nice if we could choose or not to use systemd”. “It would be nice if we could also install the project on Archlinux or Alpine or NixOS”. Yeah. No.

Again, that sounds super nice on paper, but absolutely not realistic in practice.

I was about to say something about how it would be nice to have something like a standard, default, abstract specification that web app could implement (independnetly of yunohost) that explains how the app should be installed and upgraded. Then any system like Yunohost or Freedombox or whatever could implement a layer to be able to install that app. But here again, everything is too tight coupled about the target ecosystem (just think about the dependencies and this becomes hell in less than 5 minutes). This is typically solved by containerization which is another story.

2 Likes
6

I (sysadmin some patching) don’t have an exact idea or picture, yet.

That’s also a reason why aspects like sharing things, release upstreaming ideas, roadmap complements, etc. may still be confusing.

It may all boil down to contemplate for some period of time if yunohost and freedombox developers would be open to find and pursue collaboration.

Finding a way usually only comes about by looking at things from many different angles and approches, though. Until a picture or structure starts to develop, over time, that is promising enough to pursue.

(A funny thing is that it’s currently not even clear where upstream nor downstream would be exactly. Debian and the freedombox package maintainers may be upstream for yunohost but at the same time the yunohost package repository would be upstream for a freedombox.)

I liked your (technical) responses. Let me add some of my thoughts.

point is, you don’t know in advance what are going to be the changes in future debian release, and because everything in a server distribution is pretty tight-coupled, it’s very hard to make anything “future proof”.

I think instead of trying to foresee all changes and “future-proof” things, the approach to this would be to have things in debian sid and see (get a list of errors and reports) as soon as something breaks. Allowing to see if it’s something one has to deal with sooner or later, or if it’s something that can or has to get fixed by or with others. (Freezing and stabilizing together with the next release.)

root issue for slowness of moving to the newest Debian release is : number of humans actually working on it. […] we need people to actually put their hands in the dirt and run tests and implement fixes and ask questions and just make it work.

I think part of the solution is that every yunohost bit that gets into Debian sid/testing gets much more accessible (servers around the world) and broader exposure. And if collaboration can be agreed on with the goal that the freedombox “stable” users get a way to install major ynh packages like nextcloud, it will even find the way to people in another self-hosting community with interest to test and fix packages.

I was about to say something about how it would be nice to have something like a standard, default, abstract specification that web app could implement (independnetly of yunohost)

Well, that’s part of what I like so much about yunohost. You actually stepped up and made concrete specifiactions that allow to conveniently install webapps. Now, a basic solution exists, for others to pick it up, and join the efforts, but it can likely only happen if you want that, and pursue it for a start.
Even if it’s just for debian+common-helper-package based systems, I’d expect web-apps for debian to likely have a surprising range.

Personally, I think in your shoes I would try to see if there are other devs with overlapping interests in bootstrapping this. (Like freedombox developers especially, with such closely matching things on their roadmap.)

As the common basis is Debian, not Arch, Alpine etc., and thus there are already many useful rules on how packages relate etc., I don’t think starting collaboration and merging/upstreaming some parts would be out of this world.

Please also don’t expect the following to be anything that’s already exact, but here are some more concrete thoughts to contemplate and consult about, if yunohost wants to reach into a larger developer base.

  • If collaboration is first only pursued for the web-app packaging specification and helpers, the two web server stacks may not neccessarily have to be made installable on the same machine.
  • Can the ynh and fbx web frontends both access two separate backends (for specific disjunct areas) allowing for consolidation over time? (Freedombox does have a backend/frontend separation.)
  • To be realistic, upstreaming some things will likely also mean to coordinate to step back on something and adding what’s needed to the other’s something. (See this as a crucial step to lower and focus the workload.)
    For example, to be willing to migrate to another LDAP schema, in order to be able to gain more users and developer exposure and collaboration for ynh_* helper features etc.
  • I don’t think the yunohost project or community would be reduced, even if all the infrastructure would be part of debian proper (the community may rather enlarge), because I don’t see all the web-apps to become part of the debian releases. So it’s always needed to keep up with all the web-app developments, if yu-no-host…, even if the same infrastructure is also used and maintained by others to set up and administer pure debian servers with web- and .deb packages around the world.

Maybe we should take it a little slower, and first look around for possibilities some more. But if yunohost devs are interested to increase the reach, maybe some exchange with freedombox devs can bring out the most fruitful ideas. (Not quite “putting hands in the dirt” yet, but a start still.)

Looking forward even if devs can respond to this only after some days or weeks.

7

An example.

  • If the yunohost mail server configuration could be made usable in debian by the freedombox package…

    … then yunohost could get friendly additional help from the debian developers that look at the freedombox package, as they will also be looking after all the big and small technical mailserver details to adjust so that the mailserver always keeps running smoothly. (And their help might even cover some parts of getting it into debian, as they would have to do it for themselves, as well as maintaining it afterwards.)

    … then whatever change happens in debian sid, should usually get fixed at latest during the freeze for the next stable release. (with high chances to make it dist-upgradable on the day of the release.)

If, in the process, the mailserver configuration might get installable like any other ynh_* app, then even younohost itself might get more modular.

Or, might it be better to first start the upstreaming with the ynh_* helpers, and go for the mailserver in a following step?

8

There is currently work in progress on the packaging format of apps. Migrating from an A to a B format already seems rather complex, adding support for ynh apps by freedombox at the same time would make things even more complex.

It is still possible to present the new format under construction, and if freedombox wants to draw inspiration from it or create a compatibility layer, it would be possible to discuss it, but if it has to be done, it should be remembered that the complexity should not increase disproportionately. But that’s the big risk of wanting to support different systems. Packagers will have to know freedombox as well as YunoHost, new dependency issues will arise, etc.

It is also important to realize that the original idea was to set up a foundation to simplify the packaging. The foundations of YunoHost are nginx, postfix/dovecot/rspamd, let’s encrypt, the yunohost cli and the helpers, a certain way to configure the tree, and maybe some elements like XMPP or mysql that probably shouldn’t be there by default…

Concerning the mail stack, it’s a central element, and if we’re still far from modularizing mariadb or metronome, we’re even farther from making the mail stack optional, in my opinion. To work YunoHost needs a system to communicate asynchronously with the YunoHost administrator, and currently this mail stack is the only way to notify the admin.

In the same way, yunohost helpers don’t automatically generate nginx configurations, those are the packagers who are in charge of creating / adapting these configurations. Wanting to support apache2 would add more work to the packagers.

As for developing YunoHost on debian testing, we came to a similar conclusion, unfortunately the transition to the new versions of debian has a hard time motivating contributors.

9

Hello, very interesting thoughts.

And I think really worth to talk over with the freedombox core devs.

What I can add may be that I recall having seen something that may be related to a local “system to communicate asynchronously with the YunoHost administrator”:
https://wiki.debian.org/FreedomBox/ProgressCalls/2020-06-13
It contains something about a “DBus based daemon plan”.

It’s also an interesting point that you say ynh packages currently ship nginx configurations in the packages, it leaves room from shipping additional apache snippets, up to full helper-based definitions.

10

What would be your requirements for a mail stack manager in debian / freedombox that you could use?

11

Yep, big-step updates are kind of a heavy-lifting chore, compared to the curiosity of running a (separate) debian/testing system to test newly developed features. Remember that debian/testing starts out identical to a new stable branch and is only gradually updated until the next release freeze.

Interesting still, what would be your desired requirements for a future mail stack configuration manager?

  • based on LDAP accounts
  • user management CLI commands ???
  • …?
12

Indeed, as long as these projects avoid containers, things get complex.

Actually such a format is quite easy to blueprint once you take the leap. Every server app consists of:

  • Volumes. One for config, one for data.
  • Ports.
  • Executable.
  • Environment variables.

All you need is already standard in containers Actually pretty close to https://www.linuxserver.io. The only missing piece is a manifest that ties all that together with the host system, which would be managing the proxy, sso, users, logs installation and updates.

The process of having a layer that takes the mostly standard Yunohost package and converts it to a container wouldn’t be extremely hard. Newer apps shouldn’t care about the old system, but older apps could just leverage that shim and keep on working.

13

The binary container distribution formats have, e.g. the problem of having multiple library versions in the different images, that can’t be directly updated with security patches.

Likely the problem and solution is the same as here:


The complex problems are truly solved, only by the next generation of package managers.

And a binary container distribution wouldn’t improve anything above having to have:

  • a common-base (we use the debian released source packages)
  • conventions
  • helper tools lib
  • packaged configurations

The idea in this thread would be to share convention and helpers (and upstream these into debian).

For example, to share an email stack convention with configuration helpers that can be used by Freedombox as well as YunoHost.

Or, to use the available freedombox (python-) helpers from upstream to install and manage most of the basic bulleseye server infrastructure (based on an option to install without the default apache), and this allowing to work with a more specific focus on maintaining, e.g. only the nginx and web-application packaging parts on it.

(Note that the most current freedombox helpers in debian/bullseye are also available in buster-backports.)

14

BTW: The upstream helper tools provided by the freedobox package (in Debian bullseye & buster-backports) are called “actions”.

They are also available as shell commands under /usr/share/plinth/actions.

The online-docs explain, but don’t seem to feature a list of the shipped actions.
https://docs.freedombox.org/reference/index.html

15

Here are some interesting quotes from a recent issue in freedombox’s gitlab:
https://salsa.debian.org/freedombox-team/freedombox/-/issues/1859#note_169406

[ In the past, FreedomBox ] emphasized the generic name over the app name. However, this changed […] due the following reason:

  • FreedomBox aims to be a larger ecosystem for delivering self-hosted applications to end users. This means that FreedomBox apps may not be primarily written by FreedomBox developers. They will be written by upstream authors of apps or other developers. In this case, we don’t have control on how many apps of a given type (or to some extent how) go into FreedomBox. Users will have to choose software like they choose from distribution repositories, Android, iOS, etc. We can still help users with this disadvantage with search functionality and purpose-specific guides (like guide to setting up NAS).

We will have an app store [package manager] that will allow app developers to publish their apps. Or they could publish their app directly to Debian and will show up in FreedomBox.

All it means is that we can’t pick one among Mastodon, GNU Social, something else because all of them are likely to get offered up in FreedomBox. We may have to compromise slightly on simple user experience (such as suggesting a clear solution). When app developers create apps for FreedomBox instead of the core team, I believe apps names will need to be emphasized over generic names because to each app developer, their own app is the most important one.

Currently, we have a very decent list of apps in FreedomBox which makes it quite useful as a home server but we need a lot more to cover all the cases for which a typical user is using cloud-based services. Thinking of a world where every use case in the cloud is met by a friendly, free software server at home, we need more apps. Thankfully, there are many developers and their apps able to meet this need. Some software is very well proven while others are bold experiments like Torrents were. We need to deliver the successes to our users and encourage the bold ideas by providing them an audience. We need to make it easy for a lot more people to join our movement. Allowing people to easily write and maintain apps for FreedomBox seems like a good way to do that. Again, I completely agree with you that we should not compromise on freedom, privacy, etc.

16

Well, it sounds to me that Freedombox is getting more toward an alternative to Yunohost rather than a potential upstream tool. So the only way I see to benefit from it would be to drop all overlapping features on Yunohost’s side, and port everything else to the Freedombox’s systems. That could be a very big work, but is that even desirable?
Yunohost’s set of tools are pretty mature. Maybe I don’t realize how much work is done in the background on emails’s settings (for example) or existing helpers. I think the diagnosis tool was one of the very big completed task, and the port to Buster is the current big one.

As for the work on next versions of Debian, I’m skeptical about the idea that testing things on a moving target as it evolves is really less resource consuming than testing things on the finalized version. The main benefit would be to have a Yunohost working on the latest Debian faster, because the heavy lifting would have been done along the way, but that supposes resources to follow the changes upstream, so we’re back to the resources issue.

Yunohost, not being part of Debian, does not have a requirement to be ready for the freeze, and there is not much inconvenience in having the next major version a little while after the Debian stable release.

Makes me remember my old fixed computer is still patiently waiting for me to be reinstalled and I could use that to test Yuno 4.0…

1 Like
17

Hasn’t freedombox always been an alternative, if requirements could be met?

They followed a route that was slower in the beginning, though. Developing backend and frontend tools that are aimed to be universal, and committing everything upstream. Working on and expanding the Debian infrastructure, to allow for self-hosting based on a Debian Pure Blend installation.

Now, as things are maturing in bullseye (and buster-backports) they increasingly become a useful upstream tool. Even for hosting web-application packages, or other frontends, that are not part of the Debian repository themselves.

The resource question seems to be, having to create, update and maintain everything separately, or together with more people at least for the basic parts. Allowing to focus more on distinct features.

18

The resource question seems to be, having to create, update and maintain everything separately, or together with more people at least for the basic parts. Allowing to focus more on distinct features.

But the basic parts are already created in Yunohost as well, and it they don’t represent a decent share of the work to maintain. The distinct features are the most advanced ones, and they are the ones “causing” most of the work.
Again, take the diagnosis tool. You propose to ask the devs to throw it away and rewrite it from scratch because you’re changing the core of Yunohost. Benefit: the core of Yunohost is now shared with Freedombox.
Pro: 5% of the work is gone for good.
Con: You need to redo 95% of the work done historically

The migration will take 2 years of intense effort just to get the same set of features we already have today.
It will take 5 more years after that to pay off as you share the core development.

I maybe exaggerating the numbers (but maybe be not!!), but I think you get the idea: if Yunohost was a new project starting today, no doubt it could benefit from a common upstream. But it already have a history and a legacy, and no fundamental flaws seen from my humble eyes.
The migration is not worth it, or not at the moment.

19

From the exchange above, the idea seems to clear up, that we’re most probably merely talking about coordinating a common update of the packaging format.

It’s a collaboration opportunity at hand.

The code to share likely is
https://github.com/YunoHost/yunohost/tree/stretch-unstable/data/helpers.d

with functions like the following to be made to work on both distributions:

ynh_package_is_installed() {

    # Declare an array to define the options of this helper.
    local legacy_args=p
    local -A args_array=( [p]=package= )
    local package

    # Manage arguments with getopts
    ynh_handle_getopts_args "$@"

    ynh_wait_dpkg_free
    dpkg-query --show --showformat='${Status}' "$package" 2>/dev/null \
        | grep --count "ok installed" &>/dev/null
}

Oops, well that example might actually already work as is. :slight_smile:

But of course there are things to coordinate: environment detection, conditional actions, basic generic webserver (webroot/site) setup functions? besides functions to install specific (packaged) nginx and apache configs?, etc.

Keep in mind that on the coding side,
YunoHost would basically only need to update the current implementation.
FreedomBox would have to add code to the helper functions, to call their actions on freedombox.

@matlag Your migration consideration seems to go way beyond application packaging.

The migration is not worth it, or not at the moment.

I’m not sure if your numbers (95-to-5 %) are exaggerating. But it may rather be the other way around, a complete switch would only make sense, if

after revising only 5% of the code that will be kept,
95% of legacy code is gone for good.

And if relying on a common release will allow to focus one’s work on more specific things afterwards. Maybe porting diagnostic functions to the common release, where really missing, with 25% of the former legacy maintenance workload.

Anyway, back to the topic at hand, I’d suggest the YunoHost devs could bring up their plans for the next packaging format version, and talk with the freedombox devs, to see how things may fit together.

20

After testing both projects, I can see that both focus on giving the non-techie user a way to self-host, but they have very different approaches, so I’m not sure they will ever join.

Yunohost is focused on being extremely easy for basic tasks, while asking the user to go to the terminal to do some advanced tasks, or go to github to consult documentation. Also its main focus is on apps.

Freedombox OTOH is more focused on freedom (as the name says), self-documenting and never touching the terminal. This means that the user has more stuff to read directly on freedombox UI to know what he’s doing. Also means that the config panel is harder to understand because it has a lot of more freedom-focused stuff (Tor, VPN, I2P, WireGuard, PageKite…). Installing the apps is easy though, but there are way less.

I think that for this to happen and be really worth it, there must be these pieces of the puzzle:

  1. A new standard that defines a manifest format for self-hosting apps. There are a few common things, independently of the app: supported publishing methods (HTTP behind the proxy (with or without subpath support), dedicated port(s), no publishing, VPN, Tor, pagekite…), app name, questions and answers form, supports LDAP?, supports sso?, storage path, needs access to user’s $HOME?, dependency apps, scripts to install, uninstall, purge, run, start, stop, restart; diagnoses, fail2ban rules, maintenance status, etc…
  2. A testing suite for that standard, so both projects can safely claim to support it.
  3. A testing suite for the apps, so apps can test integrations with the standard and not just with one or the other project. Possibly the suite should be able to emulate ARM with qemu.
  4. Good docs on how to create a self-hosting app for that standard.

Over time, as this manifest keeps on being added to yunohost and freedombox apps, or directly upstream projects, the devs themselves could have amore concrete work:

Yunohost and Freedombox devs can focus on the platform itself, and on adding better support to the standard where needed, or evolving the standard where needed. Also on improving the system integration, providing more solid networking, storage, sharing, login, etc. Basically making system plumbing easier and more stable.

App packagers can focus on keeping the app and manifest updated and the test suite green.

App developers can focus on developing the app, but since the manifest and test suite are a distro-agnostic standard, they can adopt also the role of the packager too. This would help end users to have support directly from upstream, and would help app developers to have a wider user base and provide faster updates.

As a side note, the longer these projects do not support containers, the faster they will die. When I say containers, I mean any system that can bundle all dependencies and become one “thing that works”. It could be docker, podman, flatpak, nix, snap… whatever, but that’s the only way this is going to make sense if we want to be able to make app devs directly support self-hosters.

Just give quick test to https://www.taisun.io/ and see the incredible power that comes with that. It lacks a lot of features that Yunohost and Freedombox have, but it became a serious alternative in no time, just because it’s using the right tools. (I don’t mean Yunohost is using wrong ones, they surely were right when the project started, they just seem to have become old).