YunoHost package maintaining: Continuous integration? Trouble with CI

Contribute Room Apps packaging
1

I have some troubles with CI in the past and today, see:

But here in this Thread i what to discuss on a higher level independent of my specific package problems.

Why? Because the quality of YunoHost packages is very important for the users and so for the complete project. I often hear from people who try YunoHost that they are disappointed because the installed packages do not work.

I as an package maintainer see several problems with the current CI infrastructure for YunoHost packages:

  1. In transparent: The maintainer don’t see much of the CI activity in the GitHub projects. So the maintainer doesn’t get information about CI status on a direct way.
  2. No special security when the status changes drastically: e.g.: level falls from 7 to 0 → After some time, the new level is transferred to the apps catalog → User gets a broken app.
  3. Currently, everything about CI is just too slow: It takes too long until a CI run is triggered and executed. This is not really continuous integration.
  4. The tests do not go deep enough: Check if root page returns status 200 doesn’t mean that everything works.
  5. Setup/usage of official CI Servers

Problem 1: Missing CI activity on GitHub

The official YunoHost CI server should add information about every run in every Pull Requests. Currently only merges to “master” are triggered. But then a Bug in “testing” or a PR can’t be captured.

It should be as simply as GitHub Action or like @tituspijean inofficial CI server is does. (But without the need to add a “!testme” comment in a PR)

Why?

Because the maintainer should faster and direct see if a PR is ok or not. Fast cycles speed up the work.
If everything takes too long, people may tend to commit untested code changes.

One solution for this: Run the package check as github actions (But it seems not easy)

Problem 2: What happens if package level falls?

This has already happened twice with my package. I think that it should be prevented to transfer a new level simply into the catalog after some time. Because the new Level may not reflect the quality of the package in every case. What happens if there is a Bug in CI setup? And a Bug in the YunoHost scripts should also not be simply accepted.

Yes we have the git branches “main” and “testing”… But because CI is so slow, it may happen that master gets a broken changes.

What can we do here? The app catalog can just use the older version with the higher level. So the end user can use the package on a normal way and get no update to a broken version.

Problem 3: CI is just too slow

I think the problem here is clear to everyone, right?
It just takes way too long for testing code changes and getting the new result from CI server.

Problem 4: The tests do not go deep enough

This probably requires a lot of work. But I think that would give the project much more stability.

It is simply not enough to test if an app is installable. We need to test if it really works. That is not easy, i known.

I think only with this kind of tests the project will achieve more stability.

Problem 5: Setup/usage of official CI Servers

I have had problems in the past to activate the CI checks and the documentation at Continuous integration | Yunohost Documentation is outdated and incomplete.
I sill didn’t understand the complete YunoHost CI setup :frowning: When is a CI run actually triggered? This is also related to the lack of intransparency, see “problem 1” above.

2

I only have a comment from the sideline. I am not maintaining a(ny) package, but thinking about starting to do it.

To me these seem like valid points that hamper you (as volunteer packager). They all can be solved with more money (for server hardware) and more volunteers (to improve the software).

If I like to start packaging as you do, would I help myself by installing Gitlab and Gitlab runner on my Yunohost? Will that enable me to do CI for my package on my own hardware, without a queue of all other packages?

1 Like
3

That was also my thought in the past. It would be super cool if you can run your own CI server via YunoHost. I started to play a little bit with GitHub - YunoHost-Apps/ci_ynh: Experimental YunoHost package check as a YunoHost app using Django... see also: "ci_ynh" -> Package Check as YunoHost app using Django But that is unfortunately all too time-consuming for me.

4

Hm. No response from some core maintainers here?!?

5

I’m not sure I understand … the dev CI is precisely here to test testing / other PRs ?

I’m not sure to understand why running as a github action would allow to “see directly” ? The CI post results in a comment, you can directly see if everything is all green or not… And in both case (current CI or github action) you are gonna need to open another tab to see details, and they are both gonna be the same speed, probably.

To put things into perspective : before the new !testme mecanism, testing apps was way more high cost : you had to manually push your code using ssh/scp to some jenkins server, and the results were not displayed at all in the PR.

Yes, there are possible inconsistencies because the level in the catalog corresponds to the level of the last reported CI job, not to the commit being served (and the commit being served is basically master/HEAD). With that said, it’s not such a huge deal because in the vast majority of cases, app levels tend to be stable. Still, we did indeed discussed that the whole thing could be smarted, but like a lot of things, it boils down to “somebody has to volunteer and do it”

The second issue you seem to describe is “what happens if the CI incorrectly returns a bad level”. In that case, we tend to re-run the CI job to confirm whether or not it’s a temporary issue or if the app is actually broken before committing the new level to the catalog.

Yes, yet it’s not trivial to optimize. There’s the new multi-worker mecanism that Kayou implemented that we should try (assuming the machine has appropriate resources)

I believe the apps which take a super long time are apps that requires building with npm build or compiling. There were discussions at some point to externalize builds which would speed up install by users and on the CI. But that’s only one aspect of the CI test time.

Manifest v2 may open new possibilities in terms of pre-installing apt dependencies, pre-fetching sources, etc. Though still will require some work.

I doubt that we could get to less that at least 20ish minutes per app though.

Yes indeed … basically that’s a 2-months full time R&D project if you want to be able to study the scope of what could be tested, what tools to do so, creating an easy way to describe what should be tested, and then implementing everything in CI ¯\_(ツ)_/¯ … and then adding tests for all relevant apps in the 200~300 apps in the catalog @_@

Yes, we’re trying to organize a doc hackathon to hopefully refresh the packaging doc …

A CI run is triggered:

  • when triggering it with !testme (on the dev CI) or related stuff for other CI like titus’
  • when you push code to master, pretty sure that trigger a job on the main CI
1 Like
6

There is the dev CI with the need for “!testme” … One problem: This “dev CI” is not really any official infrastructure of the YunoHost project, isn’t it?
The second Problem is, that the developer needs to know the “!testme” trigger.

IMHO we need a official dev CI and the run should be automatically triggered by any PR.

Yes, the dev CI commens are cool and helpfull. I’m not the biggest fan of github! But github actions are better integrated :wink:

Using github actions used resources by Microsoft and it seems YunoHost project resources are limited, isn’t it?

20Min is also a very long time :speak_no_evil:

7

This could help out: https://dash.yunohost.org/app_maintainer_dash/jensdiemer
We could maybe integrate latest dev CI run, for example, but what about when you have multiple PRs opened…

It is completely official. You may be confusing with mine, that I offered up because the main official one gets clogged up sometimes.

It really depends on the coding style of people.
Your suggestion may work if people push ready-to-test code, but it is not always the case. And the Draft feature is sometimes forgotten. Triggering the test semi-automatically with a comment is a nice way to make sure the CI is started when you want it to.
However, we could allow you to directly start it up if there’s a !testme in the body of the newly opened PR.

Github actions are limited to 20 concurrent jobs.

Still better than manually testing all installation conditions. :stuck_out_tongue: