New experimental "2.1" helpers

Contribute Room Apps packaging
1

Dear packagers,

now that the packaging v1 era is mostly behind us, and packaging v3 is not yet there, but we’re experimenting with a new mechanism allowing for multiple version of helpers to exist in parallel, in turn allowing to experiment with important changes in helpers but with a smooth transition phase.

Apps can specify in their manifest.toml which version to use using helpers_version = "X.Y" in the [integration] section.

We iterated and ended up with a “helpers v2.1” version of the helpers. This is mainly about trying to remove legacy helpers, underused options, simplifying syntax, and uniformizing helpers and variable names.

In parallel, considering the amount of various changes, it’s not reasonable to assume humans will tweak everything by hand. A small script is being developed to automatically apply 90%ish percent of changes on an existing “v2” app: Add script to autopatch apps for helpers 2.1 by alexAubin · Pull Request #2407 · YunoHost/apps · GitHub

:red_circle: However at this point, these helpers are still very much experimental and please refrain from actually merging “2.1” changes in apps even though they are theoretically supported. We need to freeze the changelog, test and debug those on a representative sample of apps. Stay tuned!

Detailed changelog (still likely to change)

Click to show/hide
  • General remarks regarding helper argument management
    • Support for so-called “legacy” positional arg was removed in getopts to try to enforce full keyword argument usage (--foo=bar) and have as much of a uniform syntax across all helpers as possible
    • However, when refactoring, many simplifications were made leading to a lot of helpers only requiring a single arg, which is now handled as a positional
    • The result is that many helpers are much simpler, usually requiring no args, or a single arg via a positional (for example: ynh_safe_rm /path/to/file)
    • Keyword arguments become sort of “the exception”, for complex helpers requiring many important args (such as ynh_setup_source, ynh_add_config, ynh_systemctl, etc…)
  • General remarks on helper naming conventions
    • aiming to uniformize naming convention, we converged to basically having a ynh_{tech} “namespacing” :
    • “tech” versions are defined via global ${tech}_version variables (such as $php_version, $nodejs_version, $ruby_version, $composer_version, …)
    • helpers to install/remove “technologies” are named ynh_{tech}_install / ynh_{tech}_remove (with tech = nodejs, ruby, go, composer, apt, …)
    • similar stuff for database helpers : ynh_{dbtype}_shell, ynh_{dbtype}_dump_db (with dbtype = mysql, psql, … mongodb? redis?)
    • config-related helpers are named ynh_config_add/remove_{tech} (with tech = nginx, phpfpm, logrotate, fail2ban, …), for example ynh_config_add_nginx
    • try to prefix all the internal / low-level helpers with _ (i.e. _ynh_foo_bar) ?
  • APT
    • simplify bloated, unecessarily complex code
    • rename helpers to ynh_apt_install_dependencies, ynh_apt_remove_dependencies, ynh_apt_install_dependencies_from_extra_repository.
    • misc internal helpers such as _ynh_apt_package_version are prefixed with _
  • Unix permissions handling
    • ynh_setup_source and ynh_config_add now have improved mechanism to chmod / chown the target to what should be appropriate permissions in say 90~95% of cases (including having www-data as group for the $install_dir or not). Therefore in the vast majority of cases you should NOT need to manually apply chmod / chowns to the install dir or the config files.
  • ynh_setup_source
    • --full_replace now to be a boolean, no need to write --full_replace=1
    • drop support for underused sources/extra_files/$sourceid mechanism, only 7ish apps actually do this and they manually cp their stuff instead of relying on the helper’s behavior
    • move the sources/patches/ mechanism to just patches/ (at the root of the app dir). The patches folder should have one subfolder per source, each containing the paches. For example, tpically patches/main/foobar.patch.
  • PHP
    • Rename $phpversion to $php_version for consistency with $nodejs_version, $ruby_version etc
    • Completely trash the fpm_usage/fpm_footprint mechanism. See documentation of the new helper for apps that really do need to customize performance-related stuff
    • The default template now includes upload_max_filesize, post_max_size and memory_limit, therefore most apps should have no need to have a conf/extra_php-fpm.conf at all.
  • Composer
    • $composer_version should be defined as global prior to calling ynh_composer_install (same as PHP / NodeJS / Ruby / Go versions)
    • Remove every arg in ynh_composer_install and ynh_composer_exec (apart for the composer command to run for ynh_composer_exec, using positional). The old --workdir is now automatically $install_dir unless $composer_workdir is defined prior to calling this helper.
    • ynh_composer_install now ONLY wgets the actual composer.phar directly instead of calling the unnecessarily complex official composer install script. ynh_composer_exec needs to be explicitly called for the actual “install” of the vendor/dependencies
    • ynh_composer_exec runs as $app instead of root, unless composer_user=root is defined prior to calling this helper
  • NodeJS / Ruby / Go
    • rework ynh_install_nodejs/ruby/go
    • no more --version arg, just define $nodejs_version, $ruby_version, $go_version as global prior to calling it
    • Tweak verbosity such that packagers are not tempted to slap an ynh_exec_warn_less in front of it …
    • Remove the need to call ynh_use_nodejs/ruby/go, it’s just called at the end of the ynh_{tech}_install helper
    • Export the modified PATH variable such that one can call npm, node, ruby, go directly … instead of the clumsy $ynh_npm/node var/aliases.
    • Because the PATH is exported and ynh_exec_as_app now uses sudo --preserve-end=PATH, there’s no need to pass $ynh_node_load_PATH when calling npm/node/ruby/go/…
    • Rework ynh_node_load_PATH : replaced by PATH_with_nodejs such that we write Environment="PATH=__PATH_WITH_NODEJS__" in the systemd config
  • Backup/restore
    • ynh_restore is renamed to ynh_restore_everything.
    • ynh_restore_file is renamed to ynh_restore to be symmetric with ynh_backup.
    • Single positional argument, no more --dest_dir, --not_mandatory or --is_big.
    • The helper handles the logic of checking if the path is $data_dir or /var/log/$app, in which case they are ignore during safety-backup-before-upgrade or if do_not_backup_data=1 (standardizing the old, removed “is_big” arg)
    • The core will now automatically delete /var/log/$app during yunohost app remove if --purge is used. The app remove script should not remove /var/log/$app/ (otherwise this means it gets deleted during removal of the app prior to restoring the safety-backup-before-upgrade …)
  • Logrotate
    • renamed to ynh_config_add/remove_logrotate for consistency with other helpers
    • Single position argument (the logfile(s)). No --specific_user option anymore (if logrotate complains, then the log dir permissions should be fixed)
  • Fail2ban
    • remove unecessary/unused --max_retry and --ports options
    • remove --use_template: just generate the template “on-the-fly” if --failregex/--logpath are provided, or use the f2b_stuff template files from the conf/ directory otherwise
  • mysql/psql:
    • no keyword arg, full positional arg
    • remove ynh_foo_setup_db and ynh_foo_remove_db (the other helpers are enough)
    • replace ynh_foo_connect_as/ynh_foo_execute* with a single ynh_foosql_db_shell that reads stdin
    • use $db_user, $db_pwd, $db_name as default values for user, password and database args, in turn heavily simplifying the syntax
  • Misc helper renaming/simplification
    • the remaining logging helpers (ynh_print_info, ynh_print_warn, ynh_script_progression) now use a positional arg (no need for --message= anymore). Drop the weight mechanism for ynh_script_progression which doesn’t make much sense since packaging v2 anyway
    • add a new ynh_app_setting_set_default to replace the unecessarily complex syntax if [ -z "${foo:-}" ]; then foo=bar; ynh_app_setting_set --key=foo --value=$foo in upgrade script
    • ynh_exec_as $app becomes ynh_exec_as_app, (with --preserve-env=PATH option in sudo, relevant in particular for nodejs/ruby/go apps)
    • ynh_secure_remove --file=/foo/bar becomes ynh_safe_rm /foo/bar (single positional arg)
    • ynh_exec_warn_less becomes ynh_hide_warnings
    • ynh_replace_string becomes ynh_replace --match=foo --replace=bar --file=/some/path
    • ynh_systemd_action:
      • renamed to ynh_systemctl
      • --service_name arg renamed to --service to be consistent with other commands
      • --line_match arg renamed to --wait_until
      • default timeout now to 60s
      • failing to find the “wait_until” pattern will be interpreted as a failure of the entire script instead of keep going (unless systemd thinks the service is “active”)
      • longer debug output in CI context to ease debugging
    • ynh_compare_current_package_version --comparison lt --version X.Y~ynhZ becomes
      ynh_app_upgrading_from_version_before and ynh_app_upgrading_from_version_before_or_equal_to
    • Replace the clumsy if [ "$upgrade_type" == "UPGRADE_APP" ] with a simpler if ynh_app_upstream_version_changed …
    • Replace if [ "${PACKAGE_CHECK_EXEC:-0}" -eq 1 ] with simpler if ynh_in_ci_tests
    • Use positional args for ynh_normalize_url_path
  • Legacy cleanups
    • backport most of the bookworm changes (remove unused stuff)
    • remove unecessary --app=$app in internals ynh_app_setting_get/set calls
    • remove support for __NAME__ and __NAMETOCHANGE__ replaced by $app in templates
    • Drop support for old .src format in ynh_setup_source
    • Remove legacy ynh_legacy_permissions_exists and ynh_legacy_permissions_delete_all
    • Remove legacy ynh_webpath_available and ynh_webpath_register
    • Remove legacy ynh_psql_test_if_first_run
    • Remove legacy --nonappend/--non-append arg in logrotate helper
    • Remove legacy ynh_backup_before_upgrade and ynh_restore_upgradebackup, handled by core in packaging v2
    • Remove legacy ynh_find_port, ynh_port_available, ynh_validate_ip4/6, keep only ynh_validate_ip
    • Remove ynh_get_debian_release, apps should just use $YNH_DEBIAN_VERSION
    • Remove ynh_require_ram and unused --ignore_swap and --only_swap options in ynh_get_ram
    • Simplify ynh_app_upstream_version and ynh_read_manifest
    • Remove legacy/unecessary/underused helpers: ynh_print_log, ynh_print_err, ynh_exec_err, ynh_exec_quiet, ynh_exec_fully_quiet, ynh_print_OFF, ynh_print_ON
    • Remove a whole bunch of unused args in mongo helpers…
    • Remove unused options --label/--show_tile/--protected in ynh_permission_update
    • Remove old internal ynh_render_template, should use ynh_add_config --jinja instead
9 Likes
2

Impressive work !