Yunohost backup create KO

martoni35 · April 15, 2019, 9:38am

Bonjour,

Depuis quelques jours, la backup yunohost ne fonctionne plus. Il semble que nextcloud pose problème lors du message d’erreur suivant rencontré :

Voici ma version de yunohost sur une VM PROXMOX :

Avec ce type d’erreur, j’ai tout d’abord vérifié l’espace disque mais c’est OK, vérifié s’il n’y avait pas des fichiers mal supprimé avec lsof et enfin, j’ai ajouté dans la config de nextcloud le paramètre backup_only_core=1 :

Ensuite, lors d’un test manuel, en settant la variable cela fonctionne :

Il semble que la variable suivante BACKUP_CORE_ONLY soit aussi checker en plus de backup_only_core (d’ailleurs, j’avais déjà ce paramètre auparavant et tout était OK) et en automatique elle est égale à 0 donc plantage car j’ai trop de data donc erreur normale :

Par contre, comme je ne veux pas du backup des datas, comment setté ce paramètre qui est arrivé depuis la dernière MAJ nextcloud en gros ?

En vous remerciant,

martoni

ljf · April 15, 2019, 11:25am

Tu utilises quelle méthode automatique ? Un truc fait main ? Le tuto borg? L’ap borg ? Archivist ?

martoni35 · April 15, 2019, 11:38am

Hello,

Pour la partie en automatique, j’ai juste en crontab la commande suivante :

yunohost backup create

Ceci fonctionnait très bien quotidiennement jusqu’à la MAJ nextcloud d’après mes recherches.

martoni

ljf · April 15, 2019, 12:23pm

La mise à jour a peut être remis backup_core_only à 0…

Je te suggère de modifier ton cron ainsi:

yunohost backup create --apps app1 app2 --system
BACKUP_CORE_ONLY=1 yunohost backup create --apps nextcloud

martoni35 · April 15, 2019, 12:29pm

Hello,

Le paramètre est bien dans la config de nextcloud :

Ta proposition contourne le souci malgré que cela marcherait.
Cependant, c’est une rustine qui ne me règlepas la root cause, on est d’accord je pense ?

Est-ce qu’il y a des fichiers de config (yunohost) qui ont évolué avec l’intégration de la dernière version de nextcloud par rapport à avant ?

martoni

ljf · April 15, 2019, 12:45pm

Après recherche il s’agit d’un refactoring du script de backup qui a fait disparaître cette option.

A priori je pense que ça concerne de nombreux scripts.

Avant:

github.com

YunoHost-Apps/nextcloud_ynh/blob/34e1085a9f94e8863fce2310a5d63aa8aaa52a9f/scripts/backup#L25


# Retrieve app settings
domain=$(ynh_app_setting_get "$app" domain)
path=$(ynh_app_setting_get "$app" path)
dbpass=$(ynh_app_setting_get "$app" mysqlpwd)


# Copy the app source files
DESTDIR="/var/www/$app"
ynh_backup "$DESTDIR" "www"


# Copy the data directory
backup_core_only=$(ynh_app_setting_get "$app" backup_core_only)
if [ -z $backup_core_only ]	# Si backup_core_only a une valeur dans le fichier settings.yml, ne sauvegarde pas le dossier data
then
	DATADIR="/home/yunohost.app/${app}/data"
	ynh_backup "$DATADIR" "data" 1
else
	echo "Data dir will not saved, because backup_core_only is set." >&2
fi


# Copy the conf files
mkdir ./conf

Après:

github.com

YunoHost-Apps/nextcloud_ynh/blob/master/scripts/backup

#!/bin/bash

#=================================================
# GENERIC START
#=================================================
# IMPORT GENERIC HELPERS
#=================================================

source ../settings/scripts/_common.sh
source /usr/share/yunohost/helpers

#=================================================
# MANAGE SCRIPT FAILURE
#=================================================

# Exit if an error occurs during the execution of the script
ynh_abort_if_errors

#=================================================
# LOAD SETTINGS

This file has been truncated. show original

ljf · April 15, 2019, 1:01pm

I made a fix, you can apply it manually and say if it works:

martoni35 · April 15, 2019, 1:16pm

Hello,

This fix was apply manually with success

The merge could be done

martoni35 · April 15, 2019, 1:25pm

Hello,

New check Ok with new changes “auto”

Maniack_Crudelis · April 15, 2019, 4:17pm

How funny …

I use for a long time this helper, design to check the available space before any backup.
This helper isn’t in the core of YunoHost because someone told me many times that it was useless because the backup process already check the available space… That I denied many times…

Never mind…

backup_core_only has been removed from the config and used as an environment variable because we had many issues with the setting into the config, especially with failed backups.
That’s not an error, nor a bug. We have changed the way this setting is handled in order to manage it from the backup helper.
We probably lack some communication about it, but the best way to manage that is to fix your backup script, not to restore the old behavior into the app.

ljf · April 15, 2019, 5:38pm

I tell you that because it’s true, the core does the check. How you explain the first error message in the first screenshot of this topic “L’espace disque est insuffisant sur /home/yunhost.backup/archives” ? This message is displayed because we check that before to do any backup. If no, there wouldn’t be free space on the ynh instance, and it’s not the case.

Note: Info message “Backing up…” have been added after and should be corrected by a message like “Adding SOMETHING to backup list”. Indeed at this step nothing is backup…

Summary of the backup process (in the case of tar method):

Collect of all path to backup and generation of database dump
Estimation of backup size
Add each path to the new archive

So in this case, the backup fail at step 2, before to do the true backup.

ljf · April 15, 2019, 5:42pm

With this change, is there a way to let end users configuring their backup to ignore big data ? If yes my PR isn’t necessary, if no I think i have made better than the “old behaviour”…

Maniack_Crudelis · April 15, 2019, 5:55pm

Of course there is, we didn’t just removed a needed option. We have replaced it because it was causing issues.
As explained in the documentation,https://github.com/YunoHost/doc/blob/master/backup.md#apps-specific-configuration, to ignore big data, the backup command should be preceded by BACKUP_CORE_ONLY=1

ljf · April 15, 2019, 6:12pm

So you think the first solution i suggested in this topic is the good one:

I see 3 (minor) limits with this way:

we loose retrocompat (but it make several month now, i don’t know if it’s really important)
users can’t use the webadmin to make backup manually with a backup_core_only preconfiguration
no way to create a mono archive with this method

I think we should search a better way with the new work on backup.

Maniack_Crudelis · April 15, 2019, 6:32pm

We loose retro-compatibility only for users that use custom backup script, not for the upgrade script.
For those users, they can just modify their scripts.

There were no way before to set backup_core_only from the webadmin, nor there is now. This settings wasn’t made to stay set, scripts remove it when it’s set.
We have to keep in mind that this setting was existing for the upgrade script.

If you need to ignore big data, you can set this variable for a “mono” archive if you want. It will be used for all apps having big data.

Gofannon · April 21, 2019, 3:29pm

Hello there,

Just bumped on a similar issue while doing a backup with the “new flag” as said here [Official app] Nextcloud

After following the documentation https://github.com/YunoHost/doc/blob/master/backup.md#apps-specific-configuration, I modified my linux command like thisBACKUP_CORE_ONLY=1 yunohost backup create --description 'Sauvegarde complete sans donnees nextcloud et transmission'

I end up with a 20 gb backup instead of 1gb Not good! After digging, it appears that transmission is not “compatible” with this new feature/behaviour.

Is there any list of apps that needs to be fixed to handle this setting?
Is there any plan to fix them?
Is there an example/documentation to modify the scripts? (I’m willing to adapt Transmission)

Maniack_Crudelis · April 21, 2019, 3:42pm

I’m currently upgrading all Official apps to last packaging standards, I’m on Synapse currently, Transmission will be the next one.
However, as I’m going to be unavailable next week, it’s not going to be until 1 or 2 weeks.

We don’t have any list of apps to upgrade. I guess though you can search backup_core_only into YunoHost-Apps to find them, https://github.com/search?q=org%3AYunoHost-Apps+backup_core_only&type=Code

Not sure any other apps than nextcloud have been updated yet. So you can have a look to nextcloud.

Gofannon · April 23, 2019, 12:17pm

we loose retrocompat (but it make several month now, i don’t know if it’s really important)

“just modify”, not sure that all people using Yunohost will be aware of this change when it will break their backups.
Especially that all applications are not compatible with the change yet (hello Transmission)

users can’t use the webadmin to make backup manually with a backup_core_only preconfiguration

There should be a way to do it graphically. Or it will come later when there will be a way to download easilly the backups from the web admin?

Could you elaborate on this please ?

There should be two backup system then? One for upgrade and one for the user to decide which application “big data” to use? (in my case, backup nextcloud but not transmission. I might also avoid backuping Nextcloud data as they are already replicated on other devices)

Or maybe, having a mechanism to choose what to backup with checkbox in GUI. (it will be able to “overide” and remove “big folder” to be backuped)

Nextcloud : backup all
transmission : backup settings only (no data)
baikal: backup all
[…]

no way to create a mono archive with this method

Not an issue for someone using the CLI

Maniack_Crudelis · April 27, 2019, 10:39pm

I was talking about user using a custom script for their backups. We can’t keep retro-compatibility for custom scripts.

By “We have to keep in mind that this setting was existing for the upgrade script.” I meant that this setting has been added for upgrade script when doing a backup before upgrade.
Indeed, some could want to use it as well for their recurrent backups. But that wasn’t the idea when the option has been added.

Now, considering the experimental feature Config-Panel, we can imagine adding a setting for backup of big data for apps that would use that kind of setting.
But this setting should be different of the one used by the upgrade script.

I understand the need of such an option. I think Config-Panel is a good way to do it, both graphically or by CLI.

martoni35 · May 9, 2019, 6:54am

Hello,

Depuis la migration en 3.5.2, j’ai à nouveau la même erreur qui est revenu !!!
J’ai tenté d’appliquer le code dans [fix] backup_core_only settings doesn’t work
En effet, la partie du code ajouté dans ce fix était absente.
Nouveau test, toujours Ko avec le même message que ce billet …

Pourriez-vous checker les modifications apportées à cette version 3.5.2 ?

En vous remerciant,

Cordialement,

martoni