How to backup your YunoHost server on another server

Edit: AN APP SIMPLIFY NOW THIS TUTORIAL, SEE HERE: https://github.com/YunoHost-Apps/borg_ynh/blob/master/README.md


I let this tuto for people who want to know the general step the app made (not exactly the same)

Context

We want to backup Camille’s server on Sam one’s.

Camille : lachenille.tld
Sam: thedriver.tld

You need to setup borg from backport and initialize a repo.

Create a private key on Camille’s server

Camille creates a private and a public key:

$ ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/root/.ssh/id_ed25519): 

Don’t specify password

Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/root/.ssh/id_ed25519.
Your public key has been saved in /home/root/.ssh/id_ed25519.pub.
The key fingerprint is:
SHA256:RDaEw3tNAKBGMJ2S4wmN+6P3yDYIE+v90Hfzz/0r73M root@lachenille.tld
The key's randomart image is:
+--[ED25519 256]--+
|o*...o.+*.       |
|*o+.  +o ..      |
|o++    o.o       |
|o+    ... .      |
| +     .S        |
|+ o .            |
|o+.o . . o       |
|oo+o. . . o ....E|
| oooo.     ..o+=*|
+----[SHA256]-----+

Then Camille send the public key to Sam. For example by email

$ cat /root/.ssh/id_ed25519.pub | mail sam@thedriver.tld

Prepare the way for the remote repository on Sam’s server

Sam creates a specific user for camille:

$ adduser camille

Sam adds Camille’s public key with borg limitation

$ echo "command=\"borg serve --restrict-to-path /home/camille/backup\",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-user-rc ssh-ed25519 AAAAC3NzaC1l[...]YRPujWoRyeVhhFZ0M root@lachenille.tld" >> /home/camille/.ssh/authorized_keys

Next Sam install borg-backup

$ apt-get install python3-pip python3-dev libacl1-dev libssl-dev liblz4-dev
$ pip3 install setuptools --upgrade
$ pip3 install borgbackup

If Sam’s server is a YunoHost, you need to avoid to backup camille’s backuped data. It’s very important if Sam’s server is backuped to Camille’s server.

$ touch /home/camille/.nobackup
$ mkdir -p /etc/yunohost/hooks.d/backup
$ cat > /etc/yunohost/hooks.d/backup/17-data_home <<EOF
#!/bin/bash

# Exit hook on subcommand error or unset variable
set -eu

# Source YNH helpers
source /usr/share/yunohost/helpers.d/filesystem

# Backup destination
backup_dir="\${1}/data/home"

# Backup user home
for f in \$(find /home/* -type d -prune | awk -F/ '{print \$NF}'); do
    if [[ ! "\$f" =~ ^yunohost|lost\+found ]]; then
        if [ ! -e "/home/\$f/.nobackup" ]; then
            ynh_backup "/home/\$f" "${backup_dir}/\$f" 1
        fi
    fi
done
EOF

Setup borg on Camille’s server

$ apt-get install python3-pip python3-dev libacl1-dev libssl-dev liblz4-dev
$ pip3 install borgbackup

Create the remote repository from Camille’s server

$ borg init -e repokey camille@lachenille.tld:backup

You need to conserver the passphrase you set

Patch Camille’s Yunohost server

(Edit by Alex : this step is not necessary with Yunohost >= 2.7.x)

Currently, you need a patch to support custom backup methods

$ cd /usr/lib/moulinette/yunohost
$ rm backup.py
$ wget https://raw.githubusercontent.com/YunoHost/yunohost/ead685c455939c1c830a426e171f3e9c6266b5ad/src/yunohost/backup.py
$ cd /usr/share/moulinette/actionsmap/
$ rm yunohost.yml
$ wget https://raw.githubusercontent.com/YunoHost/yunohost/ead685c455939c1c830a426e171f3e9c6266b5ad/data/actionsmap/yunohost.yml

And run

mkdir -p /etc/yunohost/hooks.d/backup_method
mkdir -p /usr/share/yunohost/backup_method

Add the myborg hooks on Camille’s server

This hooks is a custom backup method. Don’t forget to replace repo and BORG_PASSPHRASE. If the passphrase contains double quote or dollar don’t forgot to escape those caracters with \

/etc/yunohost/hooks.d/backup_method/05-myborg

#!/bin/bash

set -e

BORG_PASSPHRASE="XXXXXXXX"
repo=camille@thedriver.tld:backup   #$4

do_need_mount() {
    true
}

do_backup() {
    
    export BORG_PASSPHRASE
    work_dir=$1
    name=$2
    repo=$3
    size=$4
    description=$5
    LOGFILE=/var/log/backup_borg.log
    ERRFILE=/var/log/backup_borg.err
    current_date=$(date +"%d_%m_%y_%H:%M")
    pushd $work_dir
    borg create $repo::${name}_${current_date} ./ >> $LOGFILE 2>> $ERRFILE
    popd

    borg prune $repo -P ${name} --keep-daily=7 --keep-weekly=8 --keep-monthly=12 >> $LOGFILE 2>> $ERRFILE
}

do_mount() {
    export BORG_PASSPHRASE
    work_dir=$1
    name=$2
    repo=$3
    size=$4
    description=$5
    LOGFILE=/var/log/backup_borg.log
    ERRFILE=/var/log/backup_borg.err
    borg mount $repo::$name $work_dir >> $LOGFILE 2>> $ERRFILE
}

work_dir=$2
name=$3

size=$5
description=$6

case "$1" in
  need_mount)
    do_need_mount $work_dir $name $repo $size $description
    ;;
  backup)
    do_backup $work_dir $name $repo $size $description
    ;;
  mount)
    do_mount
    ;;
  *)
    echo "hook called with unknown argument \`$1'" >&2
    exit 1
    ;;
esac

exit 0

Test

$ yunohost backup create --ignore-apps --system conf_ldap -n test --methods myborg --debug --verbose

Run the backup each night

If it’s ok you can create a cron to run the command each night
/etc/cron.daily/yunohost-99-backup

#!/bin/bash

if yunohost -v | grep "version: 2." > /dev/null; then
    ignore_apps="--ignore-apps"
    ignore_system="--ignore-system"
else
    ignore_apps=""
    ignore_system=""
fi
filter_hooks() {
    ls /usr/share/yunohost/hooks/backup/ /etc/yunohost/hooks.d/backup/ | grep "\-$1_" | cut -d"-" -f2 | uniq
}

# Backup system part conf
yunohost backup create $ignore_apps -n auto_conf --methods myborg --system $(filter_hooks conf)

# Backup system data
yunohost backup create $ignore_apps -n auto_data --methods myborg --system $(filter_hooks data)

# Backup all apps independently
for app in $(yunohost app list --installed -b | grep id: | cut -d: -f2); do
    backup_methods=$(yunohost app setting $app backup_methods)

    if [ -z "$backup_methods" ]; then
        backup_methods=myborg
    fi

    if [ "$backup_methods" != "none" ]; then
        yunohost backup create $ignore_system -n auto_$app --methods $backup_methods --apps $app
    fi
done

Don’t forget to give rights to execute !

$ chmod a+x /etc/cron.daily/yunohost-99-backup

Disable the backup for a particular app

yunohost app setting strut backup_methods -v none

Backup a particular app with another methods

Create your specific methods

cp /etc/yunohost/hooks.d/backup_method/05-myborg /etc/yunohost/hooks.d/backup_method/15-mynextcloudborg
vi /etc/yunohost/hooks.d/backup_method/10-mynextcloudborg

Specify the method to use for the targeted app

yunohost app setting nextcloud backup_methods -v mynextcloudborg

[EDIT] LIMITATIONS

This tuto works if you are able to transfer your initial data within a maximum of 24 h.

So if you have an ADSL and more than 20G to transfer (at the first time) you should be careful.

How to restore

On a new server

If you are not on the same yunohost instance, you should copy the private key to access to camille server or recreate a private key and ask camille to put the new one.

Next, do:

borg list camille@lachenille.tld:backup

You need to identify recent backup. Conf, data and app have there own backup archives.

Begin by restoring conf

borg export-tar camille@lachenille.tld:backup::auto_confXXXX /home/yunohost.backup/archives/auto_confXXXX.tar.gz
yunohost backup restore auto_confXXXX

And next, restore data and each apps with the same method.

On the same server

do:

borg list camille@lachenille.tld:backup

You need to identify recent backup. Conf, data and app have there own backup archives.
Eventually remove the app you want restore.

borg export-tar camille@lachenille.tld:backup::autoXXXX /home/yunohost.backup/archives/autoXXXX.tar.gz
yunohost backup restore autoXXXX

[Optionnal] Backup on 2 servers (with a big remote backup space and a small one)

Add a new backup method and edit Passphrase and repo destination

cp /etc/yunohost/hooks.d/backup_method/05-myborg /etc/yunohost/hooks.d/backup_method/10-mysmallerborg
vi /etc/yunohost/hooks.d/backup_method/10-mysmallerborg

Change /etc/cron.daily/yunohost-99-backup with this content and edit size limit as you want:

#!/bin/bash

if yunohost -v | grep "version: 2." > /dev/null; then
    ignore_apps="--ignore-apps"
    ignore_system="--ignore-system"
else
    ignore_apps=""
    ignore_system=""
fi

# Size limit in MB per apps above which the script just backup on one server
declare -A limit
limit[nonessential]=10
limit[important]=150
limit[critical]=1024

# List of backup profiles
declare -A backup_profiles
backup_profiles[simple]="myborg"
backup_profiles[double]="mysmallerborg myborg"
backup_profiles[nobackup]="none"
if [ ! -e /etc/yunohost/hooks.d/backup_method/*-mysmallerborg ]; then
    backup_profiles[double]="myborg"
fi

# Select backup methods to apply
get_backup_methods() {
    importance=$1
    size_max=$2 # In MB or "unlimited"
    if [ "$size_max" == "unlimited" ]; then
        size_max=$(expr $(df --output=size / | sed -n '2 p') / 1000)
    fi

    if (( $size_max > ${limit[$importance]} )); then
        echo ${backup_profiles[simple]}
    else
        echo ${backup_profiles[double]}
    fi
}
filter_hooks() {
    ls /usr/share/yunohost/hooks/backup/ /etc/yunohost/hooks.d/backup/ | grep "\-$1_" | cut -d"-" -f2 | uniq
}

# Backup system part conf
yunohost backup create $ignore_apps -n auto_conf --methods $(get_backup_methods important 50) --system $(filter_hooks conf)

# Backup system data
yunohost backup create $ignore_apps -n auto_data --methods $(get_backup_methods important unlimited) --system $(filter_hooks data)

# Backup all apps independently
for app in $(yunohost app list --installed -b | grep id: | cut -d: -f2); do
    backup_methods=$(yunohost app setting $app backup_methods)

    if [ -z "$backup_methods" ]; then
        size_max=$(yunohost app setting $app max_size)
        size_max=${size_max:-unlimited}
        importance=$(yunohost app setting $app importance)
        importance=${importance:-important}
    	backup_methods=$(get_backup_methods $importance $size_max)
    fi

    if [ "$backup_methods" != "none" ]; then
        yunohost backup create $ignore_system -n auto_$app --methods $backup_methods --apps $app
    fi
done

Add max_size and importance to an app:

yunohost app setting wordpress max_size -v 500
yunohost app setting wordpress importance -v critical

yunohost app setting strut max_size -v 100
yunohost app setting strut importance -v nonessential

yunohost app setting nextcloud max_size -v 250000
yunohost app setting nextcloud importance -v important
3 Likes

There’s no PR opened for https://github.com/YunoHost/yunohost/compare/enh-615-be-able-to-backup-with-borg at the moment ?

1 Like

I successfully tested it and it works ! That’s pretty impressive :smile: :tada:

There are a few tweaks missing though :

  • I had to create the .ssh folder for Camille on thedriver.tld (Sam’s server) and take care of the permissions.
  • I had to tweak the /etc/ssh/sshd_config on thedriver.tld (Sam’s server) to allow Camille to connect through ssh
  • On lachenille.tld (Camille’s server), when running borg command with a custom identity file for the SSH repo, one needs to export the following variable : export BORG_RSG="ssh -i /path/to/private/key" (c.f. Borg doc)

Possible refinements :

  • In the automatic backup, add a prune with

$ attic prune camille@thedriver.tld:backup --keep-daily=7 --keep-weekly=4 --keep-monthly=6

  • Add quotas for Camille on Sam’s server (though it requires quite some tweaks :confused: )

Note: this tuto works if you are able to transfer your initial data within a maximum of 24 h.

So if you have an ADSL and more than 20G to transfer (at the first time) you should be careful.

Pretty nice feature !

Possible refinement #2 :
If I were to host backups for friends on my server, I would prefer having a single user account dedicated to “backup hosting” for everyone, instead of having to create a specific user for each hosted friend, managing ssh access for this user, etc.
More or less like git servers do to authenticate users with their ssh key, while still using a single “git” account. Seems to me it would reduce the adminsys burden. This might make it easier also to interface this with a tiny web ui.

Need to experiment, but the “command” in authorized_keys allows for this pretty easily, if the “restrict to path” option is specified to point to a different path inside the “backup hosting” user home dir.

Well, to be discussed, but the mid/long-term purpose is to automatize the whole process of creating the user and adding the ssh key anyway … so it will be pretty trivial.

To me it would seem weird to have a single user with many “users inside the user”. Also having separate users allow to manage quotas for them individually (though quota isn’t gonna be easy to implement from what I see, but I didnt do so much research about that yet)

Hi!
Could you add how to use borg locally ? For example on an external usb disk ? It should be quite easy, few things to change, but I’m not sure I understand this “how to” enough to do it myself…
Thanks !

Hi

I have a problem with borg
When I start the backup I have this error https://srvmaison.fr.nf/zerobin/?267b2e8d64071474#y95zysrOo3DhnLH4bqi96BjZ5TIMC/C02Fdy4d+hJts=
The second server is not a yunohost and it works with debian stretch
I have tried to start backup with this borg create serveur1@192.168.1.37:backup::test_06_10_17_16:52 ./ and I have Repository path not allowed

Here you can see how to backup on a local hard drive

1 Like

Hi everybody, @ljf ,

Do you think this work could be packaged or merged in some way in Yunohost? There is a huge interest to integrate Borg in the Yunohost backup process. I’m ready to discuss and help to do it.

Tell me how far you’ve come now!

@klorydryk
Yes it could be directly in the Yunohost core.

See here the roadmap THE Backup roadmap

Since the custom backup method, I hadn’t had time to work on it, but we could discuss it on mumble or on the dev XMPP chat if you want :slight_smile:

1 Like

Hello!

Since Yunohost 3.0, it seems that --ignore-system option is not available.
Can I just remove it from the CRON job script?

Thanks,

Léo

Yes, idem for ignore-apps . Thanks for your report, I have edited the initial post with a condition to detect if we are on 2.7 or 3.0.

1 Like

Hi ljf,

It worked fine, I just had to update the borg vers using pip3 install borgbackup again and run again mkdir -p /usr/share/yunohost/backup_method.
Once done, with your edits, that worked fine.

Thanks,

Léo

An idea for Nextcloud :
Be able do 2 backups for nextcloud : one for the data and the other for the rest via yunohost backup method & backup_core.
The data path can be specify by admin (or setup by the hook script ?)
It will be activated or disactivated by a commented line.

By this way, you let the admin restore only the config and/or the data. The restore operation is faster --> not necessary to make a big fat tar.gz with all data. Just a tar.gz with config and an restore for the data

Or it’s a bad idea ?

I am understandable ?

That’s the kind of things I’d like to do too.
As I understood, for a backup, everything is extracted/copied to a tmp folder, and then the backup is made (tar.gz, bord, whatever you want) but for nextcloud (or maybe any other app with files that are directly files and can take a huge amount of data) the copy part of the backup is (I think) a waste of time and the backup could work directly from the real files.

Maybe that’s something already done and I didn’t read enough, but that is what I understood.

No, but maybe.

To create the tar.gz yunohost use directly the original file to add it to the tar.gz.

With borg, it’s quite different, because we need to organize the directory before. So Yunohost make mount bind in readonly mode and hardlinks to achieve this. In some case, it’s not possible, so the files/dir we are not able to link are copied. It could be the case with files on another disk.

There is an option for backup only configuration of an app.

It’s already possible, but only in cli mode:

BACKUP_CORE_ONLY=1 yunohost backup create --apps nextcloud

Hi @Aleks,
I’m currently stuck with a SSH authentification issue (all details in this thread Borgserver sur une machine non yunohost) and I wish to have more details please.

  • wich permissions are set for Camille on Sam’s server?
  • did you mean you add explicitly AllowUsers camille in etc/ssh/sshd_config? Or other things?
  • I’m not sure I understand the last point…

My problem is that I can backup with borg Camille datas on Sam’s server manually (ie: send in a shell sudo borg init [...] BUT it doesn’t work when I launch the Yunohost backup command: Sam’s server always ask for a password and ignore the authentification keys previously set up :frowning:

Thanks for your help.