Restoring whole yunohost from borg backups

[chateau]

My YunoHost server

Hardware: Computer (yunohost is running in a Virtual Machine)
YunoHost version: don’t know yet, but will start from debian stretch
I have access to my server : Through SSH, through the webadmin
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : except the VM aspect mention before, no

Description of my issue

Following a hardware failure, the server I’m running has lost all data (including the whole yunohost system), so I’m trying to figure out the proper step by step process to totally reinstall yunohost from a remote borg backup that I have.
I stumbled upon many parts of what has to be done in different parts of the forum, but I’m trying to have the whole process very clear, not to do mistake as I go, that would generate more problems.
Maybe some of this process is explained in yunohost documentation, but I don’t remember seeing all the steps, and also I can’t get to access the website at the moment.

So here is what I imagine I have to do:
Can anyone correct me, or add things, if I forgot some steps?

1. Install debian stretch in the Virtual Machine
2. Install yunohost the usual way
3. Go through the postinstall process? — I’m wondering if I should do that because it will create the default user, setup a domain, but I want to reimport my previous configuration from the remote borg backup I have. Will the backup restoration replace what I’m doing at this step, and therefore I can put dummy domain and user names… ?
4. Install borg_ynh app and manually set it up the way it was before — But from what I remember, this will generate an ssh key to connect to the remote server, but the remote server is expecting the previous key, that has been lost? — But if I don’t reinstall borg, how can I reimport backups from the remote server following the process described in this thread?.
5. I guess at this point, I will have to connect to the remote backup server, and add the newly created ssh public key to it’s ssh config, so that the new server can connect to it.
6. Now, I can run borg export-tar BORG_USER@DOMAIN_OF_BACKUP_SERVER:backup::auto_conf_XXXX /home/yunohost.backup/archives/auto_conf_XXXX.tar.gz in the new server to reimport the remote backup locally.
7. Then I can run yunohost backup restore auto_conf_XXXX which will restore the configurations (= users, domains…)
8. Now I can do the same process for data (borg export-tar..., yunohost backup restore...)
9. Then I can restore each apps following the same process again — I don’t need to reinstall the apps before restoring them, right?

Maybe I’m making it more complicated than it is. It just bugs me this thing of having to first setup the server like new, install borg, to then be able to import backups. But I don’t really see another proper way I could do that.
The only other way I could imagine, would be to extract borg backups in the remote server, and then send them through ssh to the new yunohost, but it would mean decrypting the data in the remote server, which shouldn’t have to happen.

Any help, suggestions, and remarks are highly welcome in these uncomfortable times!!!

[Mamie]

I have never done it before, but I think the steps should be :

1. Install the most basic YunoHost (same version as your backups, to be sure)
2. Via whatever computer, create the .tar.gz from the backups (no need to install borg on your new server for this, unless this is for installing the borg commands)
   Note that to connect to the borg server, if everything is configured by default, you will need a private key (that you may have lost) and the archive key.
   If you have lost the archive key, I think all is lost.
   If you lost the ssh key, just add a new one on the backup server, or work directly on it, or from any computer, the important part is that the .tar.gz ends on your new server.
3. Now you can use the yunohost restore command (and I have no idea about restoring uninstalled apps but it should work)

[ljf]

How many Gigabytes did you have before the crash?
How much space do you have on the new one?
What is the upload rate on your backup server, and the download rate on your new server?

How to proceed depends on the answers to these questions…

In the easiest case, I would say that what you propose is not so bad:

1. Install yunohost (you can install a yunohost 4.0.x (buster) version even if you was in stretch before, but only if your backuped apps was up to date)
2. You need to do the postinstall to setup borg app
3. After the setup of borg, you should go in the backup server to replace the ssh key in /home/USER/.ssh/authorized_keys
4. On the new server identify which archive you want to restore:

BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg list ssh://BORG_USER@DOMAIN_OF_BACKUP_SERVER:PORT/~/backup

5. Then recreate tar.gz archives:

BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg export-tar ssh://BORG_USER@DOMAIN_OF_BACKUP_SERVER:PORT/~/backup::ARCHIVE /home/yunohost.backup/archives/ARCHIVE.tar.gz

6. Finally you can restore in the order you give (conf, data and finally apps per apps)

yunohost backup restore conf --system
yunohost backup restore data --system
yunohost backup restore nextcloud --apps
yunohost backup restore concrete5 --apps

If you have a lot of data in your nextcloud, you can use borg extract command to extract directly all nextcloud data into /home/yunohost.app/nextcloud/data/ . And next you can create a nextcloud archive without data with:

BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg export-tar -e apps/nextcloud/backup/home/yunohost.app  ssh://BORG_USER@DOMAIN_OF_BACKUP_SERVER:PORT/~/backup::ARCHIVE /home/yunohost.backup/archives/ARCHIVE.tar.gz

You can do the same process for /var/mail in data archive.

I know it’s quite difficult to do all of that, and some devlopement should appear to ease this process.

Note: it could be more difficult if you have not enough bandwidth or space, or if you use a noho.st, nohost.me or ynh.fr domain…

[chateau]

Thanks a lot @Mamie and @ljf for your answers!

To answer to @ljf questions:
There is 80Go of archives in borg backup, and 300Go of disk space in the new server.
I don’t know the upload rate of the backup server (something like the usual download rate of a home connection I guess, if you have any hint on how to check that, I can do it).
The download rate of the new server is quite high and will probably not be limiting.

I will try the process you propose @ljf, some clarification question though:

• The commands: yunohost backup restore conf --system, yunohost backup restore nextcloud --apps… don’t need to specify the name of the backup file to use? (In another thread, you were proposing to use the command: yunohost backup restore auto_XXXX.)
• I didn’t have the opportunity to update apps in the last 1-2 months, so I think it would be wiser to install yunohost debian stretch first. The problem is that it seems that the installer is now only designed for debian buster. Do you know of a way to install an older version of yunohost?

[chateau]

For the record, this is a complete description of the process I followed:

Restore a complete system from borg backups

For infos on restoring process, check this yunohost forum thread and that one, also using borg with sshkeys, the borg extract documentation, and this general tutorial on borg backup.

In the following explanations:

• the server to backup/restore will be called: yuno
• the remote server that receives and store the back will be called: rem
• rem is accessible at the domain rem.tld
• the remote user on rem which owns the borg backups will be called yurem
• backup files will be stored in rem in the directory: /home/yurem/backup

Overview

The idea here, if you need to restore a whole yunohost system is:

1. Install a new debian VM
2. Install yunohost in it the usual way
3. Go through yunohost postinstall (parameters you will supply are not crucial, as they will be replaced by the restore)
4. Install borg
5. Setup rem to accept ssh connections from yuno
6. Use borg to import backups from rem to yuno
7. Restore borg backups with the yunohost backup restore command, first config, then data, then each app one at a time
8. Remove the borg app and restore it

Make it possible for yuno to connect to rem with borg

At this stage, we will assume that yuno is a freshly installed yunohost (based on buster in my case). You should also have performed the yunohost postinstall.

If you don’t want to restore the whole system, just some apps, you can skip some of the steps below.

Install the borg yunohost app in yuno

The idea here is just to install borg, not in order to create backups, but only to use borg commands to import remote backups.

So for example, you can install it doing the following:

sudo yunohost app install borg -a "server=rem.tld&ssh_user=yurem&conf=0&data=0&apps=hextris&on_calendar=2:30"

Make sure that rem accepts ssh connections from yuno

In yuno you will need to get the ssh key that borg just created while installing: sudo cat /root/.ssh/id_borg_ed25519.pub, copy it to clipboard.

Connect via ssh to rem, go to /home/yurem/.ssh/authorized_keys, and past the borg public key you got at previous step.

Now to make sure this worked, you can try to ssh from yuno to rem.
In yuno : ssh -i /root/.ssh/id_borg_ed25519 yurem@rem.tld . If you can get into rem , without it prompting for a password, then you’re good to continue

Restore backups to yuno

For the commands in the following section to work, you will need to be root in yuno (you can become root running sudo su).

Restoration of backups can take quite a while, you’d better do them in a separate process, so that it doesn’t stop if your terminal session gets closed. For this, you can for example use tmux.

In yuno now, you should be able to list backups in rem with the following command:

SRV=yurem@rem.tld:/home/yurem/backup
BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg list $SRV

You can then reimport one to yuno with:

BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg export-tar $SRV::auto_BACKUP_NAME /home/yunohost.backup/archives/auto_BACKUP_NAME.tar.gz

And then restore the archive in yuno with:

yunohost backup restore auto_BACKUP_NAME --system # for config and data backups
yunohost backup restore auto_BACKUP_NAME --apps # for other backups (=apps)

And nextcloud? It’s super heavy!!

For nextcloud, the best is probably to reimport the backup without the data. And to import the data manually.

For that, you can do the following (as root):

SRV=yurem@rem.tld:/home/yurem/backup

# export the app without data
BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg export-tar -e apps/nextcloud/backup/home/yunohost.app $SRV::auto_nextcloud_XX_XX_XX_XX:XX /home/yunohost.backup/archives/auto_nextcloud_XX_XX_XX_XX:XX.tar.gz

# extract the data from the backup to the nextcloud folder
cd /home/yunohost.app/nextcloud
BORG_RSH="ssh -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg extract $SRV::auto_nextcloud_XX_XX_XX_XX:XX apps/nextcloud/backup/home/yunohost.app/nextcloud/
mv apps/nextcloud/backup/home/yunohost.app/nextcloud/data data
rm -r apps

# now you can simply restore nextcloud app
yunohost backup restore auto_nextcloud_XX_XX_XX_XX:XX --apps

Restore borg

Once you’ve restored the whole system, you will probably want to restore the borg app as well.

For that, remove the “dummy” borg you installed to do the restoration, and restore borg the same ways as for other apps:

sudo yunohost app remove borg
sudo yunohost backup restore auto_borg_XX_XX_XX_XX:XX --apps

[ljf]

Thanks for this tuto, i copy it in the readme of borg_ynh