[Archivist] Backup your server with rsync

,

Hi pals

Again, I’m back with a new app !
Because my previous app is quite incomplete if you don’t have anything for managed your backups, now we have Archivist \o/

This new app (like the 2 previous, more a cli script than a real app) will give you a easy way to make periodical backups and send them where you want, without any action from your part.

How it works ?
It’s quite simple, there’s a config file where you can say which part of YunoHost you want to backup, and where you can also add all the directories or files you want to backup too.

After that, at the end of this same config file, you can add as many recipient as you want.
Your backups can be encrypted before be sending, and you can even exclude some parts for a recipient, if you want.
Periodically (each day to each month, as you want), your backups will be updated, then send to your different recipients.

You can choose how your backups will be sends, for now it can be only on a local directory or by ssh with rsync.
But you can add easily a new manner of sending your backups by adding a new script. (And let the community enjoy your work by adding this script to the repository of archivist.)

To see how you can used Archivist for your usage, please have a look to the readme about the configuration of Archivist.


Salut les amis

Encore un fois, me revoilà avec un nouvelle application !
Parce que ma précédente application est un peu incomplète si vous n’avez rien pour gérer vos sauvegardes, maintenant nous avons Archivist \o/

Cette nouvelle application (comme les 2 précédentes, plus un script CLI qu’une véritable app) va vous donner un moyen simple de faire des sauvegardes régulières et les envoyer là où vous voulez, sans aucune action de votre part.

Comment ça marche ?
C’est assez simple, il y a un fichier de configuration dans lequel vous pouvez dire quelle partie de YunoHost vous voulez sauvegarder, et où vous pouvez aussi ajouter n’importe quel répertoires ou fichiers vous voulez égaement sauvegarder.

Après ça, à la fin de ce même fichier de config, vous pouvez ajoutez autant de destinataire que vous le voulez.
Vos sauvegardes peuvent être chiffrées avant d’être envoyée, et vous pouvez même exclure certaines parties pour ce destinataire, si vous le souhaitez.
Régulièrement (tout les jours à tout les mois, comme vous voulez), votre sauvegarde sera mise à jour, puis envoyée à tous vos destinataires.

Vous pouvez choisir comment vos sauvegardes sont envoyés, pour le moment ça ne peut être que dans un répertoire local ou par ssh avec rsync.
Mais vous pouvez ajouter facilement un nouveau moyen d’envoyer vos backup en ajoutant un nouveau script. (Et permettez à la communauté de profiter de votre travail en ajoutant ce script au dépôt de archivist.)

Pour savoir comment utiliser Archivist pour votre usage, jetez un coup d’oeil au readme sur la configuration d’Archivist.

4 Likes

Will it delete the old backup after the next backup is created ? And can it be use as multi-instance so that each app can be backed up at different time of the day because if the data to be backed up is huge then it can take hours to backup for one large file.

Yes, tar remove a backup before making a new one.
But, only if there any modification to do in this backup.
Somehow, it could be a nice idea to work to have a copy of each replaced backup. And keep a defined number of previous backups.

It is yes.

Salut et merci beaucoup pour cette app !

Je suis relativement novice sous linux, je voudrais me servir de ton app sur mon serveur ynh mais j’ai quelques questions concernant la configuration d’Archivist.

  • Une fois l’app installée, comment modifier la fréquence de sauvegardes et l’heure à laquelle elles seront réalisés (en pleine nuit par exemple).

  • J’ai monté un disque dur externe à l’emplacement /media/backup puis j’ai modifié la ligne “# Backup directory” du fichier Backup_list.conf de cette façon “backup_dir=/media/backup”. Peux-tu me confirmer que de cette façon les sauvegardes seront bien stockées sur mon disque dur externe.

Merci pour ton boulot et pour ton aide !

Hi,

thanks for this nice app !

I tried to run it but met an issue:

> Encrypt backups
fuse: device not found, try 'modprobe fuse' first
fuse failed. Common problems:
- fuse kernel module not installed (modprobe fuse)
- invalid options -- see usage message

I am not sure whether it lacks a dependency or a configuration step ?

I also had this warning:

Warning: ls: cannot access /etc/cron.d/yunohost*: No such file or directory

Hi nicofrand

can you say me what kind of server you use ? A VPS ? A virtual machine ?
Maybe your server doesn’t support fuse.

Did you try to load the module fuse, as the message said ?

It’s an old kimsufi, not a VPS nor a VM.

Running modprob fuse outputs

modprobe: ERROR: could not insert 'fuse': Unknown symbol in module, or unknown parameter (see dmesg)

And dmesg:

[62695517.917773] fuse: Unknown symbol setattr_prepare (err 0)
[62695517.920043] fuse: Unknown symbol sysfs_create_mount_point (err 0)
[62695517.922690] fuse: Unknown symbol sysfs_remove_mount_point (err 0)
[62695588.180485] fuse: Unknown symbol setattr_prepare (err 0)
[62695588.186081] fuse: Unknown symbol sysfs_create_mount_point (err 0)
[62695588.188379] fuse: Unknown symbol sysfs_remove_mount_point (err 0)

According to the great friend stackoverflow, it appears your kernel can be too old.

If you can’t upgrade your kernel, you can desactive the encryption which use fuse.

root@nicofrand:/home/admin# uname -r
3.16.0-4-amd64

I guess upgrading from jessie to the latest debian might do it but I am not sure YNH works on the latest debian ?

YunoHost is not compatible yet with Debian “Stretch” 9.0: Debian Stretch | YunoHost is NOT YET compatible | YunoHost N’EST PAS ENCORE compatible

Then how are people using Archivist ? The kernel version is limited by debian version.

# apt-cache search linux-image                                                                                                                                               
linux-headers-3.16.0-4-amd64 - Header files for Linux 3.16.0-4-amd64                                                                                                                                   
linux-image-3.16.0-4-amd64 - Linux 3.16 for 64-bit PCs                                                                                                                                                 
linux-image-3.16.0-4-amd64-dbg - Debugging symbols for Linux 3.16.0-4-amd64                                                                                                                            
linux-headers-4.9.0-0.bpo.3-amd64 - Header files for Linux 4.9.0-0.bpo.3-amd64                                                                                                                         
linux-headers-4.9.0-0.bpo.3-rt-amd64 - Header files for Linux 4.9.0-0.bpo.3-rt-amd64                                                                                                                   
linux-headers-4.9.0-0.bpo.4-amd64 - Header files for Linux 4.9.0-0.bpo.4-amd64                                                                                                                         
linux-headers-4.9.0-0.bpo.4-rt-amd64 - Header files for Linux 4.9.0-0.bpo.4-rt-amd64
linux-image-4.9.0-0.bpo.3-amd64 - Linux 4.9 for 64-bit PCs
linux-image-4.9.0-0.bpo.3-amd64-dbg - Debug symbols for linux-image-4.9.0-0.bpo.3-amd64
linux-image-4.9.0-0.bpo.3-rt-amd64 - Linux 4.9 for 64-bit PCs, PREEMPT_RT
linux-image-4.9.0-0.bpo.3-rt-amd64-dbg - Debug symbols for linux-image-4.9.0-0.bpo.3-rt-amd64
linux-image-4.9.0-0.bpo.4-amd64 - Linux 4.9 for 64-bit PCs
linux-image-4.9.0-0.bpo.4-amd64-dbg - Debug symbols for linux-image-4.9.0-0.bpo.4-amd64
linux-image-4.9.0-0.bpo.4-rt-amd64 - Linux 4.9 for 64-bit PCs, PREEMPT_RT
linux-image-4.9.0-0.bpo.4-rt-amd64-dbg - Debug symbols for linux-image-4.9.0-0.bpo.4-rt-amd64
linux-headers-4.9.0-2-grsec-amd64 - Header files for Linux 4.9.0-2-grsec-amd64
linux-image-4.9.0-2-grsec-amd64 - Linux 4.9 for 64-bit PCs, Grsecurity protection
linux-image-grsec-amd64 - Linux image meta-package, grsec featureset
linux-image-amd64 - Linux for 64-bit PCs (meta-package)
linux-image-amd64-dbg - Debugging symbols for Linux amd64 configuration (meta-package)
linux-image-rt-amd64 - Linux for 64-bit PCs (meta-package), PREEMPT_RT
linux-image-rt-amd64-dbg - Debugging symbols for Linux rt-amd64 configuration (meta-package)
linux-image-4.9.0-0.bpo.2-amd64 - Linux 4.9 for 64-bit PCs (signed)
linux-image-4.9.0-0.bpo.2-rt-amd64 - Linux 4.9 for 64-bit PCs, PREEMPT_RT (signed)
nvidia-kernel-3.16.0-4-amd64 - NVIDIA binary kernel module for Linux 3.16.0-4-amd64

Or, do I have to install jessie backports ?

There’s something else, I use the same kernel on my server and I have no problem with fuse.
Something in your server seems to block fuse.

Would you mind telling me the ouput of apt-cache policy fuse on your machine, please ? Did you install anything else (fuse2fs maybe ?) ?

fuse:
  Installé : 2.9.3-15+deb8u2
  Candidat : 2.9.3-15+deb8u2
 Table de version :
 *** 2.9.3-15+deb8u2 0
        500 http://ftp.debian.org/debian/ jessie/main i386 Packages
        500 http://security.debian.org/ jessie/updates/main i386 Packages
        100 /var/lib/dpkg/status

fuse2fs is not installed on my server. I’ve not installed anything else to make it working. And, more than anything else, I tried this app with a virtual machine with nothing else except YunoHost.

@Maniack_Crudelis
I am unable to send my backup through rsync ssh. What is the role of destination directory? Do we need to create the destination before hand at the other side?
Can you give an example of ssh_chroot_dir for archivist Eg. ssh_host:test.tld user , ssh_user=test, ssh_pwd=test123 ?

Hi kanhu

The destination directory should exist on the other side. This option is used by rsync to know where it’s supposed to copy the files on the other side.
You can see that as the destination of a cp, except here, this destination is relative to your ssh environment.

If you want to use ssh_chroot_dir with archivist.
Let’s say your ssh_chroot_dir is like that

ssh_user  = test
password  = test123
pub_key   = Not used...
size      = 10G

Your configuration of archivist should be like that:

type                  =rsync_ssh
destination directory =/
ssh_host              =test.tld
ssh_user              =test
ssh_port              =22
ssh_pwd               =test123

I get an error Backup YunoHost core and apps

Make a temporary backup for wordpress_backup

This backup is the same than the previous one
Clean old backup files
Encrypt backups
fuse: mountpoint is not empty
fuse: if you are sure this is safe, use the ‘nonempty’ mount option
fuse failed. Common problems:

  • fuse kernel module not installed (modprobe fuse)
  • invalid options – see usage message

Edited: After few minutes this error was gone automatically.

I still not able to see any files in ssh_chroot_dir

type =rsync_ssh
destination directory =/
ssh_host =test.tld
ssh_user =test
ssh_port =22
ssh_pwd =test123

Thu Dec 7 00:50:25 IST 2017

Build list of files to backup
Compress backups

Backup YunoHost core and apps

Make a temporary backup for wordpress_backup

This backup is the same than the previous one
Clean old backup files
Encrypt backups

The error “fuse: mountpoint is not empty” is, as it’s say, because the mountpoint is not empty.
That happens because you probably run the script too many times, and as you seen after, the error automatically gone.
That’s because, the mount point is automatically unmount after a fewer minutes unused.

But, even with that error, you should have your backups created, and duplicate.

Do you see anything about rsync after the lines ?
You should see something about what rsync doing.

I don’t see anything after these lines.
The backup should be there in the data folder of the ssh_chroot_dir?