[Archivist] Backup your server with rsync

kanhu · December 6, 2017, 10:59pm

But the mystery is rsync do not copy it any directory. I tried /data, /data/backup, etc. But there no files in it.

Copy backups files in /data.
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
building file list ... done

Number of files: 2 (dir: 2)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 83
Total bytes received: 19

sent 83 bytes  received 19 bytes  204.00 bytes/sec
total size is 0  speedup is 0.00

Edit: Can this line bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) be cause of the problem ?

Maniack_Crudelis · December 6, 2017, 11:03pm

You mean here, you tried /data as destination directory, and it gives you that output ?
With nothing transfered…

That’s weird…

The error about setlocale is nothing really important.

Can you try to do this rsync by yourself ?

kanhu · December 6, 2017, 11:33pm

ok, rsync -azv /home/yunohost.app/archivist/backup/. -e ssh test@test.tld:backup/. worked. All the files in directory are now in ssh_chroot_dir server.

Do all these need to be empty?

file_to_backup=
exclude_backup=
# exclude backup=/exclude/dir
#include backup=

Maniack_Crudelis · December 6, 2017, 11:46pm

The first file_to_backup (and exclude_backup) is used only to backup directories and files.
Here you’re backuping only an application. So it’s ok.

The second ones are optionnal too.

So that’s ok.

Why did you do your rsync in “backup/” ?
If you connect with ssh, are you in / or /data ?
Where is this directory backup ?

kanhu · December 6, 2017, 11:52pm

The /data/backup directory gets created if backup is in destination directory=backup

By default I get into /data from ssh.

This directory is in /data/backup.

Maniack_Crudelis · December 6, 2017, 11:57pm

OK, so, if by default your in /data, and if you have a directory backup. Your destination directory can be “backup”.
That’s maybe why we have some problem before, by using “/”.
But I do not understand why it doesn’t work with /data…

kanhu · December 7, 2017, 12:03am

Its not working with anything /data, /backup,data,backup,etc. I have reinstalled the archivist but still no difference. I don’t think that installing ssh_chroot_dir again will make any difference too because I haven’t changed anything in it.

Maniack_Crudelis · December 7, 2017, 12:10am

So, I don’t know why…

I have to go now, I’m going to go back tomorrow about 23h30 or something like that.
We can investigate more tomorrow.

What’s really weird is I already used this 2 apps together without problems.

kanhu · December 7, 2017, 12:19am

No problem.I will further test it with local storage.

Most probably we are missing something.

nicofrand · December 13, 2017, 6:17pm

Sorry for the delay ! I’d like to thank you for your help. I actually rebooted my server and that fixed the issue !

Now that the backups work I have to find a place where to send them !

Maniack_Crudelis · December 13, 2017, 6:46pm

Maybe you can open a new topic to start a discussion about that.
When I made archivist, I thought it would be really interresting to allow YunoHost owners to share parts of their hard disk between them with ssh_chroot_dir.

And because, you can have a encrypted version of your backup sent somewhere, it would be safe and easy to share space to allow someone to put his backup in your server and do the same with his server.
I would say, the best is to have at least 2 or 3 people with your backups (encrypted of course).

So it would be really nice to open a topic and start to discuss who can share space with others.

hercut · December 20, 2019, 9:55am

Bonjour,

Je suis entrain d’essayé cette application.

Mais certaine chose me tracasse.
Comment sont géré les sauvegarde en cas de manque de place sur le serveur ?

Je souhaiterai un petit coup de main pour que mes sauvegarde se fasse sur ma freebox, possible ?

A quand une interphase graphique ???

Merci.

Maniack_Crudelis · December 23, 2019, 1:38pm

Hello @hercut

If you backup YunoHost applications or the core, the backup process is supposed to check if there’s enough space before doing it.
For backup of directories, nothing check the available space before. It could be done though.

On your freebox you should have an ftp access, as well as windows sharing.
You can use one or the other to send your backup on the hard drive attached to your freebox.
A windows shared directory should be mounted on your server with samba. Then, you can use the local sender.
To use the ftp access, you should add a new sender that use ftp to send your files on your freebox.
It depend mainly if your server is on the same network than you freebox.

A graphical user interface would be nice. But that implies someone would build it…

pcet · November 10, 2020, 8:09pm

Quelqu’un utilise toujours cette app avec Yunohost 4 ? Je cherche une solution pour sauvegarder automatiquement Yunohost sur un hébergement mutualisé donc par FTP.

Merci !

don · November 15, 2020, 10:04pm

@pcet j’utilise toujours Archivist avec mon yunohost en version 4 avec une copie vers une autre machine via un «sender» rsync dans la config d’archivist et je peut confirmer que cela marche parfaitement avec la derniere version de yunohost, si tu souhaite utiliser du ftp pour envoyer automatiquement tes backup tu risque de devoir ajouter un «sender» custom (la demarche pour ajouter un sender a la config de archivist est expliqué ici : https://github.com/maniackcrudelis/archivist/blob/master/senders/How%20to%20add%20a%20new%20sender.md) car les sender fournie par default n’utilise que Rsync

pcet · November 17, 2020, 8:41am

Merci @don !
Comme j’ai un accès ssh je pense avoir un accès sftp à mon serveur mutualisé (chez Informaniak) (il faut que je vérifie). J’ai configuré le sender rsync_ssh mais lors du déroulement de la tâche Cron, rien n’est envoyé vers mon serveur mutualisé.

Voici la config du sender dans mon Backup_list.conf

 > recipient name=rsync_ssh
 type=rsync_ssh
# destination directory=/yunohost
 encrypt=false
 ssh_host=mon-serveur-sftp/chemindedestination
 ssh_user=mon_user_id
 ssh_port=22
# ssh_key=/home/user/.ssh/id_rsa
 ssh_pwd=*********
# ssh_options=
# pre_backup=
# post_backup=
# exclude backup=/exclude/dir
# include backup=

La fin de mon archivist.log se fait pas mention du rsync

> Clean old backup files
Tue 17 Nov 2020 05:20:59 AM UTC

Et si je lance manuellement le rsync_ssh.sender.sh j’obtiens un message d’erreur

grep: /opt/yunohost/archivist/senders/../recipient_config.conf: No such file or directory
grep: /opt/yunohost/archivist/senders/../recipient_config.conf: No such file or directory
grep: /opt/yunohost/archivist/senders/../recipient_config.conf: No such file or directory
grep: /opt/yunohost/archivist/senders/../recipient_config.conf: No such file or directory
grep: /opt/yunohost/archivist/senders/../recipient_config.conf: No such file or directory
grep: /opt/yunohost/archivist/senders/../recipient_config.conf: No such file or directory
grep: /opt/yunohost/archivist/senders/../recipient_config.conf: No such file or directory
grep: /opt/yunohost/archivist/senders/../recipient_config.conf: No such file or directory
grep: /opt/yunohost/archivist/senders/../recipient_config.conf: No such file or directory
find: ‘’: No such file or directory
sort: cannot read: /opt/yunohost/archivist/senders/../files_to_backup.list: No such file or directory
> Copy backups files in .
Conflicting password source
sshpass: invalid option -- 'l'
protocol version mismatch -- is your shell clean?
(see the rsync man page for an explanation)
rsync error: protocol incompatibility (code 2) at compat.c(178) [sender=3.1.3]
grep: /opt/yunohost/archivist/senders/../recipient_config.conf: No such file or directory

J’ai essayé de comprendre mais j’avoue que je n’y arrive pas ! Si quelqu’un à une idée ?

don · November 17, 2020, 11:44am

Il semble que ta conf dans Backup_List.conf présente des erreur (par exemple le chemin de destination va dans destination dorectory et non a la suite de ton hostname dans host) ce qui fait que la config de ton sender n’ai pas correct et donc archivist ne créer pas de récipient conf correspondant pour fournir les argument au script rsync_ssh.sender.sh , sinon vérifie ausi si tu n’as pas laisser des espace en début de ligne sur la conf de ton sender dans backup_list.conf ( cela empecherai le parsing des parrametre et donc archivist ignorerai ton sender totalement )

pcet · November 17, 2020, 1:42pm

Merci beaucoup pour ton aide @don !
En effet, il y avait des espaces en trop et le fichier recipient_config.conf s’est bien créé. Néanmoins la copie ne se fait toujours pas. J’ai donc lancé manuellement la commande rsync et j’obtiens ces erreurs (par contre je l’ai lancée dans l’autre sens depuis la console ssh de mon serveur mutualisé et la copie s’est effectuée sans problème !)

rsync serveur yunohost -> serveur mutualisé

/etc/ssh/ssh_config: line 7: Bad configuration option: listenaddress
/etc/ssh/ssh_config: line 8: Bad configuration option: listenaddress
/etc/ssh/ssh_config: line 11: Bad configuration option: hostkey
/etc/ssh/ssh_config: line 12: Bad configuration option: hostkey
/etc/ssh/ssh_config: line 13: Bad configuration option: hostkey
/etc/ssh/ssh_config: line 49: Bad configuration option: logingracetime
/etc/ssh/ssh_config: line 50: Bad configuration option: permitrootlogin
/etc/ssh/ssh_config: line 51: Bad configuration option: strictmodes
/etc/ssh/ssh_config: line 53: Bad configuration option: permitemptypasswords
/etc/ssh/ssh_config: line 55: Bad configuration option: usepam
/etc/ssh/ssh_config: line 56: Bad configuration option: uselogin
/etc/ssh/ssh_config: line 63: Bad configuration option: banner
/etc/ssh/ssh_config: line 64: Bad configuration option: printmotd
/etc/ssh/ssh_config: line 65: Bad configuration option: printlastlog
/etc/ssh/ssh_config: line 66: Bad configuration option: clientaliveinterval
/etc/ssh/ssh_config: line 67: Bad configuration option: acceptenv
/etc/ssh/ssh_config: line 70: Bad configuration option: subsystem
Unsupported Match attribute group
/etc/ssh/ssh_config line 71: Bad Match condition
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]

Commande rsync depuis serveur mutualisé : ça fonctionne mais je dois autoriser la connexion manuellement avec ce message :

The authenticity of host 'mon.serveur.yunohost.fr (ipv6 de mon serveur yunohost)' can't be established.
ECDSA key fingerprint is xxx:xxx:xxx:xx:xx:x:x:x:x:x:x.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mon.serveur.yunohost.fr,ipv6 de mon serveur yunohost' (ECDSA) to the list of known hosts.

Je pense donc qu’il doit falloir configurer ssh sur mon serveur yunohost pour que ça fonctionne

don · November 17, 2020, 2:41pm

Est ce que la connection ssh vers ton serveur mutualisé est fonctionnel parce que a première vu je dirait que c’est la ou la situation bloque, essaye de faire une connection ssh de ton yunohost vers ton mutualisé avec les info de connection que tu a utilisé dans ton sender pour vérifier si tu a pas un soucis de mot de Passe non correct ou que ton utilisateur du côté de ton mutualisé est bien autorisé a ce connecter en ssh

pcet · November 21, 2020, 12:50pm

Bon, ça fonctionne ! Pour y arriver j’ai fait deux manips :

J’ai supprimé le contenu de ssh_config qui avait les memes règles que sshd_config ce qui apparemment n’est pas normal/utile
J’ai effacé un espace en trop après l’adresse du ˋhost`

Et voilà ! Mon serveur mutualisé reçoit bien les sauvegardes !