[Archivist] Backup your server with rsync

recipient name should be unique?

Hum no, you can have as many recipient name as you want.
Can you show me your config file ? (Without any sensitive data)

Sure

## OPTIONS

# Backup directory.
backup_dir=/home/yunohost.app/archivist/backup

# Encrypted backup directory.
# Usually just next to the backup directory
enc_backup_dir=/home/yunohost.app/archivist/encrypted_backup

# Encryption
# true/false
encrypt=true
# Password for encryption
# This file should be set at chmod 400
cryptpass=/opt/yunohost/archivist/passkey

# --------------------------------------------------------------------------------------
## YUNOHOST BACKUPS

# YunoHost backups
# Specific backup made by 'yunohost backup' command
# ynh_core_backup=true/false: Make a backup of the core of YunoHost, without any apps.
ynh_core_backup=false

# ynh_app_backup=APP: Make a backup of an app.
# You can add as many lines as you want to save multiple apps.
# List all apps with sudo yunohost app list --installed | grep "id: "
# ynh_app_backup=wordpress
ynh_app_backup=wordpress

# --------------------------------------------------------------------------------------
## FILES BACKUPS

# Max size for each backup in Mb.
# Be carreful, it's only a soft limit, that means the script will try to limit each backup.
# Because, it will never made more than one backup for a single directory (Even if it's a real big directory without subdirectories). And the same if there's files next to subdirectories, only one backup will be made for all this files.
max_size=500

# Backup of files and directories
# Add as many lines as you want for each directory or file you want to be backuped.
# Exclude a directory, a file, or multiples files with a regex with the exclude_backup instruction.
# file_to_backup="/directory/to backup"
# file_to_backup=/file/to backup
# exclude_backup="/directory/to exclude"
# exclude_backup=/file/to exclude
# exclude_backup="/exclude/all/tarball/in_this_directory/.tar.gz$"
file_to_backup=
exclude_backup=

# --------------------------------------------------------------------------------------
## BACKUPS RECIPIENTS

# All options following a recipient name will be enabled for this recipient only.
# > recipient name=Name of this recipient.
# type=Choose between one of the scripts in the "senders" directory.
# destination directory=Directory where to put the backup in the other side.
# encrypt=Override main option for encryption. (true/false) (Optionnal option)
# specific option for type=See the chosen script to know what's the options.
# exclude backup=Exclude a backup file or a directory from the backup. This option can be duplicated. (Optionnal option)
# include backup=Choose the only file or directory which be send. This option can be duplicated. (Optionnal option)

# > recipient name=local example
# type=local
# destination directory=/my/local/backup
# encrypt=false
# exclude backup=/exclude/dir
# include backup=

 recipient name=rsync ssh ssh_chroot_dir server
 type=rsync_ssh
 destination directory=/
 encrypt=true
 ssh_host=test.tld
 ssh_user=test
# ssh_port=22
# ssh_key=/home/user/.ssh/id_rsa
 ssh_pwd=test123
# ssh_options=
# exclude backup=/exclude/dir
# include backup=

Oh ok I get it !

You just missed a "> " before “recipient name”.
These 2 characters are needed to split the config file by recipient.

So recipient name=rsync ssh ssh_chroot_dir server should be > recipient name=rsync ssh ssh_chroot_dir server
And remove the spaces at the beginning of each line, that could induce some errors too.

There should be information about this in the comment.

I get this error now.

-> Build the list of files for the recipient rsync ssh test.tld
./archivist.sh: line 541: /opt/yunohost/archivist/senders/.sender.sh: No such file or directory

Did you removed the space at the beginning of each line ?
Here it seems it can find the type line.

Ok I removed the spaces and now there is an another error.

-> Build the list of files for the recipient rsync ssh anmol.ga
> Copy backups files in /.
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.1]

Oh… Here the only thing I can propose is to add a verbose option to rsync to know more about what’s going on.

Here is the debug log:

-> Build the list of files for the recipient rsync ssh test.tld
> Copy backups files in /data/backup.
OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: /etc/ssh/ssh_config line 57: Applying options for *
debug1: Connecting to test.tld [XX.XXX.XXX.XX] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Debian-5+deb8u3
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u3 pat OpenSSH* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.1]

Can you move (or add) the verbose option in the rsync part. Here it’s only ssh who takes the verbose option.

Of course we’re talking about adding this in the file /opt/yunohost/archivist/senders/rsync_ssh.sender.sh ? Aren’t we ?

But rsync already have verbose in the rsync_ssh.sender.sh that is why the error is shown. No?

sudo rsync --archive --verbose --human-readable --stats --itemize-changes \
	--delete-excluded --prune-empty-dirs --exclude-from="$script_dir/../exclude_list" \
	"$backup_source/" --rsh="$ssh_command $ssh_options" $ssh_user@$ssh_host:"$dest_directory"

Oh shit, we can’t say it’s really a “verbose” debug information…

Oh wait, I just saw “connection unexpectedly closed”, and that makes me think about a common ssh error case.

Could you try to connect with the same ssh credential, just to be sure your ssh connection is ok.

The problem was the host key:

The authenticity of host 'test.tld (XX.XX.XX.XX)' can't be established.
ECDSA key fingerprint is XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'test.tld,XX.XX.XX.XX' (ECDSA) to the list of known hosts.

After which the connection could be established. But I still don’t see any files in the ssh_chroot_dir server.

Build list of files to backup
Compress backups
Backup YunoHost core and apps

Make a temporary backup for my-mind_backup

This backup is the same than the previous one
Clean old backup files
Encrypt backups
fuse: mountpoint is not empty
fuse: if you are sure this is safe, use the ‘nonempty’ mount option
fuse failed. Common problems:

 - fuse kernel module not installed (modprobe fuse)
 - invalid options -- see usage message

-> Build the list of files for the recipient rsync ssh test.tld
> Copy backups files in /data.
OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: /etc/ssh/ssh_config line 57: Applying options for *
debug1: Connecting to test.tld [XX.XX.XX.XX] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Debian-5+deb8u3
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u3 pat OpenSSH* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX
debug1: Host 'XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX' is known and matches the ECDSA host key.
debug1: Found key in /XXXX/.ssh/known_hosts:3
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /XXXX/.ssh/id_rsa
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /XXXX/.ssh/id_dsa
debug1: Trying private key: /XXX/.ssh/id_ecdsa
debug1: Trying private key: /XXX/.ssh/id_ed25519
debug1: Next authentication method: password
debug1: Authentication succeeded (password).
Authenticated to test.tld ([XX.XX.XX.XX]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LC_PAPER = en_IN
debug1: Sending env LC_ADDRESS = en_IN
debug1: Sending env LC_MONETARY = en_IN
debug1: Sending env LC_NUMERIC = en_IN
debug1: Sending env LC_TELEPHONE = en_IN
debug1: Sending env LC_IDENTIFICATION = en_IN
debug1: Sending env LANG = en_US
debug1: Sending env LC_MEASUREMENT = en_IN
debug1: Sending env LC_TIME = en_IN.UTF-8
debug1: Sending env LC_NAME = en_IN
debug1: Sending command: rsync --server -vlmogDtpre.iLsfx --log-format=%i --delete-excluded --stats . /data
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
building file list ... done
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 3612, received 1828 bytes, in 0.3 seconds
Bytes per second: sent 13223.8, received 6692.4
debug1: Exit status 0

Number of files: 2 (dir: 2)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.002 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 82
Total bytes received: 19

sent 82 bytes  received 19 bytes  40.40 bytes/sec
total size is 0  speedup is 0.00

Yes, it’s a common problem with ssh, which I forgot. My bad…

That’s strange, rsync says there 2 files, but it doesn’t transfers any of them.
There’s still nothing in include or exclude in your config file ?

Anyway, I think you can remove the verbose option which only make noise now.

Both are empty but not commented. I have tried to backup in different destination directories but none of them have any backup files.

Maybe have a look to the list of file it’s suppose to backup, here /opt/yunohost/archivist/files_to_backup.list

If your “destination directory” is still at “/”, the files should be in the directory “/home/yunohost.app/ssh_chroot_directories/test/data/”

The home/yunohost.app/archivist/encrypted_backup is empty. /home/yunohost.app/archivist/backup/ynh_backup has the backup files.
/opt/yunohost/archivist/files_to_backup.list only have /some_random_code in it.

Now I see this error:

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1183) [sender=3.1.1]

Ok, all of this is ok.

/home/yunohost.app/archivist/encrypted_backup is empty because it’s filled up only when the encrypted backups are mount.
/home/yunohost.app/archivist/backup/ynh_backup has your real backups
And /opt/yunohost/archivist/files_to_backup.list contains the names of encrypted backups, that’s why you see random codes.

And what are the “previous errors” ? If there are.

Sorry I did not saw the errors above. Here are the errors.

    bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
    building file list ... done
    rsync: failed to set times on "/.": Operation not permitted (1)
    rsync: delete_file: unlink(lib64/ld-linux-x86-64.so.2) failed: Permission denied (13)
    cannot delete non-empty directory: lib64
    rsync: delete_file: unlink(lib/libtinfo.so.5) failed: Permission denied (13)
    rsync: delete_file: unlink(lib/libselinux.so.1) failed: Permission denied (13)
    rsync: delete_file: unlink(lib/libpthread.so.0) failed: Permission denied (13)
    rsync: delete_file: unlink(lib/libpopt.so.0) failed: Permission denied (13)
    rsync: delete_file: unlink(lib/libpcre.so.3) failed: Permission denied (13)
    rsync: delete_file: unlink(lib/libncurses.so.5) failed: Permission denied (13)
    rsync: delete_file: unlink(lib/libedit.so.2) failed: Permission denied (13)
    rsync: delete_file: unlink(lib/libdl.so.2) failed: Permission denied (13)
    rsync: delete_file: unlink(lib/libc.so.6) failed: Permission denied (13)
    rsync: delete_file: unlink(lib/libbsd.so.0) failed: Permission denied (13)
    rsync: delete_file: unlink(lib/libattr.so.1) failed: Permission denied (13)
    rsync: delete_file: unlink(lib/libacl.so.1) failed: Permission denied (13)
    cannot delete non-empty directory: lib
rsync: delete_file: rmdir(data) failed: Permission denied (13)
rsync: delete_file: unlink(bin/sftp-server) failed: Permission denied (13)
rsync: delete_file: unlink(bin/sftp) failed: Permission denied (13)
rsync: delete_file: unlink(bin/rsync) failed: Permission denied (13)
rsync: delete_file: unlink(bin/rmdir) failed: Permission denied (13)
rsync: delete_file: unlink(bin/rm) failed: Permission denied (13)
rsync: delete_file: unlink(bin/mv) failed: Permission denied (13)
rsync: delete_file: unlink(bin/mkdir) failed: Permission denied (13)
rsync: delete_file: unlink(bin/ls) failed: Permission denied (13)
rsync: delete_file: unlink(bin/du) failed: Permission denied (13)
rsync: delete_file: unlink(bin/cp) failed: Permission denied (13)
rsync: delete_file: unlink(bin/cat) failed: Permission denied (13)
rsync: delete_file: unlink(bin/bash) failed: Permission denied (13)
cannot delete non-empty directory: bin
.d..t...... ./

Ok…

Sorry that’s my fault, I read another config and copy it without checking.
So, that’s means your “destination directory” shall be /data
Because, here rsync try to work in your main chroot directory. And as you can see, it hasn’t the authorisation to do so.