[Borg & Borg Server] Deduplicated, encrypted and remote backups

No, just upgrade the app

1 Like

Dear all,

I currently and successfully backup my YNH server-1 on an other YNH server-2 with the greats Borg & Borg Backup.

To bring more safety, I’d like to have a second backup server-3 on a VPS somewhere.

I thought I just would have to install a second time Borg app on my YNH-server-1, add server-3 as the backup destination, and install Borg_server on YNH_server-3.

But when I install for the second time the Borg app on my YNH-server-1, no question is asked about the destination server domain.

Am I doing anything wrong ?

Second question : Should I better backup the Borg_server app of YNH-server-2, with YNH-server-3 as destination (a backup of the backup) ?

Thanks for your kind help
cheers

Official borgbackup doc (the upstream one) explain it’s a bad idea.

COuld you explain what’s happening exactly when you install the second time ? Could you share installation logs ?

Hi !

Here are the logs :

root@box:~# yunohost app install borg
In which borg repository location do you want to backup your files ?: charly-sav-2            
Provide a strong passphrase to encrypt your backups. No blank space: 
Should Borg backup your YunoHost configuration? [yes | no] (default: yes): yes
Should Borg backup emails and user home directory? [yes | no] (default: yes): yes
Which apps should Borg backup ? (default: all): all
With which regular time schedule should the backups be performed? (see systemd OnCalendar format) (default: Daily): Daily
Info: Installing borg...
Info: [++++................] > Installing dependencies...
Info: Installing/compiling borg, this may take some time...
Info: [####++++............] > Configuring system user...
Info: [########++++........] > Configuring a systemd service...
Info: [############++++....] > Integrating service in YunoHost...
Info: [####################] > Installation of borg__2 completed

Edit 1 : Look, the “server” line is juts empty in /etc/yunohost/apps/borg__2/settings.yml :

root@box:~# cat /etc/yunohost/apps/borg__2/settings.yml
apps: all
apt_dependencies: python3-pip, python3-dev, libacl1-dev, libssl-dev, liblz4-dev, python3-jinja2,
  python3-setuptools, python3-venv, virtualenv, libfuse-dev, pkg-config
checksum__etc_sudoers.d_borg__2: 3ed14aa63d1e11abfd4d3f0b1799531c
checksum__etc_systemd_system_borg__2.service: 74407db81d632d366a91efbdf45a75ff
checksum__etc_systemd_system_borg__2.timer: c53468085887c032f414d278bd3a48e2
checksum__etc_yunohost_hooks.d_backup_method_05-borg__2_app: 5f3be6098db882e64ee947ea136d0ae6
checksum__usr_local_bin_backup-with-borg__2: c66a15d474ca50c09f67a9853a686d57
conf: '1'
current_revision: 658e255ac5eae0dfeae857b97577465cc3b67530
data: '1'
id: borg__2
install_time: 1625209031
on_calendar: Daily
passphrase: verylongpassphrasedkdkdkdkd
repository: sav-2
server: ''

Should I change this file manually to write the domain of my second backup server ?

Edit 2 : I’m not sure if it’s normal but I also didn’t receive the email with all the information that I should paste during the Borg-server installation.

Thanks !
Have a nice day !

The issue is in command line, help description are not displayed. So you don’t see my advice to answer to the question:

In which borg repository location do you want to backup your files ?: charly-sav-2

Here is the help text:

Specify a local repository like /mount/my_external_harddrive/backups or a remote repository using this format: ssh://USER@DOMAIN.TLD:PORT/~/backup . If you plan to use borgserver_ynh app : ‘USER’ is not meant to be an existing user on the guest server, instead, it will be created on the host server during the installation of the Borg Server App. With borgserver_ynh apps you can’t specify another repo path than ~/backup.

I suggest you to uninstall/reinstall.

OK, so I’ve done what is explained in the help section :

root@box:~# yunohost app install borg
In which borg repository location do you want to backup your files ?: ssh://saveruser@my.domain:22/~/backup
Provide a strong passphrase to encrypt your backups. No blank space: 
Should Borg backup your YunoHost configuration? [yes | no] (default: yes): 
Should Borg backup emails and user home directory? [yes | no] (default: yes): 
Which apps should Borg backup ? (default: all): 
With which regular time schedule should the backups be performed? (see systemd OnCalendar format) (default: Daily): Daily

But look, in the email “borg was installed”, there is no user specified (it’s empty) :

This is an automated message from your beloved YunoHost server. Specific information for the application borg. You should now install the "Borg Server" app on [saver.mydomain.fr]:22 and with the following credentials: User: Public key: ssh-ed2sdsqkd....sdlksmdl6A/H [root@box.mydomain.fr]

But now end with the backup faling.
Here is an extract from the failure email report :

Remote: Debian GNU/Linux 10
Remote: Permission denied, please try again.
Remote: Permission denied, please try again.
Remote: saveruser@saver.mydomain.fr: Permission denied (publickey,password).

It seems that something is wrong.

EDIT :
On the backup server, the access.log shows :

tail /var/log/auth.log -f
Jul  2 14:28:44 saver sshd[11368]: Failed password for saveruser from 10.8.0.11 port 55604 ssh2
Jul  2 14:28:44 saver sshd[11368]: Failed password for saveruser from 10.8.0.11 port 55604 ssh2
Jul  2 14:28:44 saver sshd[11368]: Connection closed by authenticating user saveruser 10.8.0.11 port 55604 [preauth]
Jul  2 14:28:46 saver sshd[11370]: rexec line 29: Deprecated option UsePrivilegeSeparation
Jul  2 14:28:46 saver sshd[11370]: Connection from 10.8.0.11 port 55606 on 10.8.0.8 port 22
Jul  2 14:28:46 saver sshd[11370]: Failed publickey for saveruser from 10.8.0.11 port 55606 ssh2: ED25519 SHA256:bTeOe7R......vTE9O4U
Jul  2 14:28:47 saver sshd[11370]: Failed none for saveruser from 10.8.0.11 port 55606 ssh2

Where 10.8.0.11 is the IP of the main Yunohost server.

Thanks very much in advance for your help !

Hav you install borgserver on the remote server ?

Yep!
It looks like it’s a public key issue.

I tried to reinstall everything on both sides but it didn’t work.

This keys contains a + symbol ?

Hi, sorry for the late answer, didn’t have access to my server this weekend.

The public key doesn’t contain a + symbol.

When I start the borg backup service, this is what shows the file /var/log/auth.log on the borg_server server :

Jul  5 07:57:40 saver sshd[12213]: Failed none for usersaver from 10.8.0.11 port 56154 ssh2
Jul  5 07:57:40 saver sshd[12213]: Failed password for usersaver from 10.8.0.11 port 56154 ssh2
Jul  5 07:57:40 saver sshd[12213]: Failed password for usersaver from 10.8.0.11 port 56154 ssh2
Jul  5 07:57:40 saver sshd[12213]: Connection closed by authenticating user usersaver 10.8.0.11 port 56154 [preauth]
Jul  5 07:57:41 saver sshd[12218]: rexec line 29: Deprecated option UsePrivilegeSeparation
Jul  5 07:57:41 saver sshd[12218]: Connection from 10.8.0.11 port 56156 on 10.8.0.8 port 22
Jul  5 07:57:42 saver sshd[12218]: Failed publickey for usersaver from 10.8.0.11 port 56156 ssh2: ED25519 SHA256:MCFlyA...............AJgf9T/VCE/6Y

That’s why I think it must be a public / private key issue.

Thanks in advance for your help

Hi,
So I finally installed on a different server a fresh Debian 10 (without Yunohost) as an other backup server, installed the regular borg package (apt install borgbackup), imported the public key (that has been created by borg-ynh on the main Yuonhost server), gave the appropriate rights on /home/user set the right options in the file /home/user/.ssh/authorized_keys :
command="borg serve --storage-quota 400G --restrict-to-repository /home/user/backup",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-user-rc ssh-ed25519 mypublickeyyyyyyyyyyyyyyyyyyyyyy root@mydomain.fr

And it works.

Then I tried to install a new fresh Yunohost server, install the borgserver-ynh app, followed the recommendations, but it didn’t work.

I’m kind of afraid that borgserver-ynh doesn’t actually work if it’s installed now.

Do you think this issue could be related to the newer version of Yunohost with this different ssh access policy ?

If there is anything I can do to help, let me know.

Have a nice day

The app has been upgraded for 4.2 when the 4.2 has been released: https://github.com/YunoHost-Apps/borgserver_ynh/blob/master/scripts/install#L57 (you can see in this code the user is added to ssh.app group).

It might be an error due to some specific caracters in public keys.

You can check it by comparing settings from borg_ynh

cat /etc/yunohost/apps/borg/settings.yml

And the authorized_keys file

cat /home/user/.ssh/authorized_keys

Hi there,

Same trouble, I tried reinstalling twice and then popped up a fresh backup server, but no luck. Here is the log of the first failed run. Both ynh servers are up-to-date, and both public keys are OK. The private key in /root/.ssh indeed contains a “+” symbol, but not the public one.

Thanks for your help !

Hi,

If you don’t have any other backup solution, I’d suggest you to install a fresh Debian on your backup server, install borg, create the user, import the public key (with the right options) in /home/user/.ssh/authorized_keys and it should work like this.

It’s not perfect I know, but until this issue is fixed it should do the trick.

See ya

The private key should not be a problem.

Here is a part of your log

2021-07-12 00:00:23,751: WARNING - Remote: Warning: Permanently added '[backup.maindomain.tld]:2222,[109.15.122.87]:2222' (ECDSA) to the list of known hosts.
2021-07-12 00:00:24,706: WARNING - Remote: Raspbian GNU/Linux 10
2021-07-12 00:00:24,811: WARNING - Remote: Permission denied, please try again.
2021-07-12 00:00:24,862: WARNING - Remote: Permission denied, please try again.
2021-07-12 00:00:24,913: WARNING - Remote: yunoserv@backup.maindomain.tld: Permission denied (publickey,password).

SO you need to find why the connexion can’t be made.

To do it, you can try on your server you want to backup:

ssh -i /root/.ssh/id_borg_ed25519 -p2222 yunoserv@backup.maindomain.tld -vvvv

You can also check the log of /var/log/auth.log at the same time (on the backup.maindomain.tld)

After doing some research about this issue. I made a fix:

New release for borgserver_ynh:

  • 13/07/2021 - 1.1.16~ynh6
    • [fix] Unable to connect with ssh
1 Like

\o/ !! will test asap.
and thank you @charly for the suggestion, I have other backups but was looking into this as a simpler way :smile_cat:

And nope :confused: The borg repo is created, the ssh connexion is OK, but backup still fails : https://paste.yunohost.org/raw/rozekutivu

The borg list also fails :

BORG_RSH="ssh -p2222 -i /root/.ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes " borg list yunoserv@backup.domain.tld:~/backup
Remote: Raspbian GNU/Linux 10
Remote: Killed stale lock backup.domain.tld@202481593804101.22719-0.
Remote: Removed stale exclusive roster lock for host backup.domain.tld@202481593804101 pid 22719 thread 0.
Remote: Fatal Python error: Bus error
Remote: 
Remote: Current thread 0xf7ab28e0 (most recent call first):
Remote:   File "/opt/borg-env/lib/python3.7/site-packages/borg/crypto/file_integrity.py", line 75 in write
Remote:   File "/opt/borg-env/lib/python3.7/site-packages/borg/crypto/file_integrity.py", line 29 in write
Remote:   File "/opt/borg-env/lib/python3.7/site-packages/borg/algorithms/msgpack/__init__.py", line 38 in pack
Remote:   File "/opt/borg-env/lib/python3.7/site-packages/borg/repository.py", line 613 in write_index
Remote:   File "/opt/borg-env/lib/python3.7/site-packages/borg/repository.py", line 860 in replay_segments
Remote:   File "/opt/borg-env/lib/python3.7/site-packages/borg/repository.py", line 398 in check_transaction
Connection closed by remote host

On the borgserver I see this in auth.log :

Jul 15 09:50:25 backup sshd[22727]: Connection from **client_ip**  port 48616 on **server_ip** port 2222
Jul 15 09:50:25 backup sshd[22727]: Accepted key ED25519 SHA256:/J9I71p3N1PG+sk1V2/tgCNM9unBs993UeTBWKOXIK4 found at /home
/yunoserv/.ssh/authorized_keys:1
Jul 15 09:50:25 backup sshd[22727]: Postponed publickey for yunoserv from **client_ip**  port 48616 ssh2 [preauth]
Jul 15 09:50:25 backup sshd[22727]: Accepted key ED25519 SHA256:/J9I71p3N1PG+sk1V2/tgCNM9unBs993UeTBWKOXIK4 found at /home
/yunoserv/.ssh/authorized_keys:1
Jul 15 09:50:25 backup sshd[22727]: Accepted publickey for yunoserv from **client_ip**  port 48616 ssh2: ED25519 SHA256:/J9
I71p3N1PG+sk1V2/tgCNM9unBs993UeTBWKOXIK4
Jul 15 09:50:25 backup sshd[22727]: pam_unix(sshd:session): session opened for user yunoserv by (uid=0)
Jul 15 09:50:25 backup systemd-logind[480]: New session 322 of user yunoserv.
Jul 15 09:50:26 backup systemd: pam_unix(systemd-user:session): session opened for user yunoserv by (uid=0)
Jul 15 09:50:26 backup sshd[22727]: User child is on pid 22744
Jul 15 09:50:26 backup sshd[22744]: Starting session: forced-command (key-option) 'borg serve --storage-quota 50G --restri
ct-to-repository /home/yunoserv/backup' for yunoserv from **client_ip**  port 48616 id 0
Jul 15 09:50:29 backup sshd[22744]: Close session: user yunoserv from **client_ip** port 48616 id 0
Jul 15 09:50:29 backup sshd[22744]: Received disconnect from **client_ip** port 48616:11: disconnected by user
Jul 15 09:50:29 backup sshd[22744]: Disconnected from user yunoserv 89.234.186.87 port 48616
Jul 15 09:50:29 backup sshd[22727]: pam_unix(sshd:session): session closed for user yunoserv
Jul 15 09:50:29 backup systemd-logind[480]: Session 322 logged out. Waiting for processes to exit.
Jul 15 09:50:29 backup systemd-logind[480]: Removed session 322.
Jul 15 09:50:39 backup systemd: pam_unix(systemd-user:session): session closed for user yunoserv

Should I retry a clean install of both apps ?

Which kind of rpi have you ? Is it install with raspbian or with yunohost image ?

I have a RPi-3 with external HDD : I installed raspbian and then ran the ynh install script.

It seems to be a borg problem on the server side, as a local “borg list” also fails with python errors.