Backup: borg-user on borg-server is not allowed to use borg

Hi all,

I have been struggling with borg-backup.

Installation was a breeze, but running proved difficult. After some other problems (ssh keys are not found, it seems ssh -i is not used which could be because an error with the hook-script is mentioned; worked around by renaming id_borg_ed… to id_ed…), borg returns an error because
/usr/local/bin/borg: Permission denied

Server aa.tld is backed up on server bb.tld. User root@aa.tld logs in as aatld@bb.tld.
I run
service borg start on aa.tld, which returns an error.
Journalctl -xe shows that gathering the backup works, but actually archiving returns an error:

Info: Now creating a backup archive from the files collected…
Error: Script execution failed: /etc/yunohost/hooks.d/backup_method/05-borg__2_app
Error: Custom backup method failure on 'backup' step

When I do that (as root, log in aatld@bb.tld), and try to run borg, permission is denied.
When I log into bb.tld, sudo su -, then su aatld and try to run borg, I also get permission denied.

Did I do something wrong?

ping @ljf

Hi Alex,

Thanks for jumping on my problem immediately :slight_smile:

@ljf, some more info:

Installing does give some warnings, no matter how I format the input.

Indicate the storage quota: 40G
Info: Installing application borgserver…
Warning: [WARN] Too many arguments ! "AAAAC3NzaC1lZDI1NTE5AAAAIMMhiUBJn" will be ignored.
Warning: [WARN] Too many arguments ! "/RqiRS23uMxzwExf0DOz/jq/Oulnx6Ac4b" will be ignored.
Warning: [WARN] Too many arguments ! "root@aa.tld" will be ignored.
Success! The SSOwat configuration has been generated
Success! Installation complete```

How is the resulting /home/user/.ssh/authorized_keys-line supposed to look like? I got one created by root while installing the borg-server end of the system, strictly fenced within a narrow set of permissions:
```# cat .ssh/authorized_keys.org
command="borg serve --storage-quota 50G --restrict-to-repository /home/onlineosbanl2/backup",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-user-rc ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHYQT0RgAzQNP6J5Rv7qBn50YuV734fODdHia1vubtVh root@online.osba.nl

Troubleshooting because I had no success running a backup:

  • Backup didn’t work out of the bo(rg)x
  • I tried whether keybased login worked, it didn’t
  • Tried ssh-copy-id, couldn’t because authorized_keys is owned by root on bb.tld
  • (Re)moved the authorized_keys file
  • Did a new ssh_copy_id
  • Still no luck with key based sign in
  • Debugging with ssh -vv showed that SSH tries to use id_ed… instead of id_borg_ed…, hence ssh -i
  • Renamed key from id_borg_ed25519 to id_ed25519
  • Success on login with the key
  • Backup runs until the actual archiving with borg

Looking back, I see that the fact that the hook gave an error might be the cause of the other points failing.

How to find out what is wrong with the hook?

Please let me know any debugging output that would help!

How did you check that ?

Your file /home/user/.ssh/authorized_keys is ok.

ARe you debugging in the hook OR in an other way ?

The hook should not give you an error.

Could you add set -x at the begginning of the hook, and run the service again. May be you will get more input in journalctl with set -x…

Hi, thanks for your trouble.

After following the instructions for borg (on aa.tld) and borgserver (on bb.tld), I tried running ‘service borg start’ on aa.tld.

Ok :slight_smile:

I am not sure I can read the hook, so I have some trouble debugging it. So, other ways: I read journalctl and systemctl for borg, and try to connect to bb.tld via SSH. As root on aa.tld, I ssh -i .ssh/id_borg_ed25519 -oStrictHostKeyChecking=yes aatldbak@bb.tld, and get a password prompt:


debug1: Server host key: ecdsa-sha2-nistp256 SHA256:T8ii4lLcmJA1rBJKhScqULA+hPE365ZFrwG1GH2Zu4k
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:9
debug3: load_hostkeys: loaded 1 keys from bb.tld
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:6
debug3: load_hostkeys: loaded 1 keys from 2001::1:dde::234d
debug1: Host 'bb.tld' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:9
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: .ssh/id_borg_ed25519 (0xaaab0de5d9e0), explicit
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 53
debug3: input_userauth_banner
Debian GNU/Linux 9
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering ED25519 public key: .ssh/id_borg_ed25519
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
aatldbak@bb.tld's password:

(I see ‘debug1: Offering ED25519 public key: .ssh/id_borg_ed25519’ , which mentions the private key; I swapped the keys and renamed them to id_ed25519 without -i option as well. It always asks for a password.)

I managed to have keybased login working by moving the authorized_key-file that borgserver created and setting it up again. I moved back the borg-created file, chown’d the authorized_keys file to aatldbak on bb.tld, no luck.

Remote: Warning: Permanently added 'bb.tld 2001::1:dde::234d' (ECDSA) to the list of known hosts.
Remote: Permission denied, please try again.
Remote: Permission denied, please try again.
Remote: Permission denied (publickey,password).
Connection closed by remote host. Is borg working on the server?
..... (repeat)
Remote: Permission denied, please try again.
Remote: Permission denied, please try again.
Remote: Permission denied (publickey,password).
Connection closed by remote host. Is borg working on the server?

Journalctl -xe gave
backup-with-borg[19105]: ls: cannot access '/etc/yunohost/hooks.d/backup/': No such file or directory
I don’t know what should be there, I symlinked it to backup_method. No matter.
I changed set -e to set -x in /etc/yunohost/hooks.d/backup/05-borg_app , this is the output:

Nov 25 21:32:35 aa.tld systemd[1]: Starting Run backup borg...
-- Subject: Unit borg.service has begun start-up
-- Unit borg.service has begun starting up.
Nov 25 21:32:46 aa.tld backup-with-borg[21590]: Now creating a backup archive from the files collected…
Nov 25 21:32:52 aa.tld backup-with-borg[21590]: Script execution failed: /etc/yunohost/hooks.d/backup_method/05-borg_app
Nov 25 21:32:52 aa.tld backup-with-borg[21590]: Custom backup method failure on 'backup' step
Nov 25 21:32:57 aa.tld backup-with-borg[21590]: Now creating a backup archive from the files collected…
Nov 25 21:33:02 aa.tld backup-with-borg[21590]: Script execution failed: /etc/yunohost/hooks.d/backup_method/05-borg_app
Nov 25 21:33:03 aa.tld backup-with-borg[21590]: Custom backup method failure on 'backup' step
Nov 25 21:33:09 aa.tld backup-with-borg[21590]: Collecting files to be backuped for borg…
Nov 25 21:33:13 aa.tld backup-with-borg[21590]: Now creating a backup archive from the files collected…
Nov 25 21:33:19 aa.tld backup-with-borg[21590]: Script execution failed: /etc/yunohost/hooks.d/backup_method/05-borg_app
Nov 25 21:33:19 aa.tld backup-with-borg[21590]: Custom backup method failure on 'backup' step
Nov 25 21:33:20 aa.tld backup-with-borg[21590]: Collecting files to be backuped for nextcloud…
Nov 25 21:33:21 aa.tld backup-with-borg[21590]: [++..................] > Loading installation settings...
Nov 25 21:33:23 aa.tld backup-with-borg[21590]: [##++................] > Backing up the main app directory...
Nov 25 21:33:24 aa.tld backup-with-borg[21590]: [####++..............] > Backing up nginx web server configuration...t
Nov 25 21:33:25 aa.tld backup-with-borg[21590]: [######++............] > Backing up php-fpm configuration...
Nov 25 21:33:25 aa.tld backup-with-borg[21590]: [########++++........] > Backing up the MySQL database...
Nov 25 21:33:26 aa.tld backup-with-borg[21590]: [############++......] > Backing up logrotate configuration...
Nov 25 21:33:27 aa.tld backup-with-borg[21590]: [##############++....] > Backing up fail2ban configuration...
Nov 25 21:33:29 aa.tld backup-with-borg[21590]: [################++..] > Backing up data directory...
Nov 25 21:33:31 aa.tld backup-with-borg[21590]: Now creating a backup archive from the files collected…

Perhaps I tried something that I didn’t write down. I tried with three Yunohosts; two clients and one server. Installed via the web GUI and via CLI.

If you’re still reading: thank you for your patience :slight_smile:
I hope you can give a hint what else to try!

I have several clients on one server. Note I install a borgserver_ynh for each client (on one physical server).

Thanks for your quick reply, I was unable to reply last night.

Yes, one installation for each client, so they have their own /home/username directory for storage and their own login name.

Do you know what /etc/yunohost/hooks.d/backup is supposed to contain? I made it a symlink to the backup_method, because backup was missing.

Should /home/backupuser/.ssh/authorized_keys be root-owned, or backupuser-owned?

I’ll try removing borg from all installations, see if there are any updates to debian or yunohost and re-installing. All hints welcome ! :wink:

Step by step:

yunohost app install borg on aa.tld
Read email from root@aa.tld
Copy line yunohost app install https://github-address to get the latest version to CLI on bb.tld; change ‘borg_server_ynh’ to ‘borgserver_ynh’ in the address
Answer the questions
Correct the public key in authorized_keys (there was a ‘+’ in the key, it was removed)
service borg start on aa.tld
tail -f /var/log/auth on bb.tld
Nothing happens in the auth log of bb.tld; no messages in journalctl -xe on aa.tld (I didn’t change ‘set -e’ to ‘set -x’ in the hook yet); systemctl status borg.service shows:

Nov 26 21:51:28 aa.tld backup-with-borg[27287]: [################++..] > Backing up fail2ban configuration...
Nov 26 21:51:31 aa.tld backup-with-borg[27287]: Now creating a backup archive from the files collected…
Nov 26 21:51:31 aa.tld backup-with-borg[27287]: Script execution failed: /etc/yunohost/hooks.d/backup_method/05-borg_app
Nov 26 21:51:31 aa.tld backup-with-borg[27287]: Custom backup method failure on 'backup' step
Nov 26 21:51:32 aa.tld systemd[1]: borg.service: Main process exited, code=exited, status=1/FAILURE
Nov 26 21:51:32 aa.tld systemd[1]: Failed to start Run backup borg.
Nov 26 21:51:32 aa.tld systemd[1]: borg.service: Unit entered failed state.
Nov 26 21:51:32 aa.tld systemd[1]: borg.service: Failed with result 'exit-code'.

On aa.tld:

./05-borg_app: line 31: borg: command not found
./05-borg_app: line 44: borg: command not found
./05-borg_app: line 31: borg: command not found

That is strange! aa.tld was borgserver before, I didn’t test borg on this one yet. The installation seemed OK:

# yunohost app install borg
Indicate the server where you want put your backups: bb.tld
Indicate the ssh user to use to connect on this server: aatldbak
You are now about to define a new user password. The password should be at least 8 characters - though it is good practice to use longer password (i.e. a passphrase) and/or to use various kind of characters (uppercase, lowercase, digits and special characters).
Indicate a strong passphrase, that you will keep preciously if you want to be able to use your backups: 
Would you like to backup your YunoHost configuration ? [yes | no] (default: yes): 
Would you like to backup mails and user home directory ? [yes | no] (default: yes): 
Which apps would you backup (list separated by comma or 'all') ? (default: all): 
Indicate the backup frequency (see systemd OnCalendar format) (default: Daily): 
Info: Installing application borg…
Warning: Created symlink /etc/systemd/system/multi-user.target.wants/borg.service → /etc/systemd/system/borg.service.
Warning: Created symlink /etc/systemd/system/timers.target.wants/borg.timer → /etc/systemd/system/borg.timer.
Success! The SSOwat configuration has been generated
Success! Installation complete

Yunohost also thinks borg is installed, yunohost app list

    description: Backup your server with borg.
    id: borg
    installed: True
    label: Borg
    license: BSD-3-Clause
    name: Borg

but,

Sorting... Done
Full Text Search... Done
borg-ynh-deps/now 1.1.10~ynh1 all [installed,local]
  Fake package for borg (YunoHost app) dependencies

borgbackup/oldstable 1.0.9-1 armhf
  deduplicating and compressing backup program

Installing borg via apt, try running service borg once more.

On bb.tld, /var/log/auth prints many attempts:

Nov 26 22:14:45 bb systemd: pam_unix(systemd-user:session): session closed for user aatldbak
Nov 26 22:14:49 bb sshd[25587]: Connection from 2001::1768:1:853f:c923 port 55808 on 2001::1768:1:ddde::234d port 22
Nov 26 22:14:49 bb sshd[25587]: Postponed publickey for aatldbak from 2001:982:1768:1:853f:c9a7:90bd:c923 port 55808 ssh2 [preauth]
Nov 26 22:14:49 bb sshd[25587]: Accepted publickey for aatldbak from 2001:982:1768:1:853f:c9a7:90bd:c923 port 55808 ssh2: ED25519 SHA256:U46GhNSqK+AdaY5Agdcn2J1eCu0Mm+jtfK+iRFemhVk
Nov 26 22:14:49 bb sshd[25587]: pam_unix(sshd:session): session opened for user aatldbak by (uid=0)
Nov 26 22:14:49 bb systemd-logind[604]: New session 36096 of user aatldbak.
Nov 26 22:14:49 bb systemd: pam_unix(systemd-user:session): session opened for user aatldbak by (uid=0)
Nov 26 22:14:50 bb sshd[25587]: User child is on pid 25680
Nov 26 22:14:50 bb sshd[25680]: Starting session: forced-command (key-option) 'borg serve --storage-quota 10G --restrict-to-repository /home/aatldbak/backup' for aatldbak from 2001::1768:1:853f::c923 port 55808 id 0
Nov 26 22:14:50 bb sshd[25680]: Close session: user aatldbak from 2001::1768:1:853f::c923 port 55808 id 0
Nov 26 22:14:50 bb sshd[25680]: Received disconnect from 2001:::1:853f::c923 port 55808:11: disconnected by user
Nov 26 22:14:50 bb sshd[25680]: Disconnected from 2001::1768:1:853f::c923 port 55808
Nov 26 22:14:50 bb sshd[25587]: pam_unix(sshd:session): session closed for user aatldbak
Nov 26 22:14:50 bb systemd-logind[604]: Removed session 36096.
Nov 26 22:14:50 bb systemd: pam_unix(systemd-user:session): session closed for user aatldbak

The error log looks more familiar now:

Remote: Debian GNU/Linux 9
Remote: bash: /usr/local/bin/borg: Permission denied
Connection closed by remote host. Is borg working on the server?
Remote: Debian GNU/Linux 9

Perhaps borg is not installed on bb.tld? Unfortunately borg is installed, no easy solution there :wink:
I do recall from yesterday, that the backupuser on bb.tld, aatldbak, does not have permission to run borg. That may be because it is such a narrow account: it can not normally log in. Ordinary users on the bb.tld are not allowed to run borg either (I got an ordinary user left over from the Armbian base install).

I found mention of 'try borg as non-root, with capabilities’, does that imply borg does need root on the borg server? I understood borg was running on aa.tld as root, and only needed bb.tld as datastore. Is borg needed on borgserver?

[edit]
I had some hope on a solution after reading Borg backup remotely gives error,

Oh, I found the answer here. https://pod.disroot.org/posts/e18176ef-415c-e091-c340-9b5239891533
So, doing which borg on Machine1 was giving me /usr/bin/borg and on Machine2 the output was /usr/local/bin/borg. All it takes is to create a symlink

On aa.tld:

# which borg
/usr/bin/borg    # as installed by apt, after borg_ynh did not

On bb.tld:

# which borg
/usr/local/bin/borg  #as installed by borgserver_ynh

Now each of them is symlinked to the other location.

The error log still reads:

Remote: bash: /usr/local/bin/borg: Permission denied
Connection closed by remote host. Is borg working on the server?
Remote: Debian GNU/Linux 9
Remote: bash: /usr/local/bin/borg: Permission denied
Connection closed by remote host. Is borg working on the server?
^C

I am quite convinced it should work, after all, @ljf tells it is working OK and I read no other people with problems like this. Any idea what to troubleshoot?

borg_ynh install borg with pip in a virtualenv. The version in apt is not ok.

Tomorrow, I will retest a setup from scratch to check all things works well.

Ah, that does explain something.

I did run into missing virtualenv yesterday, added it manually to one of the Yunohosts. On aa.tld tonight it was maybe already installed: there was no complaint about it missing.

Should I remove the version from apt? And the symlinks as well, I guess.

I also noticed on bb.tld, the server, that user aatldbak is not member of staff, and that execution of borg is limited to root anyway:

# ls /usr/local/bin/borg -hals
4.0K -rwxr--r-- 1 root staff 69 Nov 24 22:02 /usr/local/bin/borg
# groups aatldbak
aatldbak : aatldbak

Should the permissions of borg on bb.tld be changed, or does the installation script take care of that?

If you got time for that, that would be great. If I can be of any help, please let me know!

I was reading the feedback at the Github-pages, I see that I am writing down old news :slight_smile:

Could it be some package got updated in October (or earlier) that broke some things?

I recall seeing a conversation on the this forum from before Borg was packaged, I am toying with the idea of following those steps (if they don’t depend on the not-working apt version of Borg).

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.