Borg SSH Host key verification failed

josh · January 3, 2022, 10:30pm

//

My YunoHost server

Hardware: Two old laptops (one for server, another one for client)
YunoHost version: 4.3.5
I have access to my server : Through the webadmin and SSH
Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : Yes, maybe
If yes, please explain: Old machines, i386, had to install Yunohost on top of Buster, both server otherwise working.

Description of my issue

Hey there!

I was trying to set up a Borg Server on my Red machine, to provide backup space for my Blue machine.

However, when setting up the Borg App on the Blue machine to backup remotely on the Red machine it returned an error:

Remote: No ECDSA host key is known for red.machine.yh and you have requested strict checking.
Remote: Host key verification failed.

Error logs here:
https://paste.yunohost.org/raw/epuyuyiduc

Not sure what failed in the set up here? Could not find specific guides on setting up the server on YH.
Can anyone help here please?

Thanks!

wbk · January 4, 2022, 3:33pm

I have set up Borg about a year ago, with the same error and could not really get it working. A couple of weeks ago I tried again, and it worked directly (but I changed the installation since)
Maybe one time I first installed the server and then the client, and the other time first the cilent and then the server. I can’t remember, and have no idea why it works now.

That is something you can check. Are you OK with using SSH? You could install “Shell in a box” if you don’t have a program for SSH on your computer/telephone.

I can only help you in troubleshooting, my first try did not work, the second time worked without a problem, and now I changed it again so I can’t peek at the correct installation
Do you remember the user you let Borg Server create? I’ll use ‘queen’ in the example.

Log in with ssh admin@blue.machine.yh, providing your admin password. I think borg runs as root. To check whether there is a key available,

$ sudo su - # change to root. 
# ls /root/.ssh # list contents of root's ssh dir
#               # probably there is nothing, at the moment

A couple of files are often found in the ssh-dir:

known_hosts: those hosts have been visited before, or their info is added
authorized_keys: users identified by this key have access
id_rsa: private RSA-key
id_rsa.pub: public RSA-key
id_ecdsa: private ECDSA-key
id_ecdsa.pub: public ECDSA-key

RSA-keys are the default when generating a new key. Borg seems to use ECDSA-keys. For one reason or another, the key is not generated while installing the borg-app, or, if it is created but there is an RSA key as well, it does not select the right key.
Log in again, or if you’re still logged in, continue (as root)

# ssh-keygen -t ecdsa       #     and then type 'enter' a few times, accepting the defaults
# ssh-copy-id queen@red.machine.yh  # copy the _public_ key (it will automatically take the public key, it is safe to give to anyone _but don't give the private key_)
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_ecdsa.pub"
The authenticity of host 'red.machine.yh (ip)' can't be established.
ECDSA key fingerprint is SHA256:/VuUao.iXNG48htJpfy5U36lGAZR5bPwnaRzQao92ee.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Debian GNU/Linux 10
queen@red.machine.yh's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'queen@red.machine.yh'"
and check to make sure that only the key(s) you wanted were added.
# ssh admin@red.machine.yh
Debian GNU/Linux 10
...
queen@red.machine.yh$ # you could log in, so can Borg

Ok, you enabled and checked that root@blue can log in as queen@red, doing so using the ecdsa-key.

Now, if you try installing the Borg-client-app again on blue, does it work?

Probably things go different on your side halfway the story, let’s see how things go!

josh · January 4, 2022, 6:03pm

Thank you so much for taking the time to reply.
I’m following some of your tips and taking a gradual approach to understand and troubleshoot.

You would have to install the Borg app before the server, because only then will you have the client public key for installation.

So first thing I do have an ECDSA key pair in the Blue (client) machine.

But it seems somehow the public key is not copied to the Red server.

Then I followed your instructions on how to copy the public borg key from the Blue machine to the Red machine.

I did try to log the Blue machine (queen@red.machine.yh) but could not authenticate with the queen borg user password.

But then I deleted and re-installed the Borg server again with no errors.

No backups triggered yet, but will keep you posted. They will start at midnight I hope not turning into pumpkins.

Thank you

wbk · January 4, 2022, 7:40pm

When I read it again, my conclusion was the other way around: you have to install Borg server first, because only then you will create the user on the server (user “queen” in this example).

There a few things to check when ssh queen@red.machine.yh does not authenticate:

the user has to exist on the red side
- check /home/ -directory for regular users, or /etc/shadow for all users (the user has to be in the shadow list) to find out whether your borg-user exists
the user has to have permission to log in (see below); you can also check that after log in as admin@red.machine.yh by sudo su borg-user (your regular Yunohost users are not allowed CLI access, it will OK for the password but kick you out immediately)
the password, of course, has to match for password logon

When things check out OK locally on the red machine. you can also paste the public key directly in /home/borg_user/.ssh/authorized_keys , in the format of ‘key-type key remark’, for example ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDU7Q16...RU3OyLjIP68GILOE6Mkq+UMMV root@blue.machine.yh

When you have a look at the groups and permissions on red.machine.yh, is your borg-user there? I would expect that it is, and that it has SSH or SFTP permissions,

This way the user has permission to access the system

If you’re patient enough, lets wait and see if there’s a backup tomorrow

josh · January 4, 2022, 10:07pm

@wbk

Thanks for jumping in.

So checking in the Host machine (admin@red.machine.yh)

The user does exist both in the /home/ directory, and in the /etc/shadow

sudo su borg-user (“queen” to follow the same example) does not prompt for a password and results in a CLI prompt like this (no prepending username@host):

The borg-user (aka ‘queen’) is not in visible in the admin panel of the Host machine.

Also, before you posted your response, I noticed the Borg service on the Blue client machine failed. Checking the service logs I noticed the user was apparently “borg”, maybe because that was my initial username try the first time I attempted to install?

So I uninstalled the Borg app on the Blue client machine, and installed it again.

The same error remains on the borg service logs:

Could not start the service ‘borg’
Recent service logs:-- Logs begin at Tue 2022-01-04 02:52:30 WET, end at Tue 2022-01-04 19:51:16 WET. – Jan 04 19:28:26 systemd[1]: Starting Run backup borg… Jan 04 19:28:26 sudo[10795]: borg : /etc/sudoers.d/borg is owned by uid 994, should be 0 ; TTY=unknown ; PWD=/ ; USER=root ; Jan 04 19:28:26 sudo[10795]: sudo: /etc/sudoers.d/borg is owned by uid 994, should be 0 Jan 04 19:28:26 sudo[10795]: We trust you have received the usual lecture from the local System Jan 04 19:28:26 sudo[10795]: Administrator. It usually boils down to these three things: Jan 04 19:28:26 sudo[10795]: #1) Respect the privacy of others. Jan 04 19:28:26 sudo[10795]: #2) Think before you type. Jan 04 19:28:26 sudo[10795]: #3) With great power comes great responsibility. Jan 04 19:28:26 sudo[10795]: sudo: no tty present and no askpass program specified Jan 04 19:28:26 sudo[10795]: pam_unix(sudo:auth): conversation failed Jan 04 19:28:26 sudo[10795]: pam_unix(sudo:auth): auth could not identify password for [borg] Jan 04 19:28:26 sudo[10795]: borg : user NOT authorized on host ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/backup-with-borg borg Jan 04 19:28:26 systemd[1]: borg.service: Main process exited, code=exited, status=1/FAILURE Jan 04 19:28:26 systemd[1]: borg.service: Failed with result ‘exit-code’. Jan 04 19:28:26 systemd[1]: Failed to start Run backup borg. Jan 04 19:51:16 systemd[1]: Starting Run backup borg… Jan 04 19:51:16 sudo[16943]: borg : /etc/sudoers.d/borg is owned by uid 994, should be 0 ; TTY=unknown ; PWD=/ ; USER=root ; Jan 04 19:51:16 sudo[16943]: sudo: /etc/sudoers.d/borg is owned by uid 994, should be 0 Jan 04 19:51:16 sudo[16943]: We trust you have received the usual lecture from the local System Jan 04 19:51:16 sudo[16943]: Administrator. It usually boils down to these three things: Jan 04 19:51:16 sudo[16943]: #1) Respect the privacy of others. Jan 04 19:51:16 sudo[16943]: #2) Think before you type. Jan 04 19:51:16 sudo[16943]: #3) With great power comes great responsibility. Jan 04 19:51:16 sudo[16943]: sudo: no tty present and no askpass program specified Jan 04 19:51:16 sudo[16943]: pam_unix(sudo:auth): conversation failed Jan 04 19:51:16 sudo[16943]: pam_unix(sudo:auth): auth could not identify password for [borg] Jan 04 19:51:16 sudo[16943]: borg : user NOT authorized on host ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/backup-with-borg borg Jan 04 19:51:16 systemd[1]: borg.service: Main process exited, code=exited, status=1/FAILURE Jan 04 19:51:16 systemd[1]: borg.service: Failed with result ‘exit-code’. Jan 04 19:51:16 systemd[1]: Failed to start Run backup borg.

josh · January 5, 2022, 8:32am

I already did

chown root:root /etc/sudoers.d/borg

As mentioned here: [Borg & Borg Server] Deduplicated, encrypted and remote backups - #77 by Melchisedech

But I still get the same error logs from the service which is weird?

EDIT: had to do chown twice for it to stick, and then yunohost webadmin froze while restarting the borg service.
Apparently no action on the borg server, so I guess it’s not backing up.

Rebooted after a while. Error logs:

Could not restart the service 'borg'
Recent service logs:-- Logs begin at Wed 2022-01-05 09:35:25 WET, end at Wed 2022-01-05 10:31:35 WET. -- Jan 05 09:58:41 systemd[1]: Starting Run backup borg... Jan 05 09:58:41 sudo[2362]: borg : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/backup-with-borg borg Jan 05 09:58:41 sudo[2362]: pam_unix(sudo:session): session opened for user root by (uid=0) Jan 05 09:58:41 sudo[2371]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/yunohost app setting borg last_run -v 220105_0958 Jan 05 09:58:41 sudo[2371]: pam_unix(sudo:session): session opened for user root by (uid=0) Jan 05 09:58:57 sudo[2362]: Another YunoHost command is running right now, we are waiting for it to finish before running this one Jan 05 09:59:42 sudo[2362]: Still waiting... Jan 05 10:02:43 sudo[2362]: Still waiting... Jan 05 10:14:43 sudo[2362]: Still waiting... Jan 05 10:31:35 sudo[2371]: pam_unix(sudo:session): session closed for user root Jan 05 10:31:35 sudo[2362]: pam_unix(sudo:session): session closed for user root Jan 05 10:31:35 systemd[1]: borg.service: Main process exited, code=killed, status=15/TERM Jan 05 10:31:35 systemd[1]: borg.service: Failed with result 'signal'. Jan 05 10:31:35 systemd[1]: Stopped Run backup borg.

I don’t know how to trigger the borg back up manually. The service is up but repository not created yet. Will wait until cron does his thing at midnight. Maybe pumpkin pie tomorrow?

josh · January 6, 2022, 8:22am

Ok the command

chown root:root /etc/sudoers.d/borg

did the trick. Unsure why it froze while restarting the service, but maybe just because it takes a loooong while to process the backup the first time?

@wbk thank for all the help, copying the public key manually was a great tip.

system · January 21, 2022, 8:22am

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.