Borg: Encryption envelope checksum mismatch ; unable to use repo in any way, repair = delete

[wbk]

Hi all!

My YunoHost server

Hardware: rented VPS online
YunoHost version:

• yunohost version: 11.2.8.2 (stable)
• yunohost-admin version: 11.2.3 (stable)
• moulinette version: 11.2 (stable)
• ssowat version: 11.2 (stable)
  I have access to my server : Through SSH | through the webadmin
  Are you in a special context or did you perform some particular tweaking on your YunoHost instance ? : no
  If your request is related to an app, specify its name and version:
  id: borg
  version: 1.1.16

---

This is not really a cry for help, more a registration of how I got rid of a corrupt situation. “The Borg tag is not allowed in discuss or tutorial”, so I post in support anyway

For a while, I have been receiving mails from Borg that backups did not complete successfully.

Now that I look into it, it turns out to be going on for quite a while, ending in Encryption envelope checksum mismatch

borg check --info returns thousands of checksum errors.

Any Borg command other than check, returns errors along these lines, also delete:

# borg delete --force --force /backup/borgremote/online
Enter passphrase for key /backup/borgremote/online: 
Data integrity error: Chunk (unknown): Encryption envelope checksum mismatch
Traceback (most recent call last):
  File "/opt/borg-env/lib/python3.9/site-packages/borg/archiver.py", line 4690, in main
    exit_code = archiver.run(args)
  File "/opt/borg-env/lib/python3.9/site-packages/borg/archiver.py", line 4622, in run
    return set_ec(func(args))
  File "/opt/borg-env/lib/python3.9/site-packages/borg/archiver.py", line 177, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/opt/borg-env/lib/python3.9/site-packages/borg/archiver.py", line 1309, in do_delete
    return self._delete_repository(args, repository)
  File "/opt/borg-env/lib/python3.9/site-packages/borg/archiver.py", line 1385, in _delete_repository
    manifest, key = Manifest.load(repository, Manifest.NO_OPERATION_CHECK)
  File "/opt/borg-env/lib/python3.9/site-packages/borg/helpers.py", line 384, in load
    data = key.decrypt(None, cdata)
  File "/opt/borg-env/lib/python3.9/site-packages/borg/crypto/key.py", line 390, in decrypt
    raise IntegrityError('Chunk %s: Encryption envelope checksum mismatch' % id_str)
borg.helpers.IntegrityError: Data integrity error: Chunk (unknown): Encryption envelope checksum mismatch

I am able to run a borg check --repair on the repository, that takes quite long (all those errors…)

borg check --repair /backup/borgremote/online/
This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!

Type 'YES' if you understand this and want to continue: 

... 
Skipping corrupted chunk: Data integrity error: Chunk eb15ed151b95a824b2240ff15a7bf5a976c8a92cf06710e3ffa0b2e6f737d51c: Encryption envelope 
checksum mismatch
... 
31674 orphaned objects found!
Archive consistency check complete, problems found.

After this, borg list succeeds in listing 0 archives.

I am not quite sure what triggered the integrity error. I have three VPS-Yunohosts backing up to one at home, they all have the same problem.

Since the problem has been going on for a while and there is not a great amount of data involved, I want to delete the repository and start with an initial backup (the check --repair did not seem to have any effect any way )

After the ‘repair’, the delete still throws an error, resource busy:

# borg delete /backup/borgremote/online/
Enter passphrase for key /backup/borgremote/online: 
You requested to completely DELETE the repository *including* all archives it contains:
Type 'YES' if you understand this and want to continue: YES
Local Exception
Traceback (most recent call last):
  File "/opt/borg-env/lib/python3.9/site-packages/borg/archiver.py", line 4690, in main
    exit_code = archiver.run(args)
  File "/opt/borg-env/lib/python3.9/site-packages/borg/archiver.py", line 4622, in run
    return set_ec(func(args))
  File "/opt/borg-env/lib/python3.9/site-packages/borg/archiver.py", line 177, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/opt/borg-env/lib/python3.9/site-packages/borg/archiver.py", line 1309, in do_delete
    return self._delete_repository(args, repository)
  File "/opt/borg-env/lib/python3.9/site-packages/borg/archiver.py", line 1403, in _delete_repository
    repository.destroy()
  File "/opt/borg-env/lib/python3.9/site-packages/borg/repository.py", line 369, in destroy
    shutil.rmtree(self.path)
  File "/usr/lib/python3.9/shutil.py", line 722, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/lib/python3.9/shutil.py", line 720, in rmtree
    os.rmdir(path)
OSError: [Errno 16] Device or resource busy: '/backup/borgremote/online'

Platform: Linux sanyi.nl 5.10.0-25-amd64 #1 SMP Debian 5.10.191-1 (2023-08-16) x86_64
Linux: Unknown Linux  
Borg: 1.1.16  Python: CPython 3.9.2 msgpack: 0.5.6.+borg1
PID: 3913739  CWD: /root
sys.argv: ['/opt/borg-env/bin/borg', 'delete', '/backup/borgremote/online/']
SSH_ORIGINAL_COMMAND: None

It did seem to do the trick, though:

# borg delete --force /backup/borgremote/online/
/backup/borgremote/online is not a valid repository. Check repo config.
# borg check --info /backup/borgremote/online/
/backup/borgremote/online is not a valid repository. Check repo config.

I’m off to rebuild the repository and start backing up

---

[wbk]

TL;DR: If your Borg is out of order because any command yields an encryption envelope error and you don’t mind to rebuild the repository from scratch ( = lose all backups up till that point), you can:

• borg check --repair /path/to/repository/
• wait longer or shorter, depending on network speed and amount of data
• borg list /path/to/repository/ did not return any archives after this
• borg delete --force /path/to/repository/
• delete might throw an error, but the repository was gone in my case.