[Borg & Borg Server] Deduplicated, encrypted and remote backups

Sadly .nobackup file works only in direct subdir of /home
BUT /mnt/backups/borgarchives should not be backupped by yunohost anyway.

1 Like

Ok,
if I understood, excluding the app borg__2 will exclude also the data…
Now it is strange, this morning (early) the first backup borg from server A to B where I have change the files to backup adding exclude:borg__2 after all in the apps to backup seems do the contrary !

This is my config (webadmin)

The message this morning is not what excepted:

Creating a backup archive from the collected files...
The archive will contain about 1.2MiB of data.
Backup created
name: auto_conf
results: 
  apps: 
  system: 
    conf_ldap: Success
    conf_manually_modified_files: Success
    conf_ynh_certs: Success
    conf_ynh_settings: Success
size: 1267154
Creating a backup archive from the collected files...
The archive will contain about 177.1MiB of data.
Backup created
name: auto_data
results: 
  apps: 
  system: 
    data_home: Success
    data_mail: Success
    data_multimedia: Success
    data_xmpp: Success
size: 185651353
Collecting files to be backed up for borg__2...
Loading installation settings...
Declaring files to be backed up...
Backup script completed for borg__2. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 104.2KiB of data.
Backup created
name: auto_borg__2
results: 
  apps: 
    borg__2: Success
  system: 
size: 106717

Normally I have this type of message including all apps:

Creating a backup archive from the collected files...
The archive will contain about 1.2MiB of data.
Backup created
name: auto_conf
results: 
  apps: 
  system: 
    conf_ldap: Success
    conf_manually_modified_files: Success
    conf_ynh_certs: Success
    conf_ynh_settings: Success
size: 1267154
Creating a backup archive from the collected files...
The archive will contain about 177.1MiB of data.
Backup created
name: auto_data
results: 
  apps: 
  system: 
    data_home: Success
    data_mail: Success
    data_multimedia: Success
    data_xmpp: Success
size: 185719023
Collecting files to be backed up for borg...
Loading installation settings...
Declaring files to be backed up...
Backup script completed for borg. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 112.8KiB of data.
Backup created
name: auto_borg
results: 
  apps: 
    borg: Success
  system: 
size: 115542
Collecting files to be backed up for borg__2...
Loading installation settings...
Declaring files to be backed up...
Backup script completed for borg__2. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 104.2KiB of data.
Backup created
name: auto_borg__2
results: 
  apps: 
    borg__2: Success
  system: 
size: 106717
Collecting files to be backed up for cachet...
Loading installation settings...
Declaring files to be backed up...
Backing up the MySQL database...
Backup script completed for cachet. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 70.1MiB of data.
Backup created
name: auto_cachet
results: 
  apps: 
    cachet: Success
  system: 
size: 73465478
Collecting files to be backed up for coturn...
Loading installation settings...
Declaring files to be backed up...
Backup script completed for coturn. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 153.6KiB of data.
Backup created
name: auto_coturn
results: 
  apps: 
    coturn: Success
  system: 
size: 157323
Collecting files to be backed up for cryptpad...
Loading installation settings...
Declaring files to be backed up...
Backup script completed for cryptpad. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 681.1MiB of data.
Backup created
name: auto_cryptpad
results: 
  apps: 
    cryptpad: Success
  system: 
size: 714146088
Collecting files to be backed up for garradin...
Loading installation settings...
Declaring files to be backed up...
Backup script completed for garradin. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 54.6MiB of data.
Backup created
name: auto_garradin
results: 
  apps: 
    garradin: Success
  system: 
size: 57276614
Collecting files to be backed up for garradin__2...
Loading installation settings...
Declaring files to be backed up...
Backup script completed for garradin__2. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 353.6MiB of data.
Backup created
name: auto_garradin__2
results: 
  apps: 
    garradin__2: Success
  system: 
size: 370821306
Collecting files to be backed up for grav...
Loading installation settings...
Declaring files to be backed up...
Backup script completed for grav. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 2.1GiB of data.
Backup created
name: auto_grav
results: 
  apps: 
    grav: Success
  system: 
size: 2207442799
Collecting files to be backed up for mattermost...
Loading installation settings...
Declaring files to be backed up...
Backing up the MySQL database...
Backup script completed for mattermost. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 606.1MiB of data.
Backup created
name: auto_mattermost
results: 
  apps: 
    mattermost: Success
  system: 
size: 635593787
Collecting files to be backed up for mobilizon...
Loading installation settings...
Declaring files to be backed up...
Backing up the PostgreSQL database...
Backup script completed for mobilizon. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 425.4MiB of data.
Backup created
name: auto_mobilizon
results: 
  apps: 
    mobilizon: Success
  system: 
size: 446041825
Collecting files to be backed up for nextcloud...
Loading installation settings...
Declaring files to be backed up...
Backing up the MySQL database...
Backing up data directory...
Backup script completed for nextcloud. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 1.6GiB of data.
Backup created
name: auto_nextcloud
results: 
  apps: 
    nextcloud: Success
  system: 
size: 1745109128
Collecting files to be backed up for opensondage...
Loading installation settings...
Declaring files to be backed up...
Backing up the MySQL database...
Backup script completed for opensondage. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 15.5MiB of data.
Backup created
name: auto_opensondage
results: 
  apps: 
    opensondage: Success
  system: 
size: 16252922
Collecting files to be backed up for roundcube...
Loading installation settings...
Declaring files to be backed up...
Backing up the MySQL database...
Backup script completed for roundcube. (YunoHost will then actually copy those files to the archive).
Creating a backup archive from the collected files...
The archive will contain about 45.6MiB of data.
Backup created
name: auto_roundcube
results: 
  apps: 
    roundcube: Success
  system: 
size: 47813738

Sorry, I understand just now, I can let all, it will only backup borg__2 app but not the folder /mnt/app/borgarchives… Thanks so much

Just a last question, if I upgrade the home server B to yunohost 11 but keep the vps server A to yunohost 4.3.6.3 is not a trouble ? ( I wait for a stable and secure migration for the server outside hosted on a VPS )…

I am not totally sure that symlink here is a good idea, i prefer mount --bind See doc/external_storage.fr.md at refactor-admin-guide · YunoHost/doc · GitHub

I think it’s ok if you doesn’t try to migrate an app backup into 4.x to 11.x (or 11.x to 4.x).

I don’t really see how to explain… In your link I only see how add an external storage.
I have do few partitions in my server, this is my fstab

cat /etc/fstab 
UUID=84123fb9-bde8-41dd-a9fc-afd4fe57b0e6	/	ext4	defaults	01
UUID=cc55dbd0-1abc-411a-a7aa-8a4f6d5eaf50	/mnt/apps	ext4	defaults0	1
UUID=86131a6e-ed0d-4071-aa81-41b50e8d3a50	/mnt/backups	ext4	defaults0	0
UUID=4a5b5a94-88ea-4328-a05b-4a3fae339ed2	/mnt/media	ext4	defaults0	1
UUID=d0294479-075c-480b-972e-0131fef47a5d	/var/mail	ext4	defaults0	1
UUID=983ea66d-4732-431a-9d31-86eb85bb9431	/mnt/pad	ext4	defaults0	1
UUID=7ed24519-ae94-459b-acff-af6b12a72c2a	swap	swap	defaults	00
UUID=37dc28a5-c4ab-4260-bd4f-d9006f40cfaa	swap	swap	defaults	00
UUID=f0ccd1dd-4592-43ac-85de-ee1c149dd688	swap	swap	defaults	00

The disks (Raid1 on 3 disks):

$ lsblk
NAME              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                 8:0    0   1.8T  0 disk  
├─sda1              8:1    0  68.4G  0 part  
│ └─md1             9:1    0  68.3G  0 raid1 /
├─sda2              8:2    0     2G  0 part  [SWAP]
├─sda3              8:3    0   1.8T  0 part  
│ └─md3             9:3    0   1.8T  0 raid1 
│   ├─vg-mail     253:0    0 195.3G  0 lvm   /var/mail
│   ├─vg-data     253:1    0   293G  0 lvm   /mnt/media
│   ├─vg-CRYPTPAD 253:2    0  97.7G  0 lvm   /mnt/pad
│   ├─vg-APPS     253:3    0   293G  0 lvm   /mnt/apps
│   └─vg-backups  253:4    0 913.7G  0 lvm   /mnt/backups
├─sda4              8:4    0     1K  0 part  
└─sda5              8:5    0   1.9M  0 part  
sdb                 8:16   0   1.8T  0 disk  
├─sdb1              8:17   0  68.4G  0 part  
│ └─md1             9:1    0  68.3G  0 raid1 /
├─sdb2              8:18   0     2G  0 part  [SWAP]
└─sdb3              8:19   0   1.8T  0 part  
  └─md3             9:3    0   1.8T  0 raid1 
    ├─vg-mail     253:0    0 195.3G  0 lvm   /var/mail
    ├─vg-data     253:1    0   293G  0 lvm   /mnt/media
    ├─vg-CRYPTPAD 253:2    0  97.7G  0 lvm   /mnt/pad
    ├─vg-APPS     253:3    0   293G  0 lvm   /mnt/apps
    └─vg-backups  253:4    0 913.7G  0 lvm   /mnt/backups
sdc                 8:32   1   1.8T  0 disk  
├─sdc1              8:33   1  68.4G  0 part  
│ └─md1             9:1    0  68.3G  0 raid1 /
├─sdc2              8:34   1     2G  0 part  [SWAP]
└─sdc3              8:35   1   1.8T  0 part  
  └─md3             9:3    0   1.8T  0 raid1 
    ├─vg-mail     253:0    0 195.3G  0 lvm   /var/mail
    ├─vg-data     253:1    0   293G  0 lvm   /mnt/media
    ├─vg-CRYPTPAD 253:2    0  97.7G  0 lvm   /mnt/pad
    ├─vg-APPS     253:3    0   293G  0 lvm   /mnt/apps
    └─vg-backups  253:4    0 913.7G  0 lvm   /mnt/backups
zram0             252:0    0    32M  0 disk  [SWAP]
zram1             252:1    0    32M  0 disk  [SWAP]
zram2             252:2    0    32M  0 disk  [SWAP]
zram3             252:3    0    32M  0 disk  [SWAP]
zram4             252:4    0    32M  0 disk  [SWAP]
zram5             252:5    0    32M  0 disk  [SWAP]
zram6             252:6    0    32M  0 disk  [SWAP]
zram7             252:7    0    32M  0 disk  [SWAP]

I have move some folders with symlinks to allocate some spaces for some apps, for example:

$ ls -l /home/yunohost.app/
total 8
drwxr-xr-x 2 root       root      4096 Jan 18 17:17 coturn
lrwxrwxrwx 1 mattermost www-data    20 Dec 10  2021 mattermost -> /mnt/apps/mattermost
lrwxrwxrwx 1 mobilizon  mobilizon   19 Dec 10  2021 mobilizon -> /mnt/apps/mobilizon
drwxr-x--- 7 nextcloud  nextcloud 4096 Jan 19 14:25 nextcloud

$ sudo ls -l /home/yunohost.app/nextcloud/
total 20
lrwxrwxrwx 1 nextcloud nextcloud   25 Jan 19 14:25 data -> /mnt/media/nextcloud_data
drwxr-xr-x 2 nextcloud nextcloud 4096 Dec  9  2021 user1
drwxr-xr-x 2 nextcloud nextcloud 4096 Dec  9  2021 user2
drwxr-xr-x 2 nextcloud nextcloud 4096 Dec  9  2021 user3
drwxr-xr-x 2 nextcloud nextcloud 4096 Dec  9  2021 user4
drwxr-xr-x 2 nextcloud nextcloud 4096 Dec  9  2021 user5

$ ls -l /var/www/
total 40
drwxr-x--- 12 cachet      www-data 4096 Jun 12 12:17 cachet
lrwxrwxrwx  1 root        root       17 Feb  1 23:26 cryptpad -> /mnt/pad/cryptpad
dr-xr-x---  7 garradin    www-data 4096 Jul  3 18:22 garradin
dr-xr-x---  7 garradin__2 www-data 4096 Jul  3 18:23 garradin__2
drwsr-s--- 15 grav        www-data 4096 Apr 26 04:15 grav
drwxr-xr-x  2 root        root     4096 Dec  9  2021 html
drwxr-x--- 11 mattermost  www-data 4096 Jun 26 23:29 mattermost
drwxr-x---  7 mobilizon   www-data 4096 Jan 18 17:17 mobilizon
drwxr-x--- 13 nextcloud   www-data 4096 Apr 30 14:36 nextcloud
drwxr-x--- 17 opensondage www-data 4096 Dec  6  2021 opensondage
drwxr-x--- 14 roundcube   www-data 4096 Jun 27 17:20 roundcube

except /var/mail/ which is on the partition edited on the fstab, there some symlinks !

I do this also because I tried restore Nextcloud time ago with a borg backup where the data was not moved with a symlink but with a fstab edited and the config.php, but there was an error trying restore, I thought a symlink was better…

1 Like

I am sorry i have not linked the good tutorial:doc/external_storage.md at partition · YunoHost/doc · GitHub

Ok, thanks… now tell me, isn’t it hard to change all my configs ? Does It means delete the symlinks, move the folders in the right place (or are they yet on the good place), do all the procedure explained ?
or just unlink the symlinks and replace them with the fstab with bind ?? and reboot or first a mount -a ?

Thanks a lot @ljf , tell me if in my case can I delete the symlinks ( https://linuxize.com/post/how-to-remove-symbolic-links-in-linux/) first and just a mount --bind on as explained ? What is the difference between symlinks and mount --bind ??

I rolled out my backup copy on a new server as a move. I think it works great anyway, with the tools from Yunohost. Especially keep an eye on the DNS. Almost everything is restored. What’s great is that I started this. Restore the base on the admin interface. Which took a while, by opening the archive. But now I’m very happy out of Diaspora, which I guess is your own fault. :blush:

Ich habe meine Sicherungskopie ausgerollt, auf einem neuen Server als Umzug. Ich denke, dass es hervorragend trotzdem geht, mit den Werkzeugen von Yunohost geht. Besonders dabei die DNS im Blick halten. Es ist fast alles wieder hergestellt. Was toll ist, dass ich damit angefangen habe. Auf der Admin-Oberfläche den Grundstock wieder herzustellen. Was etwas gedauert hat, durch Öffnen des Archivs. Doch jetzt bin ich sehr glücklich außer Diaspora, was wohl an Ihnen selbst liegt. :blush:
grafik

I don’t remember well why mount bind is better, i think it’s due to the way yunohost backup the file and also how some apps could be disturb by symlinks.

I am not sure symlink target are added in backup list (or maybe they are not correctly added so restore an app could crash due to that. It’s possiblke it create unoptimize things and need a lot of more available space on the origin system to backup it.

I think you can just delete the symlink and mount bind (and add the info into fstab to be sure the mount bind will be done at boot).

1 Like

borg fails to install on fresh ynh11 updated system

I was to restore another app and tried to use borg from CLI (as other times), but borg-env failed so I “forced” upgrade (as is noted on migration info) but then borg got unistalled and now I can not install it back.

https://paste.yunohost.org/raw/qusapirozo

ynh11 on raspi4

I’m now copying old files from backup (from previous ynh 4.x) and fingers crossed untill I can reinstall on ynh11

please, as it seems it is personal problem (after one week nobody else is reporting the package as broken), could anyone tell about what the error is caused by? Some “lib” packages seem to be missing or deprecated (look the log file).

both restoring previous version from backup or force install (-F) fail

it’s now a week without any backup, so crossing fingers nothing brakes this days untill borg is back in my system :stuck_out_tongue:

thank you!

1 Like

I notice


2022-08-17 16:05:23,387: WARNING -   × python setup.py bdist_wheel did not run successfully.
2022-08-17 16:05:23,388: WARNING -   │ exit code: 1
2022-08-17 16:05:23,393: WARNING -   ╰─> [394 lines of output]
2022-08-17 16:05:23,394: WARNING -       Detected and preferring liblz4 over bundled LZ4
2022-08-17 16:05:23,395: WARNING -       /opt/borg-env/lib/python3.9/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.

You could try manually installing the pip mentioned a few lines above that,

2022-08-17 16:04:38,773: DEBUG - + /opt/borg-env/bin/python /opt/borg-env/bin/pip install wheel

So,

$ sudo pip install wheel

and see what happens. If there is an error, it is not specific to Yunohost and you may find a solution on the 'net. If there is no error, you can retry installing borg.

Good luck! :slight_smile:

1 Like

many thanks for taking your time to take a look at log :pray:

wheel is already installed though

root@ynh:~# pip install wheel
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: wheel in /usr/lib/python3/dist-packages (0.34.2)

thank you!

I saw wheel somewhere in the log and forgot to pay attention. It actually says bdist_wheel! Is that one installable manually?

– edit: I searched for bdist_wheel online, and found a post over at stackoverflow. In that case it were problems with venv’s, but perhaps it gives you a hint.

1 Like

:frowning: it is not (instalable).

many thanks :+1:

1 Like

I recreated my repo at Borgbase (deleted and created another), and the app stopped to work after updating the link. Fails on init. Should I reinstall it, or did someone had this issue already? =(

I think you should reinstall the borg app, or search in /var/log/borg the reason why it failed.

If you change the repo destination, it could be an issue with ssh key or fingerprint issue.
Try:

borg list YOUR_REPO

To know more

1 Like

that did the trick
I really try not to solve problems with re-installs. =(( Seems like I shouldn’t change the repo until Borg app get some more pace of update and maintenance.

Thank you!