What to do with large media folder(s) with YUNOHOST

I have a rather large media folder (Music/Videos/Pictures), its about 300gig plus in size. It used to reside on a QNAP NAS but I would like to move these folders away and into my YUNOHOST ecosystem.

I have a good YNH server set up which includes Nextcloud, Piwigo and Ampache. I only have a few users (4). I have been testing the likes of Ampache and Piwigo with a sample bunch of files and they all do the job very well.

My question is how to deal with such a large amount of data and keep it backed up and safe.

Is this big media folder better on an external disk?

The YNH backup will become enormous if I back it up, with days to upload into the archive folder if I need to restore it one day. Or should I just exclude the media folder from the backup?

As a back story, if it helps, my YNH machine is a VM inside a QNAP NAS. I like it where it is because the VM software (Virtualisation Station 3) takes a daily snapshot of the VM as well as a full backup of the VM to a 1TB USB drive every night at 2am.

Confused about what to do…

dj

All I can say to help is that some backup tools (for example Borg) use incremental backups, so only the first backup is big, the next ones will only send the difference, not everything.
And do not worry if you delete the first backup, data present in other backups will stay :slightly_smiling_face:

Here is my NextCloud’s backup info for example

------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:               12.21 TB             11.85 TB            447.87 GB

                       Unique chunks         Total chunks
Chunk index:                  678751             17132962

The deduplicated size (size size used on disk) is slightly greater than the actual data used even if it represent 12 TB of backup :slightly_smiling_face:

Wow. That’s a lot of data.

Hmmm. That Borg thing scares me a bit. Can you mount a USB drive and use Borg to back up to that?

Dj

I do my backup on a different drive, in USB, but it is always connected.
The default configuration use a cron to do the backups, but it may be possible to deactivate the cron and only launch it manually.
The beauty of the borg app in YunoHost is that it uses the backup process of YunoHost to make the backups (and so, the restore process can also be used after).
If you make a backup by yourself, you’ll have to be really sure of what to backup and how, and how to do the restauration, but it is also possible.

And 447GB is not really more than your 300GB :grin:

@Mamie Thanks very much. I have ordered a new 3TB USB drive. I want to do the same thing, plug in the drive, set it up and the leave it…

I had a look at the app in yunohost and I “think” I can work it out…

  • Plug in drive
  • Find it in (lsblk)
  • mount it
  • format it
  • Set up for mounting automatically
  • Set up borg to run etc

Most of this is the helpful YNH docs. As soon as the drive gets here I will give it a go!!!

Wish me luck!

Dj

1 Like

@Mamie

Sorry to trouble you…

So I bought and plugged a USB drive into my server, followed the instructions to see it, format it and mount it automatically.

In fact, I have just restarted the server, and here is my USB drive (2TB), partitioned and named.

sda/sda1 /media/storage

I then installed the Borg app and set up the repository as shown:

Same as sda/sda1?

did a:

systemctl status borg.service

Terminal stalled, ie nothing happened, so I opened another terminal and did a:

nano /var/log/backup_borg.err

and saw the following:

Can’t be found… Doh!!!

What did I miss?

Have a great day/night

dj

1st thing to know is that the first backup will be VERY LONG (for real, can be days because of the de-duplicationqprocess when you have a lot of data).

For your error, can you test a command like borg status [path] ?
Maybe there will be more details.
(A password will be requested, password that you set when you installed the app)

Hi

I’ll have a look at path, which I thought was /media/storage. Yep I set a password when I installed the app

Dj