Data rescue on system disaster

Hello.

If I do something wrong and my Yunohost system breaks but the filesystem is still accessible, where could I check for email messages arrived or sent before the incident?

If I see there are recent messages that were not included in the last backup, I guess I could copy them somehow and, once the system has been restored from backup, I could add those newer messages to my restored system.

How could I do so? (I don’t know file locations, possible permission or ownership particularities, etc).

Thank you

/var/mail

Thank you. I guess you mean I could just replace the whole folder and done.

Of course, make a backup

What about Mastodon and XMPP?

Would it be that easy? What are their locations?

For XMPP:

  • If you are running Metronome on YNH12 : /home/yunohost.app/metronome/
  • If you are running Metronome on YNH11 : /var/lib/metronome/
  • If you are running Prosody : /var/lib/prosody/
1 Like

I’m thinking of running an external NFS server on the same LAN and then configure my yunohost as an NFS client so that all the data on the directories you are telling me can be stored on my external NFS server. I think that way I could solve the problem exposed here.

My goal is to be able to restore Yunohost after an incident while keeping data intact.

Now I know the paths for mail and XMPP, but what about other apps? How do I find out which data directory or directories I should put on the NFS server?

For the time being I am interested in Cryptpad, Bookstack and Mastodon.

/home/yunohost.app /$app

Just move /home

2 Likes

Thank you. I guess I could also have /var/www in the NFS server. I think some apps store their data there.

It is good to have a recovery plan for when disaster hits.

‘Yunohost’ as such is the system that helps you install and manage apps and users. Once, say, Nextcloud or your mailserver is installed, and Yunohost itself breaks, those apps keep working.

Of greater concern is not Yunohost breaking, but one of your apps breaking. The Yunohost team does a great job of creating update packages with built-in fail-safes, but a server is a complex system and not every eventuality can be programmed against.

Therefor there are fallbacks:

  • backups on Yunohost level, provided by Yunohost on your server
  • help on your system, provided by other Yunohost users on this forum
  • backups / snapshots of your server, provided by yourself

Backups of complete working apps/systems are much easier to work with than backups of individual files (such as emails or XMPP messages).

Do you run your server at home or somewhere in a datacenter?

@jarod5001 @anubis

Sometimes I do mistakes on my Yunohost system. So, in order to be able to restore my Yunohost machine from a snapshot without losing any data, I am installing an external NFS server (on the same LAN). So, on the NFS server I will put:

/var/mail
/var/lib/metronome/
/home
/var/www
/var/log

Do you see any problem with this approach?

Can you think of any other directory I should add?
Databases?

My idea is to put in the NFS server all those directories that have data that change often. I don’t see the need to put a configuration file that does not change often. Anyway I will backup both machines (the NFS server more often).

Thank you.

I will keep databases locally. I have just read databases can be problematic on NFS. Not only about delay, but also about things like data integrity.

So now I have a third question, is there any database inside any of these directories?

/var/mail
/var/lib/metronome/
/home
/var/www
/var/log

And still have the other two previous questions:
Do you see any problem on having those directories on NFS?
Is there any other directory you would include for NFS?

no

But to restore yunohost apps, you have to restore other things in /etc, in /var/lib/mysql or /var/lib/postgresql and the ldap database…
May be you have some backup of those data into /home/yunohost.backup/archives ?

Thank you.

Yes. I see things in there, I guess they include databases.
But as they are static archives, I guess there will be no NFS-Database conflict, right?

The point is I want to move as much data as possible out from the yunohost system, as long as it is completely safe and easy to do. Few large directories is ok, I do not want a complicated solution. That way I will be able to restore more easily the yunohost system when it fails. I will be able to do it faster (as it will not have to read and write any of that heavy data on the NFS) and I will not miss any of the data written to NFS between the last backup and the yunohost failure.

I don’t know if this is a common approach, if you see any problem please let me know!

Totally not, in my experience.

What, in your mind, does a/the “Yunohost failure” look like?

  • The “Yunohost layer” does not eat data. If you remove the Yunohost packages from your Debian, all installed apps and their data will stay on your server.
  • If Yunohost breaks, your apps are still there, as is their data
  • If an upgrade breaks an app: for small apps the data is backed up, for large apps (Nextcloud, for example), the data is not touched.

People on the forum are very helpful. You get what you ask for :stuck_out_tongue:

If your goal is : quick recovery after disaster, how does your recovery plan look like with data on NFS?

  • disaster strikes
  • full server image gets restored from last backup
  • data is still there, because NFS

Possible unwanted side-effects:

  • Databases are not on NFS, so new files added to apps as Nextcloud, messages in Matrix, and the like, are not available on your restored server, because their database only got backed up with the full server backup.
  • Your unnamed backup soluition may or may not make database-safe backups, but you probably took care of that by stopping the services during backup, or you backup memory state besides the disk image.

Which kinds of disaster do you want to defend against, and which availability factor do you guarantee your users or yourself?

What kind of infrastructure do you run on? Something like Proxmox can snapshot your server (including RAM), as would a number of filesystems if you run bare-metal. You could creat daily (hourly?) snapshots, and revert to an earlier snapshot if disaster strikes.

For Yunohosts that do not run on Proxmox, I have the borg-backup directory mounted remotely, so that backups end up off-site. It does not allow hour by hour recovery (these run only daily), but it does allow to restore a single backup, or restore the whole Yunohost to another server instead that one “disappears”