Warning/General Advice/Don't be as silly as me

Configuration
1

This is pretty well all my fault and no blame on HA, but more a general comment/advice. With my experience I should have known better…

I have a debian system on x86-64 running supervised.

Too long to describe the full sequence, but I suddenly noticed that my regular backups were not completing, and I had no backup other than the ones that the system does on an addon update. There were often dB errors in mariadb. At the same time I noticed errors in dmesg, which alerted me to the fact that my ssd drive might be dying.

Unable to fix any of that, I became worried that although I could replace the hard drive, install debian then supervised and restore a backup, I didn’t have a backup. I didn’t want to lose 90 days of database, or my automations etc.

I rebooted the machine and debian said the root filesystem was no no good and it ran fsck, but it wouldn’t complete. Several times. I got a little panicy.

I “fixed” the hard drive with a boot usb with ubuntu on it, ran fsck. It took several goes, and a long time. Some of the repeated tries were possibly the result of impatiently rebooting when the fsck process was taking an inordinate time with no feedback.

Eventually the disk was rendered usable and I was able to start debian and the supervised home assistant. I immediately initiated a backup which finished 1 1/2 hours later. I sent that off to another computer. Then I tried to fix the mariadb errors, but I completely broke mariadb. It would no longer start.

I restored the mariadb section of the backup I had taken when the system started to semi-behave, and now all is sweeetish, but I still have a potentially dying SSD which I will need to replace this weekend.

Lessons, some of which I should have learned years ago:

  1. Make sure your backups are actually backing up
  2. Keep an eye on your hard drive, at any sign of error, it is probably dying and needs to be junked
  3. If your database is repeatedly erroring, see 2 and check 1.
6 Likes
2

You forgot the lesson that the backup should not be on the HA device.
You got lucky and were able to retrieve your backup, but many disk death are sudden with none or corrupt data as all you can get out of it.

1 Like
3

Wally beat me to it. The big take-away is that HA doesn’t support backing up (directly) to off-line storage. Backups are made only to the local drive (such as SSD or SD.) There are add-ons which can then copy that backup to a safer place, but you’ve got to know to install and monitor them. Even so, you’ve expended a lot of writes, reducing the lifespan of the local device, for no real benefit.

I’ve submitted a Feature Request to back up directly to another device. It’s pretty much gone unnoticed so far, but maybe a few more up-votes would blow some of the dust off.

4

Since backing up is a manual process anyway why would you need any add-on for downloading the backup after creation?
Simply make your backup in HA and then from within HA download it to a safe place. Problem solved.

5

Hm… my backups are automatic, from supervisor…

but, one advantage in having HA on Synology (or any other similar NAS…) is that it has protection against HDD dying by default. But still, i have automation set up (inside Synology) which copies (regular automatic) backup files from default location to another folder on my Syno network drive. So, if VM part of HA “dies” i still have a working backup.
My HA runs in VM on Syno, but database is separate in mariaDB - if db is “default” then backups are painfully big and slow. I don’t care too much for history, if it’s gone, it’s gone. The main thing is HA config.

6

Backups are only good if you know they will work. I had an “incident” where we tried to recover using backups that were stored off-site only to discover the backups were bad and could not be used. The backup procedures were followed, the media was stored correctly in safe place, but the backups were unreadable…all the backups; not just one or two.

Periodically test your backups to see if they work.

7

Similar issues recently, having set up an automated backup schedule to take partial partial daily backups and weekly full backups, then off load them to cloud, I thought I was covered well…

However the first time I ever needed to use a backup I discovered that although the restore worked, I was littered with mariaDB errors and noted the DB was not running.

Long story short I lost all my DB data and have no idea as to why, many reports online about the same issue as I experienced, relating to the DB running while the backup is taken, however HA version history suggest this was an issue but isn’t anymore.

I am yet to try doing another restore and left worrying that if I ever do again I will loose my DB again.

8

I agree that you don’t need an add-on. As you say, for me, it’s a manual process anyway. But there are add-ons which automate backing up and copying the backup elsewhere. They work very well.

This is what I do. But there is still one issue with this approach. The backup is created on the local (to HA) drive. That’s fine if you’re running HA on your NAS like @Protoncek does.

But if you’re following the new user recommendations and installing HA on a RPi with an SD card for the local drive, backing up is performing lots of writes, which we all know shortens the life of the SD card. Worse, they are wasted writes because if you need to restore the backup, you’ll need to do so from another location anyway. There really is no good reason to create the backup locally, since you have to copy it somewhere else anyway. My FR was to back up directly to that other location in the first place.

9

A highly write-intensive application suite like HA should not even start when intended to be run on an SD-card if you ask me, no doubt about that.
I also agree that a backup should never be stored anywhere near the source itself.
I second your FR. Until something like that is available, the HA-backup tool should remind (or better nag) the user to download the backup-file from the system…

2 Likes
10

FWIW, I constantly mention that we need to have the ability to mount network drives to the people that make decisions.

5 Likes
11

+1 for this.

Far less work than having Veeam Community Ed. backup the VM. On top of the Google Drive backup addon.

12

Thanks! I agree a good interim step would be to encourage/nag the user to download the backup at the time it’s created.

And thank you, too! For anyone interested, feel free to up-vote this FR requesting the ability to mount NAS drives. It seems like that would be an enabling technology which would make my FR much easier to implement. Failing that, I could see the backup file downloading directly to the machine that’s running the browser which kicked off the backup. Basically, I’d support anything which keeps the backup off the machine running HA!

13

I do use the google drive backup addon. But the backups were not completing so nothing was uploaded to google drive.

All good discussion here. There are also addons to upload backups to samba share or amazon s3.