Just had my first system loss!! Yay. I feel it’s a badge of honour all us HA’ers get when we catastrophically lose our installation. Usually when you are already overwhelmed and stressed with little time to fix! But ha ho all is fixed. My system has been cured.
So what is better than a cure? . . . prevention of course. My failure was the HDD. A new SSD I bought 3 months ago. All was going fine and fast but recently I wanted to start setting up some ESP32home devices and thought. You know what - think it about time I did a clonezilla. I have snapshots but wanted to take a full image for easy recovery if I needed to then to start doing it weekly. So I shut her down ready for cloning . . . then saw some ominous messages about not being able to write back standard Debian data, then more messages and then more serious ones then the shutdown failed! My supervised HA on Buster was crashed 1/2 through a shutdown. I held the power switch off and then on again and prayed.
. . . Like the Titantic, that disk and the DB on it is now lost, and the souls (bits) onboard her have gone too. No amount of rescue disks could recover the filesystem. The disk cannot even be formatted!!
So, a new disk has gone in, rebuilt and snapshot restored. Great. However, all history lost
Now for the open discussion: prevention & disaster recovery.
First prevention - how do you monitoring your systems health? I don’t mean the System Health integration as that would not have caught my errors but a real deep monitoring like synology has or OMV5. Seems a bit odd that HA, a home monitoring specialist, doesn’t first and foremost, monitor itself?! Or maybe it does and I missed the warning signs?
Second DR? HA collected a season of temp data from every room in my house last winter ready for me to make some smart logic and install new TRV valves this winter. Sadly, all the history I have now is . . . 24hrs
So I have learnt the hard way that snapshot are only config data. Even a FULL snapshot is not like a VM FULL snapshot. Nothing like it at all.
So what do you do to back up your entire system? I was going to clonezilla mine but not really a fan of taking it off line while I do. Am thinking about using linux dd to clone an image to my OVM5. Maybe there is already something out there that makes it even easier like - insert USB and press go. If system fails insert/boot from that USB and restore.
Would love to hear everyones thoughts and suggestions. Maybe a good podcast topic?