Hass.io died, hassos down, recovery procedure / post mortem?

Last night around 7pm my hass.io installation became unresponsive. The duckdns was down, I couldn’t pull up the samba share, and I couldn’t ssh to it anymore. Both network interfaces (ethernet and wifi) were still alive though because I could ping the device.

I tried power cycling it but that didn’t help so I hooked up a monitor and keyboard. When the system booted, it continuously spewed out a ton of STDERR messages. Some of them seemed to be from docker but it was very difficult to use the system because of the constant dumping of messages.

Luckily, I have a backup that I generated two weeks ago and I don’t think I’ve done anything else to the device since then. I powered down the device and pulled the SD card out. I started analysis of the file system to see if I can make heads or tails from it. The first thing I noticed was that there seemed to be duplicate partitions on the old flash card (disk1 and hassos-kernel1). I grabbed a spare flash card and used balena etcher to recreate a hassio environment and to my surprise there were fewer partitions on it:

— old drive —
/dev/sdb2 23M 6.7M 15M 32% /media/spikeygg/hassos-kernel
/dev/sdb3 91M 91M 0 100% /media/spikeygg/disk1
/dev/sdb4 23M 6.7M 15M 32% /media/spikeygg/hassos-kernel1
/dev/sdb5 91M 91M 0 100% /media/spikeygg/disk
/dev/sdb7 89M 19M 65M 23% /media/spikeygg/hassos-overlay
/dev/sdb8 29G 3.9G 24G 15% /media/spikeygg/hassos-data

— new freshly created drive —
/dev/sdb2 23M 6.7M 15M 32% /media/spikeygg/hassos-kernel
/dev/sdb3 91M 91M 0 100% /media/spikeygg/disk
/dev/sdb7 89M 1.6M 81M 2% /media/spikeygg/hassos-overlay
/dev/sdb8 976M 226M 684M 25% /media/spikeygg/hassos-data

The weird thing is, all those partitions are mountable and the duplicates seem to contain the same data as their counterparts. I don’t know what happened here, but it is definitely strange. What’s the best way to do a post mortem on Hass.io? Is there some way to keep the terminal from dumping a slew of messages?