From rescue shell to recovery - Adventures with a corrupt VirtualBox VDI

Rescue mode but no rescuer

I thought I would share my experience after a blue screen stopped my Oracle VirtualBox VM running Home Assistant from running.

I have some Linux command line experience so I thought I would get to know Home Assistant by trying to fix it with a little help from Google and ChatGPT.

The first sign of trouble was this depressing You are in rescue mode

Typing reboot at the # prompt a couple of times did not resolve the issue.

I also tried using Slot B rather than Slot A in that GNU GRUB start-up screen. At least this time I saw some containerd errors but then a blank screen and a Slot B rescue shell followed.

Containerd snookerd

The journalctl -xb command revealed lots of scary error messages linked to containerd so I resigned myself to trying to rebuild HA from scratch.

I downloaded the VDI from https://www.home-assistant.io/installation/windows/ and extracted the VDI file to the same folder as the VM. Rather than build up the VM itself from scratch, I backed up the folder and then went to the corrupt VM's settings - storage - hit add attachment and then Add (disk image file) and selected the new VDI file.

This was enough to get me to a pristine set-up and I was ready to restore a backup.

Backed up

I had been careful to configure a daily backup with at least 7 backup copies. And...of course, these were on the VDI disk that had the corrupted HA. :person_facepalming:

Then I thought "how hard could it be to get hold of that file. The VDI's just a file system after all".

It would be hard.

The good news is that VDI files can be read by 7-Zip and what I saw looked promising: a compressed file of the form {466....59}.img.

But when I tried to extract the file I got and error Unsupported compression method

Thankfully I "discovered" :robot: that there is a tool in VirtualBox that allows the extraction of this image file so I ran the command:

VBoxManage clonemedium disk {466...59}.vdi olddisk.img --format RAW

This is usually found in C:\Program Files\Oracle\VirtualBox

So I finally had a .img file that, in theory, could be mounted in Linux. I eagerly spun up a wsl (Windows Subsystem for Linux) session, created a /mnt/tmp folder and did a

mount /path/to/olddisk.img /mnt/tmp

It worked and I was able to rummage around the old disk for the backup files.

cd /mnt/tmp; ls -l

This revealed folders such as supervisor and logs.

Emergency kit emergency

I discovered that the files were kept in the supervisor folder under backup:

cd supervisor/backup; ls -lt

So all I had to do was get that tar file onto my file system with a quick copy.

cp Automatic....tar /path/to/local/folder

That was it. Now all I had to do was to go to Upload backup and select the tar file for the backup file and that would be it. But no - I had to enter an encryption key protecting the tar file backup.

It was at this point I realised that there was an encryption key somewhere on my filesystem as part of the "Emergency kit" and I could not recall what file name I had used :scream:. Surely it was stored on the old disk? The answer was yes :smiling_face_with_sunglasses: - in the supervisor/homeassistant/.storage/backup file.

grep password backup

The response is of the form "password": "xxxx-....-xxxx"

I was now able to restore the backup. And a few minutes later...success:

Lessons Learnt (promise)

I hope that this is been helpful because I struggled to find this information in one place. Here are the lessons I have learnt.

  1. Backup on external media or the cloud. HA supports Samba and you can access the backup directory by mounting it. A scheduled task, for example, can be used to copy the backups somewhere safe.
  2. Remember where you put your emergency kit.
  3. Consider taking regular snapshots of your VDI file if your hard disk has enough space.
  4. Read the rules regarding posting for new users. Most of my lovely embedded images had to be removed. At least emojis are allowed. :fireworks: