How to recover a system with a broken SSD?

Hardware
,
#1

Hello folks,

I am running Home Assistant OS on a Raspberry Pi 4 with a SSD. Stupid me thought I could save some money and bought the cheapest SSD I could find on Aliexpress. And surprise: Now it’s broken after roughly 1 year of service.

I am still able to boot and during the first few minutes everything seems to be working, but after a while Zigbee2MQTT stops working and logs a lot of file system related messages. Also the Google Drive Backup Plug-in fails after a while due to file system errors.

So I bought a new WD Red SSD, plugged both SSDs into a Debian machine, and tried to clone the bad SSD onto the new one using dd. Unfortunately this failed after a while with the following output:

$ sudo dd if=/dev/sdb of=/dev/sda status=progress
4458160640 bytes (4.5 GB, 4.2 GiB) copied, 649 s, 6.9 MB/s 
dd: error reading '/dev/sdb': Input/output error
8707384+0 records in
8707384+0 records out
4458180608 bytes (4.5 GB, 4.2 GiB) copied, 651.717 s, 6.8 MB/s

Then I tried to start again from where it failed before, but it failed instantly:

$ sudo dd if=/dev/sdb of=/dev/sda status=progress seek=8707384 skip=8707384
dd: error reading '/dev/sdb': Input/output error
0+0 records in
0+0 records out
0 bytes copied, 0.360525 s, 0.0 kB/s

So I guess the old SSD has broken blocks and cloning seems to be impossible. But I can still mount the data partition and now I am thinking of copying the most important configurations so that I don’t have to configure everything from scratch. For me the most important things are

  • Zigbee2MQTT
  • ESPHome
  • Automations
  • configuration.yaml

Is there anything else I should copy from the data partition? Or do you know if I can try other approaches for cloning the SSD?

Thanks in advance!

#2

Do you have a back in your Google drive that you can restore from as this would be the easiest option?

#3

If you have a new SSD you could try cloning the old one using Macrium Free ( I seem to think you can disable error checking) and see if it works, failing that installl HA to the new SSD and restore your most recent backup if you have one and its still relevant.

#4

Unfortunately only partial backups. I don’t know if they are helpful.

#5

If that is the case, create a backup, and download it.
Then install a fresh installation of HA on the new SSD, and copy the downloaded backup to your new installation using Samba, and restore it to your backup folder. After that you can restore your backup.

Alternatively, you can restore your copy your whole config folder (including the hidden folder .storage) using Samba and reboot :wink:

#6

full backup is not needed and may be difficult in broken state. you only need /config folder ensuring you copy the hidden .storage folder. if you have this, you may use this to restore your install by copy these files to new ssd with new/default installation.

#7

You may need any files under /usr/share/hassio/. Atleast under supervised its called so, dig a bit there. With this, you can maybe get all Addon configs back, but need to place them by hand into the new system and reinstall them by hand.

#8

I was now able to backup the config directory. I will let you know if I was successful restoring.

#9

I successfully recovered my system over the weekend. Unfortunately restoring the config directory did not work. I had to restore the partial backup, install all my plug-ins, and configure my plug-ins again. I also had to remove and re-add the MQTT integration. But now everythign is working again. Thanks for your help!

1 Like
#10

Tip:
Make a regular backup and save it somewhere else then the ssd (f.e nas, google drive)

1 Like
#11

I use samba backup

https://github.com/thomasmauerer/hassio-addons/tree/master/samba-backup

and doing my backups on usb plugged in one of my routers. It saved my life once.

#12

Personally, i do HA supervised backups only when updating one component or when i wanna test something critical. But sincei run HA supervised on ubuntu, i can use my backup program, veeam backup and replication, to backup both my boot drive (where home assistant is installed) as well as my media folder (located on another drive, but mounted under /usr/share/hassio/media). the backup is stored locally on the HA server as immutable backup (no chance for ransomware) and also copied over to my new, bigger server with raid10, since i got it gifted long after getting my HA server up and running and still wasnt in the mood to replace the smaller server.

#13

What else do you expect from Aliexpress? I don’t even buy headphones there anymore, not to mention more serious electronics, they don’t give any guarantee. SSD isnt worth saving money on. Therere some data recovery programs available for this, they can even recover files from a non-functional or broken hard drive, have you tried it? Even if your SSD failed in the motherboard, graphics card, power supply, RAM - even in this case, the hard drive may well be able to work to recover data. You can also contact a specialized service, such as I like Data Recovery or some other, it`ll greatly simplify your efforts.

#14

Thank you so much for this valuable contribution :rofl: