Today I noticed I had a Core and OS update to do.
After running one of them from my phone (and I don’t remember which it was, or their version numbers - helpful I know), I have a non-responsive HA.
I run HA OS on an HP mini desktop thing (the type companies used to wedge under monitors after 10 years ago). That’s all this machine does, there’s no strange VM setup.
There is no access from the external URL, no access from the internal IP. I plugged in a monitor and see a blank white (mostly) screen with Autoboot in the top left corner.
Plugging in a keyboard lets me hit enter, the screen flashes some text very quickly. By recording it I can just make out the end of the message “nothing bootable found”.
I’m really hoping there’s a straightforward approach here (I’ve looked a little but haven’t found a solution on the forums yet), rebuilding this thing is not on the cards for today.
My best case here is someone gets mad at me for posting a duplicate issue and points me to a thread with a solution. (I’ll take that as a win today).
Alright, seeing as this is bothering me, here’s some more details, and what I’ve tried.
The machine is a HP Compaq 8200 Elite USDT PC, the SSD is a Kingston A400 120GB.
The most troubling part is that after pulling the SSD I’m also unable to get it to show up in Disk Mgmt on my Windows 10 desktop, using a SATA-USB tool.
Dead SSD? Right after an update? That looks mighty strange, and really unlikely.
Does HA make significant use of the disk during normal operation that would cause a massively shortened lifespan?
Thing is, I do get some messages flash past about barebox right before it hangs on the Autoboot screen. And the HA PC does recognize that the drive is attached, according to it showing up in the BIOS.
I get what you’re saying here, but I’m not looking for it in Windows Explorer (mounted filesystem) I’m looking for it in diskmgmt.msc - at a disk level.
Turns out the SATA-USB device doesn’t play nice through a USB hub, plugging in to a tower case USB port made the difference.
Yes, A powered 7-Port USB hub, TP-Link.
Having bypassed that I’m now able to set the disk as offline in diskmgmt.msc and then add it as a harddrive to my Ubuntu VM (running locally in Hyper-V), so I can access the filesystem on the disk (hopefully grab some of the configuration to use in a new install, but also now is a good time to crack open some logs).
As you can see in the previous post there are several partitions, would you know where to start looking for logs that might help tell the story of what happened that caused this issue on update?
Ideally I’d be able to post this as a specific issue for the devs on github, or add some of the logs to an issue which already exists (because I can’t have been the only one).
Ah well, thanks @stevemann. I do appreciate you helping me get this far.
I flashed a different SSD with HA OS 7.6 and it flies through the boot process and gets to the HA command line without issue. I’m rather confident at this point that it’s not a BIOS or machine issue, and something to do with the boot process.
If anyone out there knows of an open issue with a request for logs on this sort of thing please let me know. I’ll keep this SSD as-is for now in-case it can be used for further diagnostics.
Using a Ubuntu VM in Hyper-V I added the old SSD as a harddrive, and could pull files off of it.
Within the hassos-data partition I went for the backups folder and the homeassistant folder that had my .yaml files. Also good to grab is the supervisor>homeassistant>.storage> lovelace dashboard files. From the VM I’d copied all this to a network share on my NAS.
With a fresh install on a ‘new’ (old but empty) SSD I booted up and was given the option to restore from a backup. This part wasn’t very clear, as the backups all said they were partial files, and the UI kinda sat there after selecting one. I think it took a reboot to kick things into action, and I saw a login page rather than the onboarding page.
After than I realized my dashboards had been updated since that backup, so I copied them from the old SSD to the new SSD within Ubuntu.
At this point I’m checking for any updates and making a FULL backup (which I will start doing more often now).
It seems I’m back up and running, but I ran into trouble trying to open the .journal files from the old SSD within Ubuntu. Something about Protocol not supported - what a nuisance.
Another thing that was not immediately obvious, after copying overall of the backups it was not immediately obvious to me how to revert to a backup point. Clicking on the backup (not the checkbox next to it) brings up the dialog you’re looking for.
In the log file, the last message I have (which is right around the time this afternoon that I hit update and took a shower - blissfully unaware of the havoc I’d just created) was: 2022-04-07 13:35:16 ERROR (MainThread) [homeassistant.components.hassio] Failed to to call /available_updates - System is not ready with state: shutdown
Edit: I’m not marking this as a solution, because starting over isn’t fixing the problem, it’s ignoring it.
I’m still interested to see if someone has a suggestion on how to repair the boot ability of the install on the old SSD.
Get a second SSD. Booting a fresh-flashed HAOS followed with a snapshot restore has saved my butt a couple of times. Do a daily full snapshot using the Samba backup add-on and you will be safe if this ever happens again.
@dabell If you can reach the barebox shell, you could manually boot into the old slot. This seems to be a more general issue though, an update shouldn’t have been started in that state.
I’m not able to interact with the barebox shell during the current boot, it flickers a message at me very quickly then drops to the blank screen with Autoboot in the top left. There is a message I can just make out that says something about pressing ‘m’ for a menu, but either I haven’t been able to get it fast enough, or it just takes me to the same Autoboot screen faster.
I’ll point a camera at it during boot and upload it here in a few.
Still a bit concerned that accepting an update caused this. I’d been running rather stable for about a month, the only change I’d made within that time was adding a new user so my father could control some lights while he visited.
Second, seemingly the only thing that redraws the screen, hitting enter. I do this repeatedly in the hope I’ll be able to capture enough of the message on video to read what it’s saying.
That’s a very odd sight… If you write e.g. devinfo and then press return. Does it change anything in the text behind the white square? The current text is barebox telling you that it failed to boot from either slot.
So bootchooser doesn’t know where the partitions are located anymore?
It seems the partitions are intact, I’m wondering if it’s possible to re-write the boot targets and have that result in the system coming back up.
I’m not well versed on how HA OS is stacked. Guess I’ll have to look at some of the dev documentation to see if it’s made clear.
Based on this experience alone it seems Barebox as bootloader - then some linux base - then some higher linux OS - then HA data and overlay data.
Yes exectly, it doesn’t know where the partitions are located.
I had this problem once to.
I have a linux test machine, i just installed windows 11, and downloaded virtualbox and downloaded the Homeassistant OS 7.5 and mount it, it works perfect.
(I’ll take that as a win today).
