Home Assistant fails to start with 503 error after creating an automation - how to debug?

Help!

I’ve been a long time HA user and have never managed to totally brick an HA server before but I fear I might have. My server is responding with a 503 error to the /manifest.json request, so it never goes past the loading screen.

What seems to have triggered this is that in trying to experiment with the new automations editor and try to move away from Node-RED, I created an automation triggered by state_changed events, with no conditions or actions, and as soon as I saved it, it seems to have made my server very unhappy indeed.

After power-cycling it, it did come back once, and I tried to delete the automation, only to have it go down again. It came back again just now after an hour, and the automation is still there, but it is unresponsive now again.

What can I do?! I am guessing the problem is my new automation is triggering too frequently?

Looks like moments after it starts it runs out of memory :frowning:

Hi Andrew, is it not possible to start in safe mode?
You are also on OS version 13 which has just been released.
Don’t you have a backup to go back to?

Hey Nick, thanks for the reply! I do save regular backups. So it sounds like that could be a fallback, but I don’t have a USB keyboard to plug in so I am trying to get one today to do that.

How do you restart in safe mode from the local shell?

ha --help

gives you an overview of what you can do from the shell but I haven’t found if/how you can start in safe mode from there.
You have the option though to restore a backup.

Don’t you have the Terminal & SSH add-on installed?

I do have the SSH addon installed but it’s not currently active. I found a USB keyboard though and did core restart --safe-mode from the local shell, it said command completed successfully but requests are still 503ing.

I did core logs and the last entry is pretty much a smoking gun:

So safe mode doesn’t seem to help. I need to delete that automation via root shell somehow

OK, finally managed to delete the automation after a few reboots and now it’s stable. Phew. I think restarting it in safe mode may have helped.

Good you got your system running again!

I use HA core as my main system and HAOS as a test but never needed to use the command line for it.
Now that you mention that command I checked and noticed there are more options in the shell.
Learned something new!