HAOS - How to restart whole installation with Supervisor down? (remote NUC HW)

Teknor · August 16, 2023, 10:43am

Seems to have a fatal error on my HAOS installation. Sorry for not providing all logs, but the system does not allow me to access or download. Front end (lovelace) is mostly working (can naviagate in the dashboards) but most other menu items are failing (like the few addons I have (terminal, esphome, frigate etc), sensor history is failing and no access to menu items to reboot the system.

Home assistant core log is available, but cannot see anything special other than complaining add-ons are not available. When

Home Assistant Core
Error while processing event EventTask(event=<Event state_changed[L]: entity_id=sensor.temp_teknisk_battery, old_state=<state sensor.temp_teknisk_battery=100; state_class=measurement, event=08500781150000e959, unit_of_measurement=%, assumed_state=True, device_class=battery, friendly_name=Temp Teknisk Battery @ 2023-08-16T12:32:29.888591+02:00>, new_state=<state sensor.temp_teknisk_battery=100; state_class=measurement, event=08500788150000e959, unit_of_measurement=%, assumed_state=True, device_class=battery, friendly_name=Temp Teknisk Battery @ 2023-08-16T12:33:17.892585+02:00>>):
12:33:20 – (ERROR) Recorder - message first occurred at August 15, 2023 at 20:04:52 and shows up 188464 times
Timeout error fetching information from http://ccab4aaf-frigate:5000/api/stats:
12:33:16 – (ERROR) Frigate (custom integration) - message first occurred at August 15, 2023 at 20:02:57 and shows up 5935 times
hci0 (BC:A8:A6:7F:38:FB): Failed to restart Bluetooth scanner: hci0 (BC:A8:A6:7F:38:FB): Failed to start Bluetooth: [org.freedesktop.DBus.Error.NameHasNoOwner] Could not activate remote peer: unit failed.
12:33:13 – (ERROR) Bluetooth - message first occurred at August 15, 2023 at 20:07:34 and shows up 1972 times
Client error on /supervisor/info request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]
12:33:00 – (ERROR) Home Assistant Supervisor - message first occurred at August 15, 2023 at 20:06:05 and shows up 2352 times
Catching up, dropped 206 old events.
12:32:07 – (WARNING) InfluxDB - message first occurred at August 15, 2023 at 20:04:52 and shows up 612 times
Error doing job: Task exception was never retrieved
12:30:52 – (ERROR) helpers/storage.py - message first occurred at August 15, 2023 at 20:04:53 and shows up 83 times
[Errno 5] I/O error: '/usr/local/lib/python3.11/site-packages/hass_frontend/frontend_latest/17375-boVibCmoIpM.js'
12:30:39 – (ERROR) components/http/static.py - message first occurred at August 15, 2023 at 21:48:26 and shows up 860 times
Error while processing event StatisticsTask(start=datetime.datetime(2023, 8, 16, 10, 5, tzinfo=datetime.timezone.utc), fire_events=True): The database connection has not been established
12:30:10 – (ERROR) Recorder - message first occurred at August 15, 2023 at 20:05:10 and shows up 199 times
Client error on api app/entrypoint.js request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]
12:29:32 – (ERROR) Home Assistant Supervisor - message first occurred at August 15, 2023 at 21:49:45 and shows up 12 times
Can't read Supervisor data:
12:29:24 – (WARNING) Home Assistant Supervisor - message first occurred at August 15, 2023 at 20:09:24 and shows up 195 times
Failed to to call /addons -
12:20:42 – (ERROR) Home Assistant Supervisor - message first occurred at August 15, 2023 at 21:50:04 and shows up 55 times
Client error on /addons request Cannot connect to host 172.30.32.2:80 ssl:default [Connect call failed ('172.30.32.2', 80)]
12:20:42 – (ERROR) Home Assistant Supervisor - message first occurred at August 15, 2023 at 21:50:04 and shows up 56 times
Error handling request
12:14:36 – (ERROR) Frigate (custom integration) - message first occurred at August 15, 2023 at 21:48:39 and shows up 27 times

For the supervisor logs, I get this error message:

Restarting HA does not make any difference, and full system reboot is not available any more.

Is there a way to force reboot from remote or repair / restart supervisor?

BebeMischa · August 16, 2023, 11:32am

Power it off and on then?

stevemann · August 16, 2023, 11:53am

I had a similar situation a couple of weeks ago. I returned from a three-week vacation and realized that there had been no backups generated in the past ten days. Home Assistant, automations and most add-ons were working, but no Zigbee or 433MHz devices. Nothing in the logs or any individual troubleshooting solved the problem until I did a reboot of the server. Everything has been working well since then.

Teknor · August 16, 2023, 11:56am

Yes, the challenge is the installation is 200km away…

stevemann · August 16, 2023, 12:00pm

Road trip.

Can you SSH into the server? If yes you can do a ha host reboot.

Teknor · August 16, 2023, 12:50pm

I have installed the ssh & terminal addon, but the addon menu window is all white. If the URL is generic (after :8123/ ) I can try a direct URL. If so, what is the URL for the terminal /ssh?

edit: tried http://x.y.z:8123/hassio/addon/core_ssh/ but window is all white.
also tried /hassio/ingress/core_ssh without luck.

tom_l · August 16, 2023, 1:32pm

If you enable Advanced Mode in your profile you should see an extra set of options expandable at the bottom of the restart pop-up:

Untitled

Teknor · August 16, 2023, 2:26pm

Yes @tom_l , I’m in advanced mode and system reboot etc was available before. Unfortunately not now. And editing profile info does not work:

Seriously need a (remote) reboot

stevemann · August 16, 2023, 2:35pm

How do you connect to the remote server? DuckDNS? Other?

tom_l · August 16, 2023, 2:45pm

Oh dear.

If your supervisor is playing up as per your logs you are not going to be able to use addons either.

Teknor · August 16, 2023, 8:16pm

I use ZeroTier which is still working (for a strange reason). Had DuckDNS earlier, but went over to ZT. Has been stable and (for what I know) secure for a long time.

stevemann · August 16, 2023, 9:21pm

This looks interesting. So using Zerotier you can connect to a device on a remote network as if it were on the local network?

Teknor · August 16, 2023, 10:00pm

Yes. I’m not an expert in the details how zerotier works but it lets you connect securely to a remote installation (reversed proxy / vpn).

tom_l · August 17, 2023, 3:06am

Yeah even though it uses http it is end to end encrypted. That freaked me out for a bit before I realised.

My employer has some sort of block on nabu casa (even though they deny it) but zero tier works fine.

Very easy to set up.

stevemann · August 18, 2023, 4:52am

Have you solved your reboot problem yet? Re-reading your original post, can you see the Developer’s page? You might be able to use a service call to reboot the host.

service: hassio.host_reboot
data: {}

Or write an automation or a script to make the service call. How about Node Red?

francisp · August 18, 2023, 9:35am

If supervisor is hanging, hassio.host_reboot won’t work.

gpambrozio · October 26, 2023, 5:58pm

This just happened to me for a second time. The first time I had to restart my pi 4 by switching power off and on. Would love to have a better solution to this as powering off this way might lead to a whole sd corruption.

boheme61 · October 26, 2023, 6:10pm

connect a keyboard and a monitor to your pi, and reboot from the console there

gpambrozio · October 27, 2023, 7:44pm

Even that didn’t work for me. I think that, since I booted with no monitor connected, connecting a monitor and keyboard didn’t show anything. In the end I still just powered it off and on. Would be really nice to have a way to do this remotely too as this might happen when on vacation or in situations where the server is hard to access…

boheme61 · October 27, 2023, 7:49pm

then you just have to install an add-on, in the default/native add-on folder, there are 2 different Terminals with SSH, 1 more advance, but what you need is the SSH Feature, to be able to access from another pc/mac