Weird crashes in the morning

fribse · September 24, 2022, 12:52pm

Hi All

The last few weeks I’ve had weird crashes in the morning.
I’m running
Home Assistant 2022.9.6 Supervisor 2022.09.1 Operating System 9.0 on a Proxmox PVE.
So when I get up, the first few automations work, and then suddenly nothing works.
If I open up home assistant it doesn’t respond.
Looking at the console, nothing is shown, I can still SSH into the ‘terminal and ssh’ addon.
I pulled the supervisor logs from the console window and that shows:

[core-ssh ~]$ ha supervisor logs
22-09-24 06:54:08 WARNING (MainThread) [supervisor.addons.options] Option 'dark_mode' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 06:54:08 WARNING (MainThread) [supervisor.addons.options] Option 'require_ssl' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 06:54:56 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
22-09-24 06:59:14 WARNING (MainThread) [supervisor.addons.options] Option 'dark_mode' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 06:59:14 WARNING (MainThread) [supervisor.addons.options] Option 'require_ssl' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 07:04:20 WARNING (MainThread) [supervisor.addons.options] Option 'dark_mode' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 07:04:20 WARNING (MainThread) [supervisor.addons.options] Option 'require_ssl' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 07:05:56 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/info access from cebe7a76_hassio_google_drive_backup
22-09-24 07:05:56 INFO (MainThread) [supervisor.api.middleware.security] /backups access from cebe7a76_hassio_google_drive_backup
22-09-24 07:09:26 WARNING (MainThread) [supervisor.addons.options] Option 'dark_mode' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 07:09:26 WARNING (MainThread) [supervisor.addons.options] Option 'require_ssl' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 07:14:32 WARNING (MainThread) [supervisor.addons.options] Option 'dark_mode' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 07:14:32 WARNING (MainThread) [supervisor.addons.options] Option 'require_ssl' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 07:19:38 WARNING (MainThread) [supervisor.addons.options] Option 'dark_mode' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 07:19:38 WARNING (MainThread) [supervisor.addons.options] Option 'require_ssl' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 07:21:51 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
22-09-24 07:22:01 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
22-09-24 07:22:01 WARNING (MainThread) [supervisor.misc.tasks] Watchdog miss API response from Home Assistant
22-09-24 07:22:16 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
22-09-24 07:22:52 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
22-09-24 07:23:28 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
22-09-24 07:23:32 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
22-09-24 07:24:04 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
22-09-24 07:24:32 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
22-09-24 07:24:32 ERROR (MainThread) [supervisor.misc.tasks] Watchdog found a problem with Home Assistant API!
22-09-24 07:24:32 INFO (SyncWorker_5) [supervisor.docker.interface] Restarting ghcr.io/home-assistant/qemux86-64-homeassistant
22-09-24 07:24:40 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
22-09-24 07:25:16 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
22-09-24 07:26:12 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:26:23 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
22-09-24 07:26:27 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:26:41 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:26:46 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:26:54 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
22-09-24 07:26:56 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:27:01 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:27:11 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:27:16 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:27:24 ERROR (MainThread) [supervisor.utils] Can't execute restart while a task is in progress
22-09-24 07:27:26 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:27:31 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:27:33 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:
22-09-24 07:27:41 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:27:46 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:27:56 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:28:01 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:28:11 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'fribertmqtt'
22-09-24 07:28:13 WARNING (MainThread) [supervisor.homeassistant.websocket] Connection is closed
22-09-24 07:28:13 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Received message 257:None is not str
22-09-24 07:28:13 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
22-09-24 07:28:13 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config: Cannot connect to host 172.30.32.1:8123 ssl:False [Connection reset by peer]
22-09-24 07:28:13 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Received message 257:None is not str
22-09-24 07:28:13 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
22-09-24 07:28:13 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config: Cannot connect to host 172.30.32.1:8123 ssl:False [Connect call failed ('172.30.32.1', 8123)]
22-09-24 07:28:13 INFO (MainThread) [supervisor.auth] Home Assistant not running, checking cache
22-09-24 07:28:13 INFO (MainThread) [supervisor.homeassistant.core] Wait until Home Assistant is ready
22-09-24 07:28:18 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
22-09-24 07:28:18 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.RUNNING
22-09-24 07:28:18 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
22-09-24 07:28:19 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
22-09-24 07:28:19 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/sdesalve/hassio-addons repository
22-09-24 07:28:19 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/danmed/TasmoBackupV1 repository
22-09-24 07:28:19 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/esphome/home-assistant-addon repository
22-09-24 07:28:19 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/zigbee2mqtt/hassio-zigbee2mqtt repository
22-09-24 07:28:19 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/sabeechen/hassio-google-drive-backup repository
22-09-24 07:28:19 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/esphome/hassio repository
22-09-24 07:28:19 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/hassio-addons/repository repository
22-09-24 07:28:19 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/mKeRix/hassio-repo repository
22-09-24 07:28:19 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
22-09-24 07:28:20 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.RUNNING
22-09-24 07:28:21 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
22-09-24 07:28:21 INFO (MainThread) [supervisor.store] Loading add-ons from store: 79 all - 0 new - 0 remove
22-09-24 07:28:21 INFO (MainThread) [supervisor.store] Loading add-ons from store: 79 all - 0 new - 0 remove
22-09-24 07:28:28 WARNING (MainThread) [supervisor.addons.options] Option 'dark_mode' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 07:28:28 WARNING (MainThread) [supervisor.addons.options] Option 'require_ssl' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 07:28:47 INFO (MainThread) [supervisor.auth] Auth request from 'core_mosquitto' for 'mqttintegration'
22-09-24 07:28:48 INFO (MainThread) [supervisor.auth] Home Assistant not running, checking cache
22-09-24 07:29:31 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
22-09-24 07:29:31 INFO (MainThread) [supervisor.homeassistant.core] Detect a running Home Assistant instance
22-09-24 07:29:31 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_appdaemon
22-09-24 07:29:31 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
22-09-24 07:29:31 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
22-09-24 07:29:31 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered
22-09-24 07:29:31 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered
22-09-24 07:29:31 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
22-09-24 07:29:31 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
22-09-24 07:29:31 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
22-09-24 07:30:49 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from core_ssh
22-09-24 07:30:49 INFO (MainThread) [supervisor.api.middleware.security] /network/info access from core_ssh
22-09-24 07:30:49 INFO (MainThread) [supervisor.api.middleware.security] /host/info access from core_ssh
22-09-24 07:30:49 INFO (MainThread) [supervisor.api.middleware.security] /core/info access from core_ssh
22-09-24 07:32:41 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from core_ssh

Which doesn’t say anything to me, except that for me it looks like the ‘node-red’ is suddenly crashing and restarting (I think the ‘Option’ messages only appears when it starts).
This morning I restarted the Supervisor, and then it all started working again, I’m not sure if that solved it, or it was just lucky timing.

tom_l · September 24, 2022, 11:17pm

Yeah this morning at ~07:30 this happened:

Untitled

Before I restarted I went through every addon looking at the CPU use - no issue.

I looked at my core and supervisor CPU use - no issue.

No issues in any of the logs (system, supervisor, host, addons)

Reloading scripts and automations made no difference (no infinite loops going on).

So I gave up looking and restarted.

If it happens again I’ll run the profiler service.

EDIT: you need to remove these two configuration options from the Node Red addon configuration:

22-09-24 07:09:26 WARNING (MainThread) [supervisor.addons.options] Option 'require_ssl' does not exist in the schema for Node-RED (a0d7b954_nodered)
22-09-24 07:14:32 WARNING (MainThread) [supervisor.addons.options] Option 'dark_mode' does not exist in the schema for Node-RED (a0d7b954_nodered)

it’s probably not causing your issue though.

fribse · September 25, 2022, 7:04am

Hi @tom_l
Thankyou, I had a look at the config, it didn’t show those lines, but when I edited in YAML it showed up, so that’s done, must be oooold stuff getting changed at some point.