Supervisor stops working with various error messages - how to debug

Hi

Ive been running HAOS on Raspberry PI for a while now. no issues. What seems to correlate to recent HAOS point release update week (happened within last few days) instability started to occur. My Backup list stopped showing all my backups. After reading others issues, I removed old backups to reduce the volume but that made no difference. I can now no longer make a backup from the UI with no visual cue as to why it failed. Clicking the button does nothing.

The logs show these errors and others which show that for some reason, the supervisor is ‘not well’ . How can I determine how and why this happened/ how to resolve?

Any thoughts please.

Logger: homeassistant.components.hassio
Source: components/hassio/init.py:536
Integration: Home Assistant Supervisor (documentation, issues)
First occurred: 8:42:27 AM (1 occurrences)
Last logged: 8:42:27 AM

Can’t read Supervisor data:

other error occurrences reported from supervisior…

  • Timeout on /backups/reload request
  • Timeout on /backups request
  • Timeout on /host/info request
  • Timeout on /addons request

Error fetching hassio data: Error on Supervisor API:

---- Supervisor Logs from last reboot.

23-09-07 08:17:23 INFO (MainThread) [supervisor.store] Loading add-ons from store: 73 all - 0 new - 0 remove
23-09-07 08:17:23 INFO (MainThread) [supervisor.store] Loading add-ons from store: 73 all - 0 new - 0 remove
23-09-07 08:18:11 WARNING (MainThread) [supervisor.misc.tasks] Watchdog miss API response from Home Assistant
23-09-07 08:20:11 ERROR (MainThread) [supervisor.misc.tasks] Watchdog found a problem with Home Assistant API!
23-09-07 08:20:11 ERROR (MainThread) [supervisor.misc.tasks] Home Assistant watchdog reanimation failed!
23-09-07 08:21:35 INFO (MainThread) [supervisor.backups.manager] Found 579 backup files
23-09-07 08:24:05 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
23-09-07 08:24:05 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0bbb674_nodered
23-09-07 08:24:05 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
23-09-07 08:24:06 INFO (MainThread) [supervisor.homeassistant.core] Detect a running Home Assistant instance
23-09-07 08:39:47 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
23-09-07 08:40:24 INFO (MainThread) [supervisor.backups.manager] Found 579 backup files

Also later up the log chain.

  File "/usr/src/supervisor/supervisor/api/store.py", line 179, in <listcomp>
    self._generate_addon_information(self.sys_addons.store[addon])
  File "/usr/src/supervisor/supervisor/api/store.py", line 114, in _generate_addon_information
    ATTR_ADVANCED: addon.advanced,
                   ^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/addons/model.py", line 232, in advanced
    return self.data[ATTR_ADVANCED]
           ^^^^^^^^^
  File "/usr/src/supervisor/supervisor/store/addon.py", line 19, in data
    return self.sys_store.data.addons[self.slug]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'a0bbb674_wireguard'
23-09-07 08:09:50 INFO (MainThread) [supervisor.store] Loading add-ons from store: 73 all - 0 new - 0 remove
23-09-07 08:09:50 INFO (MainThread) [supervisor.store] Loading add-ons from store: 73 all - 0 new - 0 remove
23-09-07 08:14:37 INFO (MainThread) [supervisor.homeassistant.core] Detect a running Home Assistant instance
23-09-07 08:14:37 INFO (MainThread) [supervisor.addons] Phase 'AddonStartup.APPLICATION' starting 7 add-ons

I’m not using wireguard so not sure if this is internal but curious as to why this appears in the log as well?

The Supervisor loads all the add-ons from the store. There is a race condition in current stable release, it should be addressed in 2023.9.0 version of Supervisor (currently on the beta channel, see also Update store data in one task to prevent races by mdegat01 · Pull Request #4519 · home-assistant/supervisor · GitHub).

None of these two issue seem related to the backup problem though. Is it maybe just a bit slow to respond? If this still happens, probably a bug report in the GitHub - home-assistant/supervisor: 🏡 Home Assistant Supervisor repository along with full log files make sense.