Home-assistant stops data processing each day (sometimes twice)

Hello,

since the last update to 2021.1.5 my home-assistant is not reliable anymore as it was before. I’m running it on a RPI4 on a supervised OS.

I can see that data processing is stopped, as the graphs are not updated anymore. Also automations do not trigger anymore. But my node-red automations still work, as I received the notifications about activities from node-red (using pushover).

These are the last lines of the log from this night before it stopped and before the restart at 08:15:

2021-01-28 17:57:15 WARNING (MainThread) [homeassistant.components.zwave] Z-Wave not ready after 300 seconds, continuing anyway
2021-01-28 23:33:41 WARNING (MainThread) [homeassistant.components.webhook] Received message for unregistered webhook a0704c90b6076be2ad174faa108829ac61e863376b07807192f9d69b70b3180f from 192.168.20.105
2021-01-29 00:38:41 WARNING (MainThread) [homeassistant.components.webhook] Received message for unregistered webhook a0704c90b6076be2ad174faa108829ac61e863376b07807192f9d69b70b3180f from 192.168.20.105
2021-01-29 04:12:00 ERROR (Recorder) [homeassistant.components.recorder.util] Error executing query: Couldn't parse datetime string: '173f8bb786b42fc9615c79013add5538'
2021-01-29 04:12:00 ERROR (Recorder) [root] Uncaught thread exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/src/homeassistant/homeassistant/components/recorder/__init__.py", line 364, in run
    if not purge.purge_old_data(self, event.keep_days, event.repack):
  File "/usr/src/homeassistant/homeassistant/components/recorder/purge.py", line 30, in purge_old_data
    states = execute(query, to_native=True, validate_entity_ids=False)
  File "/usr/src/homeassistant/homeassistant/components/recorder/util.py", line 78, in execute
    result = [
  File "/usr/src/homeassistant/homeassistant/components/recorder/util.py", line 78, in <listcomp>
    result = [
  File "/usr/src/homeassistant/homeassistant/components/recorder/util.py", line 80, in <genexpr>
    for row in (
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 100, in instances
    cursor.close()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.raise_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 80, in instances
    rows = [proc(row) for row in fetch]
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 80, in <listcomp>
    rows = [proc(row) for row in fetch]
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 579, in _instance
    _populate_full(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 725, in _populate_full
    dict_[key] = getter(row)
ValueError: Couldn't parse datetime string: '173f8bb786b42fc9615c79013add5538'
2021-01-29 04:58:33 WARNING (MainThread) [homeassistant.components.webhook] Received message for unregistered webhook a0704c90b6076be2ad174faa108829ac61e863376b07807192f9d69b70b3180f from 192.168.20.105
2021-01-29 08:05:44 WARNING (MainThread) [homeassistant.helpers.service] Unable to find referenced entities switch.steckdose5
2021-01-29 08:15:48 ERROR (MainThread) [pydeconz.websocket] Client connection error
2021-01-29 08:16:40 WARNING (zeroconf-ServiceBrowser__http._tcp.local.-_googlecast._tcp.local.-_elg._tcp.local.-_Volumio._tcp.local.-_esphomelib._tcp.local.-_dkapi._tcp.local.-_ipp._tcp.local.-_plugwise._tcp.local.-_bond._tcp.local.-_ssh._tcp.local.-_viziocast._tcp.local.-_touch-able._tcp.local.-_daap._tcp.local.-_axis-video._tcp.local.-_nut._tcp.local.-_homekit._tcp.local.-_mediaremotetv._tcp.local.-_spotify-connect._tcp.local.-_wled._tcp.local.-_ipps._tcp.local.-_hap._tcp.local.-_xbmc-jsonrpc-h._tcp.local.-_api._udp.local.-_printer._tcp.local.-_miio._udp.local._287) [zeroconf] Error sending through socket 27
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/zeroconf/__init__.py", line 2952, in send
    bytes_sent = s.sendto(packet, 0, (real_addr, port))
OSError: [Errno 101] Network unreachable

This is how it looks like in a graph. Last data at 04:07, new data at 08:22 after restart:

These are the last lines of the logbook before restart, no entries between 04:11:14 and restart at 08:22

This happens every day, sometimes twice, since the last update to 2021.1.5. Before it was really stable without any issues.

Any ideas where to look at the root cause?

Best regards
Stefan

Why is recorder handling this string as a date? Could it be a value received by a sensor or set to an input_*, that is designed for dates only?

don’t know. What is the best way to identify this?
I did not change my devices connected to HA, no new ones.

Looking at the stack trace, I think this bug is related to the “purge” function of the recorder. Could be caused by an invalid date that made it into the database.

If you don’t rely on state history, I recommend deleting the database (usually file home-assistant_v2.db) and restart HA.

I’ve tried to delete my db, but it did not help.
As I switched back to the original one, I can see no data flow from deCONZ where all my aqara sensors are. Seems there is an issue with the data flow inside this integration.
deCONZ was also updated a few days ago.

Update and closure from my side: After a deletion of the db and several days of monitoring no new stop occured. It seems, the db was corrupt.