Ha crashing regular but irregularly

Killerm · November 22, 2021, 9:19am

Hi,
as title says my HA is crashing regular but irregularly (randomly maybe once in week or two). This mean that I’m able to connect to my instance and almost all sensor are " Unavailable" and cannot use add-ons, use automations or even boot ha from gui(need to power it off from cable). Lights and ac are still working thought HA.

Went trough logs are they look like this. I have Pi 4, 8GB where HA are installed on ssd and I’m using MariaDB.

First erro
r

After this most of integrations and devices report errors:

Few more database errors

Logger: homeassistant.components.recorder
Source: components/recorder/__init__.py:969
Integration: Recorder (documentation, issues)
First occurred: 10:00:06 AM (28 occurrences)
Last logged: 11:05:16 AM

Unhandled database error while processing event <Event time_changed[L]: now=2021-11-22T10:10:46.001285+02:00>: (MySQLdb._exceptions.OperationalError) (2005, "Unknown MySQL server host 'core-mariadb' (-3)") (Background on this error at: https://sqlalche.me/e/14/e3q8)
Unhandled database error while processing event <Event time_changed[L]: now=2021-11-22T10:11:16.001599+02:00>: (MySQLdb._exceptions.OperationalError) (2005, "Unknown MySQL server host 'core-mariadb' (-3)") (Background on this error at: https://sqlalche.me/e/14/e3q8)
Unhandled database error while processing event <Event time_changed[L]: now=2021-11-22T10:11:46.001014+02:00>: (MySQLdb._exceptions.OperationalError) (2005, "Unknown MySQL server host 'core-mariadb' (-3)") (Background on this error at: https://sqlalche.me/e/14/e3q8)
Unhandled database error while processing event <Event time_changed[L]: now=2021-11-22T10:12:16.002080+02:00>: (MySQLdb._exceptions.OperationalError) (2005, "Unknown MySQL server host 'core-mariadb' (-3)") (Background on this error at: https://sqlalche.me/e/14/e3q8)
Unhandled database error while processing event <Event time_changed[L]: now=2021-11-22T10:12:46.002538+02:00>: (MySQLdb._exceptions.OperationalError) (2005, "Unknown MySQL server host 'core-mariadb' (-3)") (Background on this error at: https://sqlalche.me/e/14/e3q8)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 307, in connect
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 430, in checkout
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 427, in checkout
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 580, in get_connection
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 611, in __connect
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 605, in __connect
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 578, in connect
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 584, in connect
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/__init__.py", line 130, in Connect
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/connections.py", line 185, in __init__
MySQLdb._exceptions.OperationalError: (2002, "Can't connect to MySQL server on 'core-mariadb' (115)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/recorder/__init__.py", line 697, in _process_one_event_or_recover
  File "/usr/src/homeassistant/homeassistant/components/recorder/__init__.py", line 830, in _process_one_event
  File "/usr/src/homeassistant/homeassistant/components/recorder/__init__.py", line 969, in _send_keep_alive
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1519, in connection
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1529, in _connection_for_bind
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 747, in _connection_for_bind
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3166, in connect
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3245, in raw_connection
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3215, in _wrap_pool_connect
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2069, in _handle_dbapi_exception_noconnection
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3212, in _wrap_pool_connect
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 307, in connect
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 767, in _checkout
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 430, in checkout
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 427, in checkout
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 580, in get_connection
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 611, in __connect
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 605, in __connect
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/create.py", line 578, in connect
  File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 584, in connect
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/__init__.py", line 130, in Connect
  File "/usr/local/lib/python3.9/site-packages/MySQLdb/connections.py", line 185, in __init__
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (2002, "Can't connect to MySQL server on 'core-mariadb' (115)")
(Background on this error at: https://sqlalche.me/e/14/e3q8)

Seems some kinda database errors but do anybody have knowledge what is going on and how to fix this?

WallyR · November 22, 2021, 12:31pm

Some of these errors seems to be network related, like “cannot connect to host api.githib.com”.
Maybe all of all or most of them are.

Check your internet conncetion by doing a constant ping to a known good site, like google at 8.8.8.8. In windows this is done with ping -t in Linux its just ping. You can direct the output to a file by adding > c:\pingtest.txt on Windows, so you can run it for several days and go back and look at it.

Pinging 8.8.8.8 shows you the entire route to Google, but not where the error is.
Pinging you routers internal IP shows you if its the router that restarts.
Pinging first hop after your router shows you if its your line. (use tracert 8.8.8.8 on Windows to see hops and traceroute 8.8.8.8 on Linux).
Try pinging your public IP also. (look it up at myip.com)

Killerm · November 22, 2021, 6:16pm

I think network related issues came after initial issues because I’m am able to ping google and my public ip without issues. So wont think networking causes ha to crash

WallyR · November 23, 2021, 12:50am

Ping is just a picture of the moment.
You need to keep the ping running continuously to get a broader spectrum and be able to hit the times where you are actually having issues.

And yes network issues can send HA in a spin if many sensors are using external DNS or external servers.
If its your router that is rebooting at times and this router is a switch too, then the network might get broken up more from the reboot.

Killerm · November 24, 2021, 7:12am

At the time ha started to crash I was working from home office and were at teams meeting and could not even see one second break in connection.

I will try to put ping request 24/7 and save them to txt file.

turbokongen · October 27, 2022, 9:21am

Did you find a solution to this?
I have the exact same issue, though using the built in sql database.
If I had checked “Keep me logged in”, I am able to access the instance over internet, but all internals are unavailable, add ons give 502 error. Database is unavailable, and the first error is corrupt database.
Roughly 20 hours later the instance become unreachable.
This indicate that the network is ok.

Edit:
I cannot SSH into homeassistant instance or host. (I have set up port 22222 for that)