Home Assistant Keeps Crashing / Restarting Often

Jon123 · September 29, 2022, 5:45pm

I’ve been running HA in this VM for a year or more at this point without much issue.

The host here is Ubuntu 20.04.4.
VirtualBox 6.1.34.
Guest is the HA OVA release, updated to 2022.9.7.

The other night the host crashed (I have not figured out why yet, it has never happened before). After that HA was acting strange. I had originally posted over here.

But since then I’ve updated to 2022.9.7, and the system is - somewhat stable, so I think it makes sense to move this to a different troubleshooting thread…

HA (core) is restarting often though. Last night for example it restarted about 10 times in an hour. But as of right now it’s been running for about 5 hours without restarting (no changes made to lead to that).

I looked in to a few related issues like this: Loop restarting · Issue #74606 · home-assistant/core · GitHub, but that seems to be an OOM issue, and I’m not seeing any OOM kills in the logs.

I have the home-assistant.log.fault file. This shows errors of type “Fatal Python error: Segmentation fault” and also some “Fatal Python error: Bus error”. Unforutnately these don’t have time stamps, but I have seen the file is updated each time the restart occurs (if I manually check it at the right time).

The most recent Seg Fault Log

Fatal Python error: Segmentation fault

Thread 0x00007f09eedffb30 (most recent call first):
File “/usr/local/lib/python3.10/concurrent/futures/thread.py”, line 81 in _worker
File “/usr/local/lib/python3.10/threading.py”, line 953 in run
File “/usr/local/lib/python3.10/threading.py”, line 1016 in _bootstrap_inner
File “/usr/local/lib/python3.10/threading.py”, line 973 in _bootstrap

Thread 0x00007f09ef3ffb30 (most recent call first):
File “/usr/local/lib/python3.10/concurrent/futures/thread.py”, line 81 in _worker
File “/usr/local/lib/python3.10/threading.py”, line 953 in run
File “/usr/local/lib/python3.10/threading.py”, line 1016 in _bootstrap_inner
File “/usr/local/lib/python3.10/threading.py”, line 973 in _bootstrap

Thread 0x00007f09ef9ffb30 (most recent call first):
File “/usr/local/lib/python3.10/concurrent/futures/thread.py”, line 81 in _worker
File “/usr/local/lib/python3.10/threading.py”, line 953 in run
File “/usr/local/lib/python3.10/threading.py”, line 1016 in _bootstrap_inner
File “/usr/local/lib/python3.10/threading.py”, line 973 in _bootstrap

Thread 0x00007f09f0ffbb30 (most recent call first):
File “/usr/local/lib/python3.10/concurrent/futures/thread.py”, line 81 in _worker
File “/usr/local/lib/python3.10/threading.py”, line 953 in run
File “/usr/local/lib/python3.10/threading.py”, line 1016 in _bootstrap_inner
File “/usr/local/lib/python3.10/threading.py”, line 973 in _bootstrap

Thread 0x00007f09f10ffb30 (most recent call first):
File “/usr/local/lib/python3.10/concurrent/futures/thread.py”, line 81 in _worker
File “/usr/local/lib/python3.10/threading.py”, line 953 in run
File “/usr/local/lib/python3.10/threading.py”, line 1016 in _bootstrap_inner
File “/usr/local/lib/python3.10/threading.py”, line 973 in _bootstrap

Thread 0x00007f09f19ffb30 (most recent call first):
File “/usr/local/lib/python3.10/concurrent/futures/thread.py”, line 81 in _worker
File “/usr/local/lib/python3.10/threading.py”, line 953 in run
File “/usr/local/lib/python3.10/threading.py”, line 1016 in _bootstrap_inner
File “/usr/local/lib/python3.10/threading.py”, line 973 in _bootstrap

Thread 0x00007f09f1fd4b30 (most recent call first):
File “/usr/local/lib/python3.10/logging/handlers.py”, line 1491 in dequeue
File “/usr/local/lib/python3.10/logging/handlers.py”, line 1542 in _monitor
File “/usr/local/lib/python3.10/threading.py”, line 953 in run
File “/usr/local/lib/python3.10/threading.py”, line 1016 in _bootstrap_inner
File “/usr/local/lib/python3.10/threading.py”, line 973 in _bootstrap

Current thread 0x00007f09f607bab0 (most recent call first):
File “/usr/local/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py”, line 356 in _decorate_cls_with_warning
File “/usr/local/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py”, line 110 in decorate
File “/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/interfaces.py”, line 1474 in
File “”, line 241 in _call_with_frames_removed
File “”, line 883 in exec_module
File “”, line 688 in _load_unlocked
File “”, line 1006 in _find_and_load_unlocked
File “”, line 1027 in _find_and_load
File “/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py”, line 12 in
File “”, line 241 in _call_with_frames_removed
File “”, line 883 in exec_module
File “”, line 688 in _load_unlocked
File “”, line 1006 in _find_and_load_unlocked
File “”, line 1027 in _find_and_load
File “/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/events.py”, line 9 in
File “”, line 241 in _call_with_frames_removed
File “”, line 883 in exec_module
File “”, line 688 in _load_unlocked
File “”, line 1006 in _find_and_load_unlocked
File “”, line 1027 in _find_and_load
File “”, line 241 in _call_with_frames_removed
File “”, line 1078 in _handle_fromlist
File “/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/init.py”, line 18 in
File “”, line 241 in _call_with_frames_removed
File “”, line 883 in exec_module
File “”, line 688 in _load_unlocked
File “”, line 1006 in _find_and_load_unlocked
File “”, line 1027 in _find_and_load
File “/usr/local/lib/python3.10/site-packages/sqlalchemy/init.py”, line 9 in
File “”, line 241 in _call_with_frames_removed
File “”, line 883 in exec_module
File “”, line 688 in _load_unlocked
File “”, line 1006 in _find_and_load_unlocked
File “”, line 1027 in _find_and_load
File “/usr/src/homeassistant/homeassistant/components/recorder/statistics.py”, line 17 in
File “”, line 241 in _call_with_frames_removed
File “”, line 883 in exec_module
File “”, line 688 in _load_unlocked
File “”, line 1006 in _find_and_load_unlocked
File “”, line 1027 in _find_and_load
File “”, line 241 in _call_with_frames_removed
File “”, line 1078 in _handle_fromlist
File “/usr/src/homeassistant/homeassistant/components/recorder/init.py”, line 23 in
File “”, line 241 in _call_with_frames_removed
File “”, line 883 in exec_module
File “”, line 688 in _load_unlocked
File “”, line 1006 in _find_and_load_unlocked
File “”, line 1027 in _find_and_load
File “”, line 241 in _call_with_frames_removed
File “”, line 1078 in _handle_fromlist
File “/usr/src/homeassistant/homeassistant/components/energy/websocket_api.py”, line 15 in
File “”, line 241 in _call_with_frames_removed
File “”, line 883 in exec_module
File “”, line 688 in _load_unlocked
File “”, line 1006 in _find_and_load_unlocked
File “”, line 1027 in _find_and_load
File “”, line 241 in _call_with_frames_removed
File “”, line 1078 in _handle_fromlist
File “/usr/src/homeassistant/homeassistant/components/energy/init.py”, line 10 in
File “”, line 241 in _call_with_frames_removed
File “”, line 883 in exec_module
File “”, line 688 in _load_unlocked
File “”, line 1006 in _find_and_load_unlocked
File “”, line 1027 in _find_and_load
File “/usr/src/homeassistant/homeassistant/components/analytics/analytics.py”, line 12 in
File “”, line 241 in _call_with_frames_removed
File “”, line 883 in exec_module
File “”, line 688 in _load_unlocked
File “”, line 1006 in _find_and_load_unlocked
File “”, line 1027 in _find_and_load
File “/usr/src/homeassistant/homeassistant/components/analytics/init.py”, line 10 in
File “”, line 241 in _call_with_frames_removed
File “”, line 883 in exec_module
File “”, line 688 in _load_unlocked
File “”, line 1006 in _find_and_load_unlocked
File “”, line 1027 in _find_and_load
File “”, line 1050 in _gcd_import
File “/usr/local/lib/python3.10/importlib/init.py”, line 126 in import_module
File “/usr/src/homeassistant/homeassistant/loader.py”, line 662 in get_component
File “/usr/src/homeassistant/homeassistant/setup.py”, line 190 in _async_setup_component
File “/usr/local/lib/python3.10/asyncio/events.py”, line 80 in _run
File “/usr/local/lib/python3.10/asyncio/base_events.py”, line 1896 in _run_once
File “/usr/local/lib/python3.10/asyncio/base_events.py”, line 600 in run_forever
File “/usr/local/lib/python3.10/asyncio/base_events.py”, line 633 in run_until_complete
File “/usr/src/homeassistant/homeassistant/runner.py”, line 119 in run
File “/usr/src/homeassistant/homeassistant/main.py”, line 191 in main
File “/usr/src/homeassistant/homeassistant/main.py”, line 203 in
File “/usr/local/lib/python3.10/runpy.py”, line 86 in _run_code
File “/usr/local/lib/python3.10/runpy.py”, line 196 in _run_module_as_main

Extension modules: _cffi_backend, multidict._multidict, yarl._quoting_c, ciso8601, aiohttp._helpers, aiohttp._http_writer, _brotli, aiohttp._http_parser, aiohttp._websocket, cchardet._cchardet, frozenlist._frozenlist, yaml._yaml, markupsafe._speedups, PIL._imaging, simplejson._speedups, sqlalchemy.cimmutabledict, greenlet._greenlet, sqlalchemy.cprocessors (total: 18)

I’m happy to look up other logs. I have the VM set up with SSH access to the root OS if needed. I’ve been using journalctl to look at some logs, but haven’t been able to figure out what is causing this.

Jon123 · September 29, 2022, 10:48pm

So after a good run (11 hours), this afternoon it’s been crash-tastic, having restarted about 8 times in the last 3 hours.

Jon123 · September 30, 2022, 1:48pm

Last night was rough. At some point HA crashed but didn’t come back up.
I had to force reboot the VM.
And now HA is in some state of broken again… Supervisor isn’t loading

SIGSEGV somewhere

Sep 30 10:38:20 homeassistant dockerd[394]: panic: runtime error: invalid memory address or nil pointer dereference
Sep 30 10:38:20 homeassistant dockerd[394]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x22c8a4d]
Sep 30 10:38:20 homeassistant dockerd[394]: goroutine 694 [running]:
Sep 30 10:38:20 homeassistant dockerd[394]: github.com/docker/docker/daemon/logger/journald.(*journald).Log(0xc0017d25b8, 0xc00021e000)
Sep 30 10:38:20 homeassistant dockerd[394]: github.com/docker/docker/daemon/logger/journald/journald.go:106 +0x10d
Sep 30 10:38:20 homeassistant dockerd[394]: github.com/docker/docker/daemon/logger.(*Copier).copySrc(0xc000efc9c0, {0x2b9bf78, 0x6}, {0x7fe6f9611e90, 0xc000efc940})
Sep 30 10:38:20 homeassistant dockerd[394]: github.com/docker/docker/daemon/logger/copier.go:128 +0xf0d
Sep 30 10:38:20 homeassistant dockerd[394]: created by github.com/docker/docker/daemon/logger.(*Copier).Run
Sep 30 10:38:20 homeassistant dockerd[394]: github.com/docker/docker/daemon/logger/copier.go:48 +0x71

eightiescalling · September 30, 2022, 2:29pm

How many HACS type add-ons or third party integrations are installed?

The only time I’ve had anything like this it was tracked down to a problematic integration. Once that was disabled (removed in the end as an alternative was available) then HA was stable again.

I’d suggest going through any plug-ins/integrations - whether that be remove all, check sanity then build up again or trying to cut them back a bit and see if you catch the problem child.

Jon123 · September 30, 2022, 3:13pm

Thanks for responding!

In HACS I only have 3 integrations, and none of them were recently added/updated. I also checked the logs and I don’t see any indication that it’s caused by any of these integrations.

I’ll try disabling them all temporarily to see if that helps anything.

My suspicion here is something is corrupted somewhere - but I can’t figure out from the logs where/what it could be.

Jon123 · September 30, 2022, 7:22pm

Disabling those HACS integrations didn’t help, unfortunately.

I did end up finding here a suggestion to try “ha core rebuild”. This seems to have brought HA back up. Now it’s a matter of whether it keeps having the crashing issue.

Jon123 · October 1, 2022, 2:25am

Unfortunately I’m still seeing constant crashes(about 10 just in the last 4 hours).

One of the times, the OS went down as well: I couldn’t SSH in, and at the console I couldn’t do anything. I sent the ACPI shutdown signal but nothing happened so I had to reset the VM.
On the that reboot I noticed HA reverted to OS 8.5 (must be some form of safe boot) - but this also hasn’t helped.

Jon123 · October 3, 2022, 3:41pm

I posted a quick update on the issue on github. It hasn’t been quite long enough to call this solved, but updating Virtualbox from 6.1.32 to 6.1.38 has resulted in HA running for the last 20 hours. This could be a fluke, but wanted to add it here in case someone finds it in the future and I forget to come back to post another update.

UPDATE: So, it’s been 2.5 days now without a crash. I think this is solved

Powlus · November 15, 2022, 12:53pm

I created an account just to thank you for this post, I have been having the exact same issues on the exact same VBox version and have just updated. Will confirm in a few days if all is stable

Jon123 · November 15, 2022, 7:53pm

I hope it solves your problem, it was driving me crazy.

cnschulz · January 26, 2023, 3:42am

Im going mad. This is happening for me and I’m on v7 of virtualbox :\ I moved from a Linux (ubuntu) environment to Windows and did not modify the VM at all. Crashes at least once a day. I have rebuilt the entire VM (still using my HA backup) to no avail. Any help appreciated.

Jon123 · January 26, 2023, 9:09pm

Unfortunately, I can’t help much. I don’t run Windows at all, and I’m still running Virtual Box 6.1.38 on linux.

Maybe try v6 of VirtualBox to see if it helps?

I think you’re better off creating a separate topic to share your details similar to how I did in my original post. It could be something completely different.

Also check GitHub issues… I didn’t read this one, but: Home assistant 7.2 not working in Virtual box · Issue #1737 · home-assistant/operating-system · GitHub