Since last upgrade, I have been having issues with HA crashing every day and discovered it’s due to log file growing until no disk space left.
After reboot it will run fine for an hour or so then logs go crazy with following error repeating multiple times every 0.001 sec
~~~~~~~~~~~~~~~~~~~^^
TypeError: 'NoneType' object is not callable
2025-03-14 16:20:22.009 ERROR (MainThread) [homeassistant] Error doing job: Exception in callback _SelectorSocketTransport._read_ready() (None)
Traceback (most recent call last):
File "/usr/local/lib/python3.13/asyncio/events.py", line 89, in _run
self._context.run(self._callback, *self._args)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/asyncio/selector_events.py", line 962, in _read_ready
self._read_ready_cb()
~~~~~~~~~~~~~~~~~~~^^
TypeError: 'NoneType' object is not callable
2025-03-14 16:20:22.009 ERROR (MainThread) [homeassistant] Error doing job: Exception in callback _SelectorSocketTransport._read_ready() (None)
Traceback (most recent call last):
File "/usr/local/lib/python3.13/asyncio/events.py", line 89, in _run
self._context.run(self._callback, *self._args)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/asyncio/selector_events.py", line 962, in _read_ready
Can’t remember the previous version but it was around 6 months.
The error doesn’t really give me any indication of the failing config. does anyone have an idea what may be causing this ?
i have changed logging default to critical to solve this but not ideal. I will look to rebuild later as i can’t find anything that seems likely culprit. i have updated all other components. think I need to start from scratch.
It’s tricky to diagnose as it takes a few hours to start the errors after reboot. Then log was growing to 60GB in 24hrs and crashing soon after once it used all space.
Would be good if there was a rolling log option or log file size limit to prevent system crash, but also seems excessive that anything can spam the logs so fast. some sort of limiter or compressing/summerising repeating entries might help.
after checking out this thread
https://community.home-assistant.io/t/2024-5-tracking-down-instability-issues-caused-by-integrations/724441
i went down the “runaway template” option as i have quite a few and didn’t see any memory issues.
with some with renders of 59000 (starting at 4 on fresh boot), i assume this is runaway and in the high catagory and therefore likely cause.
seems weird as these have been working great for years and i have not changed any. but in interest of stablility i guess i can live without templates
2025-03-16 09:11:16.941 CRITICAL (SyncWorker_11) [homeassistant.components.profiler] RenderInfo object in memory: <RenderInfo Template<template=({{ '%0.1f' | format(states('sensor.main_power') | float(0) - states('sensor.load_power_tumble_dryer') | float(0) - states('sensor.load_power_comms_cabinet') | float(0) - states('sensor.coffee_machine_power') | float(0) - states('sensor.load_power_office_workstation') | float(0) - states('sensor.power_cooker_power') | float(0) - states('sensor.livingroom_tv_cabinet_power') | float(0) - states('sensor.office_ac_power') | float(0) - states('sensor.living_room_ac_power') | float(0) - states('sensor.washing_machine_power') | float(0) - states('sensor.load_power_fridge') | float(0))}}) renders=59400> all_states=False all_states_lifecycle=False domains=frozenset() domains_lifecycle=frozenset() entities=frozenset({'sensor.coffee_machine_power', 'sensor.load_power_fridge', 'sensor.washing_machine_power', 'sensor.load_power_tumble_dryer', 'sensor.load_power_office_workstation', 'sensor.main_power', 'sensor.load_power_comms_cabinet', 'sensor.power_cooker_power', 'sensor.livingroom_tv_cabinet_power', 'sensor.living_room_ac_power', 'sensor.office_ac_power'}) rate_limit=None has_time=False exception=None is_static=False>
Hello digidog,
What kind of hard drive are you using? Is it healthy?
Thanks SG
It runs as a VM on ESXi with Samsung 1TB NVMe SSD
There are a few other vm’s on there, but still only around half the SSD in use.
I don’t think there are any disk issues. Nothing showing errors and everything running fine. it would be big job to remove for scan but i will look into this when i rebuild.
It wasn’t templates as the issue came back after i disabled them. I’m leaning towards one of the HACS components being likely cause. System has been running for 4 years and taken a bit of abuse with various experiments, my guess is one is maybe not 100% compatible, i’ve tried removing as many as i can for now.
It has been rock solid till last update but has now returned to being stable, Once i excluded homeassistant from errors, the log is now at 2MB total after 24 hours. CPU is steady and all is working well.
Whilst i realise hiding the logs is not a cure (and hides other potential logs), it will help until i can rebuild and start from scratch. whatever is producing the error is not taxing the CPU or memory in this unlogged state.
The cause of the crashing and high CPU was the logger, just overloaded with the error until it eventually crashes the system once all disk is used ~ 70GB. (in approx 24Hrs)