After some recent change/event, Home Assistant on my rPi3 native was delaying iOS notifications, slow to execute automations, and slow to respond to UI requests. top says that the hass process is pegged to 100%.
Any easy way to profile/debug hass to determine if it’s a native or HACS integration (or some config issue)?
First thing is to check the logs.
I had the same a couple of days ago, and the log allowed me to find out that my Harmony hub was not behaving, and HA kept stumbling auto-discoverying it through SSDP. Just disabled the ssdp integration, that I didn’t actually need.
I have logs on debug and I see a lot of verbosity from zeroconf (with seemingly no way to disable it in the YAML config) but nothing indicates that’s it.
What Linux Python realtime profilers can I attach to the process to get some spindump data? Anything else I can try besides turning off the integrations one by one?
Maybe this can help
Got py-spy to profile hass. Disabled all integrations. Not sure how to make sense of why it’s pegging one whole core:
Collecting samples from '/usr/local/bin/python3.8 /home/pi/.local/bin/hass' (python v3.8.5)
Total Samples 13300
GIL: 0.00%, Active: 1000.00%, Threads: 12
%Own %Total OwnTime TotalTime Function (filename:line)
700.00% 700.00% 864.1s 864.1s _worker (concurrent/futures/thread.py:78)
200.00% 200.00% 256.2s 256.2s dequeue (logging/handlers.py:1427)
100.00% 100.00% 103.3s 103.3s do_handshake (ssl.py:944)
0.00% 0.00% 4.19s 4.19s read (ssl.py:1099)
0.00% 0.00% 2.40s 2.40s _compile_bytecode (<frozen importlib._bootstrap_external>:580)
0.00% 0.00% 1.24s 5.43s readinto (socket.py:669)
0.00% 0.00% 0.770s 0.770s _path_stat (<frozen importlib._bootstrap_external>:87)
0.00% 0.00% 0.690s 0.690s _build_ssl_context (httplib2/__init__.py:169)
0.00% 0.00% 0.680s 0.850s getmodule (inspect.py:745)
0.00% 0.00% 0.590s 0.590s _write_to_self (asyncio/selector_events.py:138)
0.00% 0.00% 0.530s 0.530s get_data (<frozen importlib._bootstrap_external>:972)
0.00% 0.00% 0.480s 0.690s parseImpl (pkg_resources/_vendor/pyparsing.py:3400)
0.00% 0.00% 0.440s 0.440s get_data (<frozen importlib._bootstrap_external>:973)
0.00% 0.00% 0.390s 0.390s raw_decode (json/decoder.py:353)
0.00% 0.00% 0.380s 0.380s read (gzip.py:487)
0.00% 0.00% 0.330s 0.330s _create_fn (dataclasses.py:398)
0.00% 0.00% 0.300s 0.300s _add_read_data (gzip.py:506)
0.00% 0.00% 0.280s 0.280s getmodule (inspect.py:744)
0.00% 0.00% 0.270s 0.270s do_handshake (ssl.py:1309)
0.00% 0.00% 0.250s 0.270s _exec_code_in_env (sqlalchemy/util/langhelpers.py:228)
0.00% 0.00% 0.230s 0.230s getaddrinfo (socket.py:918)
0.00% 0.00% 0.220s 0.220s islink (posixpath.py:167)
0.00% 0.00% 0.210s 0.210s __new__ (abc.py:85)
0.00% 0.00% 0.180s 0.180s getmodule (inspect.py:747)
0.00% 0.00% 0.180s 0.180s _fill_cache (<frozen importlib._bootstrap_external>:1483)
0.00% 0.00% 0.170s 0.180s __init__ (pkg_resources/_vendor/packaging/version.py:207)
0.00% 0.00% 0.160s 0.160s ismodule (inspect.py:70)
0.00% 0.00% 0.150s 0.150s _real_close (socket.py:494)
0.00% 0.00% 0.150s 0.150s _compile_dict (voluptuous/schema_builder.py:550)
0.00% 0.00% 0.140s 0.140s isdir (genericpath.py:42)
After disabling all integrations, it ended up being native SSL support. I disabled the below and CPU was normal. WTF?
http:
ssl_certificate: /home/pi/.homeassistant/dehydrated/certs/$domain/fullchain.pem
ssl_key: /home/pi/.homeassistant/dehydrated/certs/$domain/privkey.pem
trusted_proxies:
- 192.168.0.0/24
trusted_networks: 192.168.86.0/24
I was planning to switch to an nginx proxy anyway but I wonder if anyone else has hit this bug.