Debugging hass process 100% CPU

After some recent change/event, Home Assistant on my rPi3 native was delaying iOS notifications, slow to execute automations, and slow to respond to UI requests. top says that the hass process is pegged to 100%.

Any easy way to profile/debug hass to determine if it’s a native or HACS integration (or some config issue)?

First thing is to check the logs.

I had the same a couple of days ago, and the log allowed me to find out that my Harmony hub was not behaving, and HA kept stumbling auto-discoverying it through SSDP. Just disabled the ssdp integration, that I didn’t actually need.

I have logs on debug and I see a lot of verbosity from zeroconf (with seemingly no way to disable it in the YAML config) but nothing indicates that’s it.

What Linux Python realtime profilers can I attach to the process to get some spindump data? Anything else I can try besides turning off the integrations one by one?

Maybe this can help

Got py-spy to profile hass. Disabled all integrations. Not sure how to make sense of why it’s pegging one whole core:

Collecting samples from '/usr/local/bin/python3.8 /home/pi/.local/bin/hass' (python v3.8.5)
Total Samples 13300
GIL: 0.00%, Active: 1000.00%, Threads: 12

  %Own   %Total  OwnTime  TotalTime  Function (filename:line)
700.00% 700.00%   864.1s    864.1s   _worker (concurrent/futures/
200.00% 200.00%   256.2s    256.2s   dequeue (logging/
100.00% 100.00%   103.3s    103.3s   do_handshake (
  0.00%   0.00%    4.19s     4.19s   read (
  0.00%   0.00%    2.40s     2.40s   _compile_bytecode (<frozen importlib._bootstrap_external>:580)
  0.00%   0.00%    1.24s     5.43s   readinto (
  0.00%   0.00%   0.770s    0.770s   _path_stat (<frozen importlib._bootstrap_external>:87)
  0.00%   0.00%   0.690s    0.690s   _build_ssl_context (httplib2/
  0.00%   0.00%   0.680s    0.850s   getmodule (
  0.00%   0.00%   0.590s    0.590s   _write_to_self (asyncio/
  0.00%   0.00%   0.530s    0.530s   get_data (<frozen importlib._bootstrap_external>:972)
  0.00%   0.00%   0.480s    0.690s   parseImpl (pkg_resources/_vendor/
  0.00%   0.00%   0.440s    0.440s   get_data (<frozen importlib._bootstrap_external>:973)
  0.00%   0.00%   0.390s    0.390s   raw_decode (json/
  0.00%   0.00%   0.380s    0.380s   read (
  0.00%   0.00%   0.330s    0.330s   _create_fn (
  0.00%   0.00%   0.300s    0.300s   _add_read_data (
  0.00%   0.00%   0.280s    0.280s   getmodule (
  0.00%   0.00%   0.270s    0.270s   do_handshake (
  0.00%   0.00%   0.250s    0.270s   _exec_code_in_env (sqlalchemy/util/
  0.00%   0.00%   0.230s    0.230s   getaddrinfo (
  0.00%   0.00%   0.220s    0.220s   islink (
  0.00%   0.00%   0.210s    0.210s   __new__ (
  0.00%   0.00%   0.180s    0.180s   getmodule (
  0.00%   0.00%   0.180s    0.180s   _fill_cache (<frozen importlib._bootstrap_external>:1483)
  0.00%   0.00%   0.170s    0.180s   __init__ (pkg_resources/_vendor/packaging/
  0.00%   0.00%   0.160s    0.160s   ismodule (
  0.00%   0.00%   0.150s    0.150s   _real_close (
  0.00%   0.00%   0.150s    0.150s   _compile_dict (voluptuous/
  0.00%   0.00%   0.140s    0.140s   isdir (

After disabling all integrations, it ended up being native SSL support. I disabled the below and CPU was normal. WTF?

  ssl_certificate: /home/pi/.homeassistant/dehydrated/certs/$domain/fullchain.pem
  ssl_key: /home/pi/.homeassistant/dehydrated/certs/$domain/privkey.pem

I was planning to switch to an nginx proxy anyway but I wonder if anyone else has hit this bug.