High CPU usage after 0.113

Configuration
20

image
image1728×500 38.8 KB

Here is my py-spy recording, hopefully it helps.

21

Thank you for the py-spy recording. Would you please post the full .svg file as the detail is lost when in the screenshot.

22
23

Thank you for that. Would you please post a py-spy dump as well (After the cpu has crept up)

24

I noticed my HA instance having higher CPU usage lately too, restarting just HA seems to turn everything back to normal again and the drop is pretty obvious:

image
image505×502 19.3 KB

Next monday I’ll be home and will install these debugging tools!

25

If you are seeing the cpu creeping up issue, do you have homekit_controller enabled? It looks like there may be another code path that does not use the shared zeroconf instance that needs to be converted.

We thought we got them all in https://github.com/home-assistant/core/pull/37691 but it appears something still needs to be found.

It may be the same issue here Python3 high CPU Usage

I’m collecting details in https://github.com/home-assistant/core/issues/38668

26

If it is any use this is my list of installed integrations:

In the UI:

  • certificate expiry
  • esphome
  • google cast
  • hacs
  • home assistant ios
  • lifx
  • mobile app
  • mqtt
  • panasonic viera
  • simpleicons
  • speedtest.net

In HACS:

  • hacs
  • bwalarm (ak47 edition)
  • sun2
  • bom forecast
  • garbage collection
  • simpleicons
27

Well, I don’t have a homekit installed but it is showing up in the logs as being loaded. Maybe it loads if it discovers anything Apple on the network?

Log line is: Setting up stage 2: {‘weather’, ‘group’, ‘sonos’, ‘input_boolean’, ‘tts’, ‘panel_custom’, ‘somfy_mylink’, ‘ssdp’, ‘zeroconf’, ‘cast’, ‘discovery’, ‘cover’, ‘roku’, ‘vesync’, ‘notion’, ‘script’, ‘synology_dsm’, ‘zwave’, ‘influxdb’, ‘stream’, ‘conversation’, ‘unifiprotect’, ‘media_player’, ‘rachio’, ‘hacs’, ‘sun’, ‘system_health’, ‘mqtt’, ‘persistent_notification’, ‘switch’, ‘logbook’, ‘homekit_controller’, ‘unifi’, ‘nest’, ‘history’, ‘ipp’, ‘sensor’, ‘automation’, ‘light’, ‘binary_sensor’, ‘tplink’, ‘updater’}

The ones in Integration UI are: vesync, google cast, HACS, MQTT, Nest, Notion, Roku, Sonos, Tp-Link Kasa Smart, Ubiquiti Unifi, Ubiquiti Protect, ZWave.

In addition to those, there’s a garadget and darksky module loaded through config.

28

I’ve sorted out the issue with homekit_controller (details -> https://github.com/home-assistant/core/issues/38668)

Please keep posting the py-spy record (svg) and dump (text) output as there is probably another integration that needs to be fixed.

29

Here you go. Gave it a few days for average CPU to steadily climb:

Thread 0x5608F5D338B0 (idle): "MainThread"
    select (selectors.py:468)
    _run_once (asyncio/base_events.py:1823)
    run_forever (asyncio/base_events.py:570)
    run_until_complete (asyncio/base_events.py:603)
    run (asyncio/runners.py:43)
    run (homeassistant/runner.py:119)
    main (homeassistant/__main__.py:312)
    <module> (homeassistant/__main__.py:320)
    _run_code (runpy.py:87)
    _run_module_as_main (runpy.py:194)
Thread 247 (idle): "Thread-1"
    dequeue (logging/handlers.py:1427)
    _monitor (logging/handlers.py:1478)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 248 (idle): "SyncWorker_0"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 251 (idle): "SyncWorker_1"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 252 (idle): "SyncWorker_2"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 253 (idle): "SyncWorker_3"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 254 (idle): "SyncWorker_4"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 255 (idle): "SyncWorker_5"
    wait (threading.py:306)
    wait (zeroconf/__init__.py:2326)
    request (aioesphomeapi/host_resolver.py:44)
    resolve_host (aioesphomeapi/host_resolver.py:64)
    run (concurrent/futures/thread.py:57)
    _worker (concurrent/futures/thread.py:80)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 256 (idle): "Recorder"
    wait (threading.py:302)
    get (queue.py:170)
    run (homeassistant/components/recorder/__init__.py:340)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 0x7F334B954B20 (active): "zeroconf-Engine-257"
    update_record (zeroconf/__init__.py:1608)
    update_record (netdisco/mdns.py:21)
    update_record (zeroconf/__init__.py:2588)
    handle_response (zeroconf/__init__.py:2628)
    handle_read (zeroconf/__init__.py:1400)
    run (zeroconf/__init__.py:1309)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 258 (idle): "zeroconf-Reaper_258"
    wait (threading.py:306)
    wait (zeroconf/__init__.py:2326)
    run (zeroconf/__init__.py:1417)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 0x7F334B5CCB20 (idle): "bellows.thread_0"
    select (selectors.py:468)
    _run_once (asyncio/base_events.py:1823)
    run_forever (asyncio/base_events.py:570)
    _thread_main (bellows/thread.py:28)
    run (concurrent/futures/thread.py:57)
    _worker (concurrent/futures/thread.py:80)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 260 (idle): "Thread-6"
    loop (paho/mqtt/client.py:1163)
    loop_forever (paho/mqtt/client.py:1782)
    _thread_main (paho/mqtt/client.py:3428)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 261 (idle): "zeroconf-ServiceBrowser__googlecast._tcp.local._261"
    wait (threading.py:306)
    wait (zeroconf/__init__.py:2326)
    run (zeroconf/__init__.py:1655)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 264 (idle): "Thread-10"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 265 (idle): "Thread-11"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 266 (idle): "Thread-12"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 267 (idle): "Thread-13"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 268 (idle): "Thread-14"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 269 (idle): "Thread-15"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 270 (idle): "Thread-16"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 271 (idle): "Thread-17"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 272 (idle): "Thread-18"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 273 (idle): "Thread-19"
    initialize_connection (pychromecast/socket_client.py:428)
    _check_connection (pychromecast/socket_client.py:672)
    run_once (pychromecast/socket_client.py:554)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 274 (idle): "Thread-20"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 275 (idle): "Thread-21"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 276 (idle): "Thread-22"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 281 (idle): "Thread-27"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 282 (idle): "Thread-28"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 283 (idle): "Thread-29"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 296 (idle): "job_queue"
    wait (threading.py:302)
    wait (threading.py:558)
    _main_loop (telegram/ext/jobqueue.py:284)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 297 (idle): "dispatcher"
    wait (threading.py:306)
    get (queue.py:179)
    start (telegram/ext/dispatcher.py:217)
    _thread_wrapper (telegram/ext/updater.py:156)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 299 (idle): "updater"
    read (ssl.py:1099)
    recv_into (ssl.py:1241)
    readinto (socket.py:669)
    _read_status (http/client.py:264)
    begin (http/client.py:303)
    getresponse (http/client.py:1332)
    _make_request (telegram/vendor/ptb_urllib3/urllib3/connectionpool.py:398)
    urlopen (telegram/vendor/ptb_urllib3/urllib3/connectionpool.py:614)
    urlopen (telegram/vendor/ptb_urllib3/urllib3/poolmanager.py:244)
    request_encode_body (telegram/vendor/ptb_urllib3/urllib3/request.py:148)
    request (telegram/vendor/ptb_urllib3/urllib3/request.py:68)
    _request_wrapper (telegram/utils/request.py:203)
    post (telegram/utils/request.py:307)
    get_updates (telegram/bot.py:1975)
    decorator (telegram/bot.py:65)
    polling_action_cb (telegram/ext/updater.py:278)
    _network_loop_retry (telegram/ext/updater.py:319)
    _start_polling (telegram/ext/updater.py:297)
    _thread_wrapper (telegram/ext/updater.py:156)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 300 (idle): "f0528f02-d625-43cf-9ab5-f91e7626b1c8_0"
    wait (threading.py:302)
    get (queue.py:170)
    _pooled (telegram/ext/dispatcher.py:154)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 301 (idle): "f0528f02-d625-43cf-9ab5-f91e7626b1c8_1"
    wait (threading.py:302)
    get (queue.py:170)
    _pooled (telegram/ext/dispatcher.py:154)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 304 (idle): "f0528f02-d625-43cf-9ab5-f91e7626b1c8_2"
    wait (threading.py:302)
    get (queue.py:170)
    _pooled (telegram/ext/dispatcher.py:154)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 305 (idle): "f0528f02-d625-43cf-9ab5-f91e7626b1c8_3"
    wait (threading.py:302)
    get (queue.py:170)
    _pooled (telegram/ext/dispatcher.py:154)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 307 (idle): "Thread-42"
    read_events (watchdog/observers/inotify_c.py:296)
    run (watchdog/observers/inotify_buffer.py:59)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 308 (idle): "Thread-3"
    wait (threading.py:302)
    get (watchdog/utils/delayed_queue.py:54)
    read_event (watchdog/observers/inotify_buffer.py:43)
    queue_events (watchdog/observers/inotify.py:129)
    run (watchdog/observers/api.py:146)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 309 (idle): "Thread-2"
    wait (threading.py:306)
    get (queue.py:179)
    dispatch_events (watchdog/observers/api.py:360)
    run (watchdog/observers/api.py:199)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 316 (idle): "Thread-50"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 317 (idle): "Thread-51"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 45982 (idle): "zeroconf-Engine-45982"
    run (zeroconf/__init__.py:1304)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 45983 (idle): "zeroconf-Reaper_45983"
    wait (threading.py:306)
    wait (zeroconf/__init__.py:2326)
    run (zeroconf/__init__.py:1417)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
30 
Thread 255 (idle): "SyncWorker_5"
    wait (threading.py:306)
    wait (zeroconf/__init__.py:2326)
    request (aioesphomeapi/host_resolver.py:44)
    resolve_host (aioesphomeapi/host_resolver.py:64)
    run (concurrent/futures/thread.py:57)
    _worker (concurrent/futures/thread.py:80)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)

It looks like aioesphomeapi is starting another Zeroconf engine.

https://github.com/home-assistant/core/issues/38248#issuecomment-671098906 suggested adding logging when this is detected. Will see if we can get something added.

I’ll also look into getting aioesphomeapi to use the shared Zeroconf instance

1 Like
31

Esphome changes


1 Like
32

Another report here indicating the Ikea component may also have some bearing on the issue:

33

Similar issue when updating from 0.112.4 to 0.113.3 (HA supervised).
The cpu usage is rising over time, a reboot “resets” the cpu usage.

Integrations:

  • Certificate Expiry
  • Google Cast
  • IFTTT
  • mobile_app
  • xiaomi_aqara

dump, top and record available here:

34

How many cast devices do you have?

35

14 devices and 11 entities according to Google Cast integration. The majority of them are offline and only powered on and connected to the network when needed (Chromecast audio/video, Android TV set top boxes) and a few are always on (Google Home)

36

There isn’t anything that really stands out in the data. What % had cpu usage risen to when the data was captured?

37

It’s rising up to 30% after a day (when the data was captured). It’s usually around 3% after a restart of HA and used to stay around that value on 0.112.4

38

Would you please capture another one when it gets around 70-75%

39

I had to roll back to 0.112.0. I was getting only about 2 hours of runtime before 0.113.3 would become completely unresponsive. I’m running an HA docker image with a Supervisor docker image. I was running the latest versions available at the time. I’m not running any of the components mentioned above as possible causes. I’ll do some more troubleshooting next week if the issue isn’t identified before them.