High CPU usage after 0.113

[bdraco (J. Nick Koston)]

Thank you for the py-spy recording. Would you please post the full .svg file as the detail is lost when in the screenshot.

[apop]

pyspy.svg

[bdraco (J. Nick Koston)]

Thank you for that. Would you please post a py-spy dump as well (After the cpu has crept up)

[carlos.vroque]

I noticed my HA instance having higher CPU usage lately too, restarting just HA seems to turn everything back to normal again and the drop is pretty obvious:

Next monday I’ll be home and will install these debugging tools!

[bdraco (J. Nick Koston)]

If you are seeing the cpu creeping up issue, do you have homekit_controller enabled? It looks like there may be another code path that does not use the shared zeroconf instance that needs to be converted.

We thought we got them all in https://github.com/home-assistant/core/pull/37691 but it appears something still needs to be found.

It may be the same issue here Python3 high CPU Usage

I’m collecting details in https://github.com/home-assistant/core/issues/38668

[tom_l]

If it is any use this is my list of installed integrations:

In the UI:

• certificate expiry
• esphome
• google cast
• hacs
• home assistant ios
• lifx
• mobile app
• mqtt
• panasonic viera
• simpleicons
• speedtest.net

In HACS:

• hacs
• bwalarm (ak47 edition)
• sun2
• bom forecast
• garbage collection
• simpleicons

[decairn (Keith Hudson)]

Well, I don’t have a homekit installed but it is showing up in the logs as being loaded. Maybe it loads if it discovers anything Apple on the network?

Log line is: Setting up stage 2: {‘weather’, ‘group’, ‘sonos’, ‘input_boolean’, ‘tts’, ‘panel_custom’, ‘somfy_mylink’, ‘ssdp’, ‘zeroconf’, ‘cast’, ‘discovery’, ‘cover’, ‘roku’, ‘vesync’, ‘notion’, ‘script’, ‘synology_dsm’, ‘zwave’, ‘influxdb’, ‘stream’, ‘conversation’, ‘unifiprotect’, ‘media_player’, ‘rachio’, ‘hacs’, ‘sun’, ‘system_health’, ‘mqtt’, ‘persistent_notification’, ‘switch’, ‘logbook’, ‘homekit_controller’, ‘unifi’, ‘nest’, ‘history’, ‘ipp’, ‘sensor’, ‘automation’, ‘light’, ‘binary_sensor’, ‘tplink’, ‘updater’}

The ones in Integration UI are: vesync, google cast, HACS, MQTT, Nest, Notion, Roku, Sonos, Tp-Link Kasa Smart, Ubiquiti Unifi, Ubiquiti Protect, ZWave.

In addition to those, there’s a garadget and darksky module loaded through config.

[bdraco (J. Nick Koston)]

I’ve sorted out the issue with homekit_controller (details -> https://github.com/home-assistant/core/issues/38668)

Please keep posting the py-spy record (svg) and dump (text) output as there is probably another integration that needs to be fixed.

[apop]

Here you go. Gave it a few days for average CPU to steadily climb:

Thread 0x5608F5D338B0 (idle): "MainThread"
    select (selectors.py:468)
    _run_once (asyncio/base_events.py:1823)
    run_forever (asyncio/base_events.py:570)
    run_until_complete (asyncio/base_events.py:603)
    run (asyncio/runners.py:43)
    run (homeassistant/runner.py:119)
    main (homeassistant/__main__.py:312)
    <module> (homeassistant/__main__.py:320)
    _run_code (runpy.py:87)
    _run_module_as_main (runpy.py:194)
Thread 247 (idle): "Thread-1"
    dequeue (logging/handlers.py:1427)
    _monitor (logging/handlers.py:1478)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 248 (idle): "SyncWorker_0"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 251 (idle): "SyncWorker_1"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 252 (idle): "SyncWorker_2"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 253 (idle): "SyncWorker_3"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 254 (idle): "SyncWorker_4"
    _worker (concurrent/futures/thread.py:78)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 255 (idle): "SyncWorker_5"
    wait (threading.py:306)
    wait (zeroconf/__init__.py:2326)
    request (aioesphomeapi/host_resolver.py:44)
    resolve_host (aioesphomeapi/host_resolver.py:64)
    run (concurrent/futures/thread.py:57)
    _worker (concurrent/futures/thread.py:80)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 256 (idle): "Recorder"
    wait (threading.py:302)
    get (queue.py:170)
    run (homeassistant/components/recorder/__init__.py:340)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 0x7F334B954B20 (active): "zeroconf-Engine-257"
    update_record (zeroconf/__init__.py:1608)
    update_record (netdisco/mdns.py:21)
    update_record (zeroconf/__init__.py:2588)
    handle_response (zeroconf/__init__.py:2628)
    handle_read (zeroconf/__init__.py:1400)
    run (zeroconf/__init__.py:1309)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 258 (idle): "zeroconf-Reaper_258"
    wait (threading.py:306)
    wait (zeroconf/__init__.py:2326)
    run (zeroconf/__init__.py:1417)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 0x7F334B5CCB20 (idle): "bellows.thread_0"
    select (selectors.py:468)
    _run_once (asyncio/base_events.py:1823)
    run_forever (asyncio/base_events.py:570)
    _thread_main (bellows/thread.py:28)
    run (concurrent/futures/thread.py:57)
    _worker (concurrent/futures/thread.py:80)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 260 (idle): "Thread-6"
    loop (paho/mqtt/client.py:1163)
    loop_forever (paho/mqtt/client.py:1782)
    _thread_main (paho/mqtt/client.py:3428)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 261 (idle): "zeroconf-ServiceBrowser__googlecast._tcp.local._261"
    wait (threading.py:306)
    wait (zeroconf/__init__.py:2326)
    run (zeroconf/__init__.py:1655)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 264 (idle): "Thread-10"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 265 (idle): "Thread-11"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 266 (idle): "Thread-12"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 267 (idle): "Thread-13"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 268 (idle): "Thread-14"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 269 (idle): "Thread-15"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 270 (idle): "Thread-16"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 271 (idle): "Thread-17"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 272 (idle): "Thread-18"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 273 (idle): "Thread-19"
    initialize_connection (pychromecast/socket_client.py:428)
    _check_connection (pychromecast/socket_client.py:672)
    run_once (pychromecast/socket_client.py:554)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 274 (idle): "Thread-20"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 275 (idle): "Thread-21"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 276 (idle): "Thread-22"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 281 (idle): "Thread-27"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 282 (idle): "Thread-28"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 283 (idle): "Thread-29"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 296 (idle): "job_queue"
    wait (threading.py:302)
    wait (threading.py:558)
    _main_loop (telegram/ext/jobqueue.py:284)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 297 (idle): "dispatcher"
    wait (threading.py:306)
    get (queue.py:179)
    start (telegram/ext/dispatcher.py:217)
    _thread_wrapper (telegram/ext/updater.py:156)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 299 (idle): "updater"
    read (ssl.py:1099)
    recv_into (ssl.py:1241)
    readinto (socket.py:669)
    _read_status (http/client.py:264)
    begin (http/client.py:303)
    getresponse (http/client.py:1332)
    _make_request (telegram/vendor/ptb_urllib3/urllib3/connectionpool.py:398)
    urlopen (telegram/vendor/ptb_urllib3/urllib3/connectionpool.py:614)
    urlopen (telegram/vendor/ptb_urllib3/urllib3/poolmanager.py:244)
    request_encode_body (telegram/vendor/ptb_urllib3/urllib3/request.py:148)
    request (telegram/vendor/ptb_urllib3/urllib3/request.py:68)
    _request_wrapper (telegram/utils/request.py:203)
    post (telegram/utils/request.py:307)
    get_updates (telegram/bot.py:1975)
    decorator (telegram/bot.py:65)
    polling_action_cb (telegram/ext/updater.py:278)
    _network_loop_retry (telegram/ext/updater.py:319)
    _start_polling (telegram/ext/updater.py:297)
    _thread_wrapper (telegram/ext/updater.py:156)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 300 (idle): "f0528f02-d625-43cf-9ab5-f91e7626b1c8_0"
    wait (threading.py:302)
    get (queue.py:170)
    _pooled (telegram/ext/dispatcher.py:154)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 301 (idle): "f0528f02-d625-43cf-9ab5-f91e7626b1c8_1"
    wait (threading.py:302)
    get (queue.py:170)
    _pooled (telegram/ext/dispatcher.py:154)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 304 (idle): "f0528f02-d625-43cf-9ab5-f91e7626b1c8_2"
    wait (threading.py:302)
    get (queue.py:170)
    _pooled (telegram/ext/dispatcher.py:154)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 305 (idle): "f0528f02-d625-43cf-9ab5-f91e7626b1c8_3"
    wait (threading.py:302)
    get (queue.py:170)
    _pooled (telegram/ext/dispatcher.py:154)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 307 (idle): "Thread-42"
    read_events (watchdog/observers/inotify_c.py:296)
    run (watchdog/observers/inotify_buffer.py:59)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 308 (idle): "Thread-3"
    wait (threading.py:302)
    get (watchdog/utils/delayed_queue.py:54)
    read_event (watchdog/observers/inotify_buffer.py:43)
    queue_events (watchdog/observers/inotify.py:129)
    run (watchdog/observers/api.py:146)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 309 (idle): "Thread-2"
    wait (threading.py:306)
    get (queue.py:179)
    dispatch_events (watchdog/observers/api.py:360)
    run (watchdog/observers/api.py:199)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 316 (idle): "Thread-50"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 317 (idle): "Thread-51"
    run_once (pychromecast/socket_client.py:561)
    run (pychromecast/socket_client.py:531)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 45982 (idle): "zeroconf-Engine-45982"
    run (zeroconf/__init__.py:1304)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)
Thread 45983 (idle): "zeroconf-Reaper_45983"
    wait (threading.py:306)
    wait (zeroconf/__init__.py:2326)
    run (zeroconf/__init__.py:1417)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)

[bdraco (J. Nick Koston)]

Thread 255 (idle): "SyncWorker_5"
    wait (threading.py:306)
    wait (zeroconf/__init__.py:2326)
    request (aioesphomeapi/host_resolver.py:44)
    resolve_host (aioesphomeapi/host_resolver.py:64)
    run (concurrent/futures/thread.py:57)
    _worker (concurrent/futures/thread.py:80)
    run (threading.py:870)
    _bootstrap_inner (threading.py:932)
    _bootstrap (threading.py:890)

It looks like aioesphomeapi is starting another Zeroconf engine.

https://github.com/home-assistant/core/issues/38248#issuecomment-671098906 suggested adding logging when this is detected. Will see if we can get something added.

I’ll also look into getting aioesphomeapi to use the shared Zeroconf instance

[bdraco (J. Nick Koston)]

Esphome changes

Add the ability to use the shared Home Assistant Zeroconf instance

Use the shared Zeroconf instance in esphome

[tom_l]

Another report here indicating the Ikea component may also have some bearing on the issue:

High CPU usage after 0.113 (also)

I downgraded to 0.112.3 and the problem went away. I then went back to 0.113.3 and through trial and error I figured out the high CPU usage is being caused by the Ikea Trådfri component. I also noticed that when I was having the high CPU usage, the Home Assistant Docker container was also pushing around 100Kb/sec over the network. That also stopped when I removed the Ikea Trådfri component from my configuration. Since the Ikea component has never really worked correctly in the Docker contain…

[chocomega]

Similar issue when updating from 0.112.4 to 0.113.3 (HA supervised).
The cpu usage is rising over time, a reboot “resets” the cpu usage.

Integrations:

• Certificate Expiry
• Google Cast
• IFTTT
• mobile_app
• xiaomi_aqara

dump, top and record available here:

dump.txt and 2 more files

[bdraco (J. Nick Koston)]

How many cast devices do you have?

[chocomega]

14 devices and 11 entities according to Google Cast integration. The majority of them are offline and only powered on and connected to the network when needed (Chromecast audio/video, Android TV set top boxes) and a few are always on (Google Home)

[bdraco (J. Nick Koston)]

There isn’t anything that really stands out in the data. What % had cpu usage risen to when the data was captured?

[chocomega]

It’s rising up to 30% after a day (when the data was captured). It’s usually around 3% after a restart of HA and used to stay around that value on 0.112.4

[bdraco (J. Nick Koston)]

Would you please capture another one when it gets around 70-75%

[khouse75]

I had to roll back to 0.112.0. I was getting only about 2 hours of runtime before 0.113.3 would become completely unresponsive. I’m running an HA docker image with a Supervisor docker image. I was running the latest versions available at the time. I’m not running any of the components mentioned above as possible causes. I’ll do some more troubleshooting next week if the issue isn’t identified before them.

[tht (Thomas)]

I do have a similar experience. CPU usage started to go up on 0.113.0 and even removing almost all integrations didn’t help. I have now moved to 0.114.0 but same result here.

The Sonos integration seems to add quite a bit to CPU usage. But even when removing it, CPU usage stays quite high (~10-50% on [email protected] while nothing happens).

Py-Spy results for the last minutes where CPU-usage was about 45%.

profile.svg

I’ve now increased logging to “info” just to make sure there is nothing happening but I still can’t see what Home-Assistant is doing.

Some informations about my installation:

• HASS in Docker on Intel NUC (i7 CPU)
• Recording to MariaDB
• KNX and MQTT Integrations, iRobot + two iOS devices
• HACS is installed but all custom integrations uninstalled (except for HACS itself)

I’m running out of ideas