Python3 high CPU Usage

Did anything change recently that would have resurrected this issue?

I started getting high CPU usage recently and HA became sluggish it was almost not functional (Ok maybe it wasn’t that bad but it’s definitely below expectation).

I went thru all of my integrations and removed everything and nothing helped…until I remembered about the discussion of the “unavailable sensors” template causing issues.

So I had two sensors set up using different templates for a few months now with no issues…until recently.

After removing those two sensors everything calmed down and CPU load is back to normal.

here is a screenshot of the differences between with the templates and without. It’s pretty stark:

ex

the first low section is with neither template active. then the 100% load is with both active. then I disabled one of them. re-enabled both. then disabled both again.

here are the templates:

sensor:
  - platform: template
    sensors:
      unavailable_entities:
        #entity_id: sensor.time
        friendly_name: Unavailable Entities
        unit_of_measurement: items
        icon_template: >
          {% if states('sensor.unavailable_entities')|int == 0 %} mdi:check-circle
          {% else %} mdi:alert-circle
          {% endif %}
        value_template: >
          {% set update = states('sensor.time') %}
          {{expand(states.binary_sensor, states.sensor, states.switch, states.variable, states.media_player, states.light)|selectattr('state', 'in', ['unavailable','unknown','none'])
            |reject('in', expand('group.entity_blacklist'))
            |reject('eq', states.group.entity_blacklist)
            |list|length}}
        attribute_templates:
          entities: >
            {% set update = states('sensor.time') %}
            {{expand(states.binary_sensor, states.sensor, states.switch, states.variable, states.media_player, states.light)|selectattr('state', 'in', ['unavailable','unknown','none'])
              |reject('in', expand('group.entity_blacklist'))
              |reject('eq' , states.group.entity_blacklist)
              |reject('eq' , states.group.battery_status)
              |map(attribute='entity_id')|list|join(', ')}}
        
      sensors_unavailable:
        friendly_name: 'Entities that need attention'
        entity_id: sensor.time
        value_template: >
          {{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | list | length }}
        attribute_templates:
          entities: >
            {{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | map(attribute='entity_id') | list | join('\n') }}

Again, did something change recently that would have caused the issue to re-appear?

2 Likes

So to add another datapoint, I was seeing a large amount of CPU usage and my startup would hang if I had default-config or dhcp enabled (which is part of default-config). Disabling ipv6 on my router (latest openwrt) seems to have fixed it and its now faster than I ever remember it being. My cpu usage on my NUC has dropped from ~12% to <2%.

Hello @bdraco
would you like to help me out? I wish I could read the speed scope py-spy profile by myself. Maybe you can tell me what to look at?

Should I pm you the .svg file?

Feel free to send it

Hi everyone
What is the consensus on this issue? Is this individual?
I am running it on NUC and noticed CPU usage increase as well

_worker can be ignored as its idle time.

eintr_retry_call looks like a bug in pyudev. It could be related to Stack trace from MonitorObserver thread · Issue #194 · pyudev/pyudev · GitHub

write is from a tarfile / backup creation.

I have an Intel Nuc and its always running between 60-100 cpu use. I have tried the profiler and created to files. But I have no idea how to look at the files. Can someone help me or guide me how to do that? :slight_smile:

I believe the docs on the profiler page give a few options on how to view the files:

I also have this issue with mostly 100% cpu. But I’m wondering why.

Can someone read these files, as I can’t get to read it with those programs: HERE

What about addons. Which ones do you have installed?

I’m running HA in docker. I only have integrations:
Wemo
Marantz
Viera VT60 DLNA
Slimme meter
Google Cast
Govee
HACS
Brother Printer
Libreelec
LG webOS Smart TV
Lifx
Home
Mobile app
MQTT
Netgear
Node-RED companion
Profiler
Squeezebox
Tuya
Xiaomi Miio
Sun

configuration.yaml

# Configure a default setup of Home Assistant (frontend, api, etc)
default_config:

# Text to speech
tts:
  - platform: google_translate

automation: !include automations.yaml
script: !include scripts.yaml
scene: !include scenes.yaml

# GROUPS
light:
  - platform: group
    name: Eetkamer
    entities:
      - light.eetkamer_spots
      - light.eetkamer_1
      - light.eetkamer_2
      - light.eetkamer_3
  - platform: group
    name: Play
    entities:
      - light.play_l
      - light.play_r
  - platform: group
    name: Spots Tuin 
    entities: 
      - light.tuin_spot_links 
      - light.tuin_spot_rechts 
  - platform: group
    name: Spots Erker 
    entities: 
      - light.erker_spot_links 
      - light.erker_spot_rechts
  - platform: group
    name: Alle Lampen Woongedeelte
    entities:
      - light.erker_spot_links      
      - light.erker_spot_rechts
      - light.tuin_spot_links
      - light.tuin_spot_rechts
      - light.eetkamer_spots
      - light.eetkamer_1
      - light.eetkamer_2
      - light.eetkamer_3
      - light.play_l
      - light.play_r
      - light.keuken_ikea
      - light.hoeklicht
      - light.staande_lamp
      - light.hanglamp
  - platform: group
    name: Keuken
    entities:
      - light.keuken1
      - light.keuken2
      - light.keuken3

camera:
  - platform: xiaomi_cloud_map_extractor
    host: !secret xiaomi_vacuum_host
    token: !secret xiaomi_vacuum_token
    username: !secret xiaomi_cloud_username
    password: !secret xiaomi_cloud_password
    draw: ['all']
    attributes:
      - calibration_points


device_tracker:
  - platform: google_maps
    username: ********@gmail.com

What other containers do you have then?

zigbee2mqtt, mosquito, portainer.

When I stop homeassistant, the high CPU is gone and python3 is not shown in top.

To test, I made another Homeassistant test container. No configuration, no integrations, nothing. Cpu is now low.

So what makes the cpu so high on my main HA? Could it be the home-assistant_v2.db file. That’s about 500 MB.

Nope that’s quite small.

Backuped the database and tried with a fresh one, but that did not make a difference.

I think I narrowed it down. I disabled the custom_components folder. And now the CPU is max 15%.

And the issue was this: Xiaomi Cloud Map Extractor

Here is your event loop graph. Most of the time in there is spent on denonavr but overall its not that much.