Python3 high CPU Usage

[tom_l]

Okay. I think I’m getting to the bottom of this.

I have quite few API calls like this to montor addon and system metrics:

- platform: rest
  resource: https://homeassistant:8123/api/hassio/addons/core_mariadb/stats
  name: MariaDB CPU
  unit_of_measurement: '%'
  value_template: '{{ (value_json.data.cpu_percent|float)|round(2) }}'
  scan_interval: 60
  verify_ssl: false
  headers:
    Authorization: !secret ha_api_token
    Content-Type: application/json
  json_attributes_path: "$.data"
  json_attributes:
    - memory_percent

After giving them all prime number scan intervals my core CPU usage has stabilised to this use pattern:

Expanded:

@bdraco these API calls seem to be quite CPU intensive. Is there anything that can be done to optimise them?

[Shaad (David)]

Too much monitoring kills monitoring !

[tom_l]

To be honest the addon monitoring has never shown an issue.

If there is no way to reduce the impact of this monitoring I’ll probably delete the sensors and view, (and then 3 months later I’ll need it to diagnose an issue).

[bdraco (J. Nick Koston)]

@tom_l

@bdraco these API calls seem to be quite CPU intensive. Is there anything that can be done to optimise them?

These look like they are expensive on the supervisor or operating system side. Unfortunately that is a bit outside of what I can help with as I generally only work on core and frontend and don’t have the same level of insight into the other code bases.

[tom_l]

No problem. Thanks for letting me know.

[drthanwho (Thanasis)]

I’ve tried about every option I could find and can’t get py-spy installed.

On Rpi4 with HAOS the only way to do it seemed with precompiled binary but getting:

ERROR: py_spy-0.3.4-py2.py3-none-manylinux2014_aarch64.whl is not a supported wheel on this platform.

Then tried to do it with cargo but that also fails halfway through compiling with

error: failed to run custom build command for `syn v1.0.61`

Caused by:
  could not execute process `/tmp/cargo-installBNVoFA/release/build/syn-a31747dac66b12a2/build-script-buil                                                           d` (never executed)

Caused by:
  Permission denied (os error 13)
warning: build failed, waiting for other jobs to finish...
error: failed to compile `py-spy v0.3.3`, intermediate artifacts can be found at `/tmp/cargo-installBNVoFA`

Caused by:
  build failed

[bdraco (J. Nick Koston)]

Please see post #74

Python3 high CPU Usage

Just as a follow up in case anyone else comes here looking for the same info: I had to first add “–cap-add SYS_PTRACE” to my docker run command to enable py-spy to run without permission errors. Next, exec into your container by either using the console in portainer or via SSH. docker exec -it <your_container_name> /bin/bash Once inside the container then run the following to install py-spy: echo 'manylinux1_compatible = True' > /usr/local/lib/python3.8/site-packages/_manylinux.py pip inst…

[drthanwho (Thanasis)]

Ah I was reading through all of that yesterday and completely missed it. In my case I’m getting RAM increasing linearly from baseline till it maxes out and container dies off till it restarts. Would this work for RAM too?

[bdraco (J. Nick Koston)]

If the cause of the issue is also a cpu hog then it might show it. They tend to be related so there is a good chance it will help point you in the right direction.

[drthanwho (Thanasis)]

Hmm I wouldn’t say it does as much. CPY hardly goes up a lot but it kind of seems like it does to a degree. It’s always the RAM the goes very obviously haywire. But let’s see.

What does the profiler.memory service do btw?

[bdraco (J. Nick Koston)]

You are probably looking for https://www.home-assistant.io/integrations/profiler/#service-profilerstart_log_objects

If you start logging objects when one of those memory spikes happens you should be able to see what is being created from the log.

[drthanwho (Thanasis)]

Thanks for the info! I didn’t realise there was a page in the docs. I’ll be ready when it happens again

[bdraco (J. Nick Koston)]

You could probably just leave the logger running as it shouldn’t have too much impact on cpu.

[drthanwho (Thanasis)]

Left it run for a few hours today. The issue now is I can see it increasing as a trend from the graph but it’s happening very very slowly over the last 24h compared to going 38 to 90% over 1-2h before

My question though is how do I interpret the output profile publishes in the logs?

[bdraco (J. Nick Koston)]

It is outputting the number of each object type in memory. If you post some of the log I can give you better guidance

[drthanwho (Thanasis)]

OK was removing everything not coming from Profiler otherwise was getting crowded
This is everything since starting it last night. Also in the process of me doing this it seems I caught it right in the act. Now even CPU is going up. Was slowly going up and all of a sudden sky rocketed

EDIT: I take it back whatever caused the abnormal CPU/RAM of the last 20mins has now died down. RAM is back at the previous but elevated levels of around 47% for now

https://gist.github.com/drthanwho/f87fcd5292043a20ec32cfe9a3978250

[bdraco (J. Nick Koston)]

Is the python process actually increasing in memory, or is it a subprocess?

[bdraco (J. Nick Koston)]

Maybe https://github.com/home-assistant/core/issues/47711.

[johanschelin]

I may have the same issue, but wat more specifically do you mean was the error and solution you applied?

[gieljnssns (Giel)]

I’m having high cpu again

Py-spy Dropbox - spy-2021.5.5.svg - Simplify your life