Python3 high CPU Usage

Installation Home Assistant OS
70

OK, success I think, thank you for the help. I couldn’t execute that docker command from teh Terminal/SSH facility, the command is not recognised. However, I can get into the VM console, logging onto the HA CLI anbd then logging on as root. That docker command then works. Doing a ps -ef shows me that there is indeed a python3 process running. I tried running py-spy as above but again the command couldn’t be found. I then tried the binary from the second link you provided and the command ran and created a svg graph. So I’m just waiting now for the runaway process to happen again then I can post the results.

I can’t imagine there is any way for anyone to work out that this is how you’re supposed to get in to run the py-spy…as I said, it would be helpful if this was available as an add-on. It could even monitor the process itself and alert if it was running a high CPU level for a certain length of time.

1 Like
71

I’m running HA Container (not HassOS) but I’m getting the same result. Which other hoop did you need to jump thru?

72

Take a look two posts above that one: Python3 high CPU Usage

Let me know if you need more specific assistance, but hopefully that helps.

73

Thanks!

That got that going but now I’m not sure what the command is to actually run py-spy in “HA Container”.

I’ve exec’ed into the container and I found the PID using top but when I run:

py-spy record --pid 206 --output /config/www/py-spy.svg --duration 120

I get:

“Permission Denied: Try running again with elevated permissions by going ‘sudo env “PATH=$PATH” !!’”

Then when I try to run anything with sudo it says “sudo: command not found”.

Any idea what the correct command to run is inside the container to actually get py-spy to run in “HA Container” (not HassOS)

74

Just as a follow up in case anyone else comes here looking for the same info:

I had to first add “–cap-add SYS_PTRACE” to my docker run command to enable py-spy to run without permission errors.

Next, exec into your container by either using the console in portainer or via SSH.

docker exec -it <your_container_name> /bin/bash

Once inside the container then run the following to install py-spy:

echo 'manylinux1_compatible = True' > /usr/local/lib/python3.8/site-packages/_manylinux.py

pip install py-spy

After that run top to find the PID of HA. You should see something like the following:

ex1
ex1713×218 13.6 KB

the one you want is the python3 program for HA (first one in the list there).

then using the PID from above run the following (replacing my PID of 205 with your own):

py-spy record --pid 205 --output /config/www/py-spy.svg --duration 120

This will run a sample and eventually write a py-spy.svg to your www folder.

The only thing I couldn’t get to work is the recoprding didn’t stop automatically after 120 seconds. I had to stop it manually by hitting ctrl-c.

Also I kept getting a continous string of warnings that the sdampling was behind and to reduce the sampling rate. I don’t know how to do that so I just let it run. I’m not sure if it impacted the accuracy of the recording or not tho TBH.

4 Likes
75

For those who can’t get py-spy to work, I’ve put together a profiler integration that will generate a callgrind.xxx.out file that can be viewed in qcachegrind

4 Likes
76

I’ve updated the Profiler integration steps

The integration can now be configured from the UI so its a bit easier to get setup.

1 Like
77

For me it came back with 0.116.
Below 2 120s PySpys from 2 different processes.

https://drive.google.com/drive/folders/1tTJuvIjl7SM4LSe8br5xdelYg8gBx_Ee?usp=sharing

78

where/how do you want us to send you the profiler data? rather not send it all publicly :wink:

btw, is it too much to ask to have the profiler write files in its own folder, or maybe a configurable folder. It now writes to /config, which feels a bit ‘unsafe’, and messy btw.

79

Contact details are here: https://github.com/home-assistant/core/issues/40292#issuecomment-702744364

80

yeah, I was in that discussion, but it was not very public, and about py-spy.

Now that this Profiler add-on is publicly released, I could imagine Nick not wanting all profiles to be sent to his private email address…

81

Feel free to send it on discord or to my email nick [at] koston [dot] org. No worries on the additional email.

82

The py-spys you posted are surprisingly clean. stream is taking a bit of cpu time, but nothing else shows up. Can you try a shorter duration?

83

cool. consider it done.

84

Ok, im doing it again. Will send it via email as asked above.
This is an excerpt from HTOP showing the 2 processes.

image
image1115×421 67.2 KB

I am sending each a 10, 20, 30 & 60 seconds. Hope that helps :slight_smile:

Screenshot from PySpy TOP:

image
image763×543 32.6 KB

image
image755×550 33 KB

85

Thanks for the py-spys. It looks like t might be related to a camera or stream. Can you try disabling the stream integration?

86

Hi, second day that load average gets over 1 and the CPU History (from within Home Assistant systemmonitor platform) shows that at 4:08am, I lose that history (ie, flat line). I usually runs at 0.20 load average. Not sure if it’s related. Restarting Home Assistant from ‘Configuration/Server Controls’ solves it.

But to try and find what’s causing this, I followed the instruction you gave to install py-spy in docker (I’m running a hassio in a supervised installation on a Raspberry 4) and get the following :

pi@homeassistant:~ $ sudo docker exec -it homeassistant /bin/bash
bash-5.0# echo 'manylinux1_compatible = True' > /usr/local/lib/python3.8/site-packages/_manylinux.py
bash-5.0# pip install py-spy
ERROR: Could not find a version that satisfies the requirement py-spy (from versions: none)
ERROR: No matching distribution found for py-spy

Running Home Assistant Core 116.0. Don’t know why it doesn’t want to install and I’m no closer at figuring what’s hogging Home Assistant :frowning:

87

The new monitoring tools are revealing this:

Core CPU

core
core1739×434 50.9 KB

Supervisor CPU

superv
superv1739×434 41.8 KB

System Monitor CPU

sysmon
sysmon1739×434 67.2 KB

Profiler is not revealing anything amiss. What next?

88

Hi, where did you get the new system metrics history graph? All I have are bar charts under Supervisor and can’t find what you’re showing anywhere. Thanks

89

Here, though I slowed them down to 30 second updates:

1 Like