If its running in docker, you can get into the container to use py-spy with docker exec -it homeassistant /bin/bash
OK, success I think, thank you for the help. I couldn’t execute that docker command from teh Terminal/SSH facility, the command is not recognised. However, I can get into the VM console, logging onto the HA CLI anbd then logging on as root. That docker command then works. Doing a ps -ef shows me that there is indeed a python3 process running. I tried running py-spy as above but again the command couldn’t be found. I then tried the binary from the second link you provided and the command ran and created a svg graph. So I’m just waiting now for the runaway process to happen again then I can post the results.
I can’t imagine there is any way for anyone to work out that this is how you’re supposed to get in to run the py-spy…as I said, it would be helpful if this was available as an add-on. It could even monitor the process itself and alert if it was running a high CPU level for a certain length of time.
1 Like
I’m running HA Container (not HassOS) but I’m getting the same result. Which other hoop did you need to jump thru?
Take a look two posts above that one: Python3 high CPU Usage
Let me know if you need more specific assistance, but hopefully that helps.
Thanks!
That got that going but now I’m not sure what the command is to actually run py-spy in “HA Container”.
I’ve exec’ed into the container and I found the PID using top but when I run:
py-spy record --pid 206 --output /config/www/py-spy.svg --duration 120
I get:
“Permission Denied: Try running again with elevated permissions by going ‘sudo env “PATH=$PATH” !!’”
Then when I try to run anything with sudo it says “sudo: command not found”.
Any idea what the correct command to run is inside the container to actually get py-spy to run in “HA Container” (not HassOS)
Just as a follow up in case anyone else comes here looking for the same info:
I had to first add “–cap-add SYS_PTRACE” to my docker run command to enable py-spy to run without permission errors.
Next, exec into your container by either using the console in portainer or via SSH.
docker exec -it <your_container_name> /bin/bash
Once inside the container then run the following to install py-spy:
echo 'manylinux1_compatible = True' > /usr/local/lib/python3.8/site-packages/_manylinux.py
pip install py-spy
After that run top to find the PID of HA. You should see something like the following:
the one you want is the python3 program for HA (first one in the list there).
then using the PID from above run the following (replacing my PID of 205 with your own):
py-spy record --pid 205 --output /config/www/py-spy.svg --duration 120
This will run a sample and eventually write a py-spy.svg to your www folder.
The only thing I couldn’t get to work is the recoprding didn’t stop automatically after 120 seconds. I had to stop it manually by hitting ctrl-c.
Also I kept getting a continous string of warnings that the sdampling was behind and to reduce the sampling rate. I don’t know how to do that so I just let it run. I’m not sure if it impacted the accuracy of the recording or not tho TBH.
4 Likes
For those who can’t get py-spy to work, I’ve put together a profiler integration that will generate a callgrind.xxx.out file that can be viewed in qcachegrind
4 Likes
I’ve updated the Profiler integration steps
The integration can now be configured from the UI so its a bit easier to get setup.
1 Like
For me it came back with 0.116.
Below 2 120s PySpys from 2 different processes.
https://drive.google.com/drive/folders/1tTJuvIjl7SM4LSe8br5xdelYg8gBx_Ee?usp=sharing
where/how do you want us to send you the profiler data? rather not send it all publicly 
btw, is it too much to ask to have the profiler write files in its own folder, or maybe a configurable folder. It now writes to /config, which feels a bit ‘unsafe’, and messy btw.
Contact details are here: https://github.com/home-assistant/core/issues/40292#issuecomment-702744364
yeah, I was in that discussion, but it was not very public, and about py-spy.
Now that this Profiler add-on is publicly released, I could imagine Nick not wanting all profiles to be sent to his private email address…
Feel free to send it on discord or to my email nick [at] koston [dot] org. No worries on the additional email.
The py-spys you posted are surprisingly clean. stream is taking a bit of cpu time, but nothing else shows up. Can you try a shorter duration?
cool. consider it done.
Ok, im doing it again. Will send it via email as asked above.
This is an excerpt from HTOP showing the 2 processes.
I am sending each a 10, 20, 30 & 60 seconds. Hope that helps
Screenshot from PySpy TOP:
Thanks for the py-spys. It looks like t might be related to a camera or stream. Can you try disabling the stream integration?
Hi, second day that load average gets over 1 and the CPU History (from within Home Assistant systemmonitor platform) shows that at 4:08am, I lose that history (ie, flat line). I usually runs at 0.20 load average. Not sure if it’s related. Restarting Home Assistant from ‘Configuration/Server Controls’ solves it.
But to try and find what’s causing this, I followed the instruction you gave to install py-spy in docker (I’m running a hassio in a supervised installation on a Raspberry 4) and get the following :
pi@homeassistant:~ $ sudo docker exec -it homeassistant /bin/bash
bash-5.0# echo 'manylinux1_compatible = True' > /usr/local/lib/python3.8/site-packages/_manylinux.py
bash-5.0# pip install py-spy
ERROR: Could not find a version that satisfies the requirement py-spy (from versions: none)
ERROR: No matching distribution found for py-spy
Running Home Assistant Core 116.0. Don’t know why it doesn’t want to install and I’m no closer at figuring what’s hogging Home Assistant 
The new monitoring tools are revealing this:
Core CPU
Supervisor CPU
System Monitor CPU
Profiler is not revealing anything amiss. What next?
Hi, where did you get the new system metrics history graph? All I have are bar charts under Supervisor and can’t find what you’re showing anywhere. Thanks