not sure if this is possible with all installation methods, but if the pgrep-utility is available you can query thd pid dynamically in the command for py-spy like:
py-spy dump --pid $(pgrep -u homeassistant hass )
here looking for processes named “hass” run by user homeassistant
Armin
thanks, I’ll give that a try.
but, how is that easier than checking top ?`
or, would this incorporate the correct pid directly into the other commands, is that what this does
yes, it puts the correct pid directly into the py-spy command ( or any other command).
pgrep -u homeassistant hass
just returns the pid, perhaps a little bit better than using top, as you do not have to know where in top to look for the pid of the process
using this with $(…) puts the output of the command directly as a parameter into any command in the shell
py-spy dump --pid $(pgrep -u homeassistant hass )
dumps all threads for the homeassistant process and you don’t need to know how to find the pid
Armin
cool, I’ve updated the instructions above with this method. (left the original as reference, because as you said, not being sure this works for all platforms)
Great contribution, thanks!
I get “no such file or directory” for this command.
Added “wget” before the URL.
Thank you. I think I may have solved my issue though. Will see if I need it in the next 24 hrs.
thanks,
out of curiosity: how could you edit my post?
is this because of it being a community guide anyone can edit?
Yes. Read the category description:
The Community Guides section is a place to share guides/tutorials with our community. Every post/topic in this section works like a Wiki and can be edited and improved by anybody.
Please note, guides provided in this section may be outdated/broken and are not supported by Home Assistant. Use these at your own risk.
So that saves py_spy-0.3.14-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl in /config, now what?
I see no compressed file.
The whl-File is the compressed file
mkdir pyspy
cd pyspy
unzip ../py_spy-0.3.14-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl
cd py_spy-0.3.14.data/scripts
from the scripts folder you can execute pyspy with
./pyspy
Armin
Thanks Armin.
I did not actually solve my issue and it occurred again just now. So just successfully ran py-spy for 120s.
Following Marius’s instructions it created an SVG file and dumped a bunch of info to my screen.
From memory the last time I did this it produced data files that Nick could interpret. I can’t find those anywhere. What should I send to Nick for analysis?
Perhaps you are looking for the “–format” parameter of the record command? Did you produce raw format?
py-spy record --help
I guess it depends on your problem? When I used py-spy I had a something leaking threads, used “pyspy dump” to monitor the threads and a command line entity to monitor the number of threads, my system was getting unstable after having 100s of threads created … at the end a custom integration was identified.
Armin
Maybe I was thinking of the Profiler integration?
Another option is to use gProfiler. It uses py spy also. Instructions are here: GitHub - Granulate/gprofiler: gProfiler is a system-wide profiler, combining multiple sampling profilers to produce unified visualization of what your CPU is spending time on.
You can generate different reports of the running threads, processes, …
Example:
I’m trying to diagnose high CPU issue on an RPi4.
I’ve disabled almost all integrations, and still seeing a solid ~65% cpu usage. (very stable - CPU usage stays at that level)
I have installed py-spy, but can’t find the PID for home assistant. top shows nothing with “homeassistant” or python or any other obvious process.
I note that for both top and htop, all the visible processes are ~0% CPU usage, yet I can see all four cores are running about 65% each. i.e I can’t see any processes actually using any CPU…yet the CPU is very busy
It appears as if whatever process is using all the CPU is hidden from me?
pgrep -u homeassistant shows unknown user.
I’m pretty lost here…I cant py-spy a process if I can’t see that process?
Here is the output from ps aux
PID USER TIME COMMAND
1 root 0:00 /package/admin/s6/command/s6-svscan -d4 – /run/service
17 root 0:00 s6-supervise s6-linux-init-shutdownd
19 root 0:00 /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -d3 -c /run/s6/basedir -g 3
26 root 0:00 s6-supervise s6rc-oneshot-runner
27 root 0:00 s6-supervise ttyd
28 root 0:00 s6-supervise sshd
29 root 0:00 s6-supervise s6rc-fdholder
37 root 0:00 /package/admin/s6/command/s6-ipcserverd -1 – /package/admin/s6/command/s6-ipcserver-acc
310 root 0:00 ttyd -d1 -i hassio --writable -p 62211 tmux -u new -A -s homeassistant zsh -l
313 root 0:00 sshd: /usr/sbin/sshd -D -e [listener] 0 of 10-100 startups
415 root 0:00 sshd: redacted [priv]
417 julz 0:00 sshd: redacted@pts/0
418 root 0:00 sudo -i
420 root 0:00 sudo -i
421 root 0:16 -zsh
1013 root 0:00 ps aux
Update: I couldn’t figure out how to see the processes directly from the terminal (I guess it’s running ina docker that is hidden from me, and I don’t know how to access).
But I installed Glances, and found it is the hassio_dns process that is using 162% cpu.
I checked my DNS entry, it was correct (8.8.8.8). So I changed it to point at my local DNS server, (that all my DHCP clients use), and that still had the same issue. I switched again to 1.1.1.1, same problem.
I note the dns logs have many lines like:
[ERROR] plugin/errors: 2 . NS: dial tcp 1.1.1.1:853: connect: connection refused
Anyhow, this is no longer a py-spy issue, so I’ll refrain from commenting further in this thread, and go try and figure out this DNS issue. Just leaving this info here in case it helps anyone else track down their CPU usage issues.
This thread looks promising regarding High CPU usage from Hassio DNS
You can build it manually from this PR fix line number for python 3.12 by penguin-wwy · Pull Request #666 · benfred/py-spy · GitHub
I tried building above mentioned py-spy PR for python3.12 myself (forked dev312 branch on GitHub, and build with GitHub Actions: simplify workflow push · TimSoethout/py-spy@dcdec48 · GitHub, see wheel.zip download at the bottom.)
I used the py_spy-0.3.14-py2.py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64 build for my Odroid C2.
This results in error when run in my homeassistant docker on hassio:
$ ./py-spy top --pid 67
thread 'Error: <unnamed>' panicked at 'Failed to get process executable name. Check that the process is running.called `Result::unwrap_err()` on an `Ok` value: ()
', Reason: src/sampler.rs:59No such file or directory: (os error 492
)note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Reason: No such file or directory (os error 2)
Any pointers?
(I’m debugging this to find out a memory leak that reboots my whole device every ~3h when memory get to ~90%.)