Instructions to install Py-spy on HAOS

Community Guides
4

A few notes:

  • with the community add-ons repo it’s possible to remotely log in from an SSH client also with username: and password: if they are specified in the add-on config. To disable protection it’s not enough to just toggle the switch, you also need to restart the addon.
  • it’s worth getting the latest version of py_spy from Releases · benfred/py-spy · GitHub
  • to get the correct pid of the HA process you should run first ps or top and look for the pid of python3 -m homeassistant --config /config. In my case this was 66, so I had to use the commands above with 66 not 60.
  • if the funky-named record file 66-2023-03-02T20:37:26+01:00.svg is not visible on your smb share you should rename it to something more natural, with cp '66-2023-03-02T20:37:26+01:00.svg' 66-1.svg or better add an output filename option like -o homeassistant_$(date +"%Y_%m_%d_%H_%M").svg to the command.
  • to continuously grab data in 120s chunks while waiting for something to happen you could do something like while true; do ./py-spy record --pid 66 -o homeassistant_$(date +"%Y_%m_%d_%H_%M").svg --duration 120 --rate 100; sleep 1; done but keep in mind you need to press Ctrl+C very quickly twice in order to escape from the endless loop.
5

on the community add-on:

I don’t use that, as mentioned, because we can not copy and paste, which is a real nuisance. Hence my preference to use an external Terminal. Much better solution, also, if your instance might not be reachable via the frontend for some reason, you can still use the terminal window (if setup properly ofc, for port 22222, see linked instructions to do so)

thanks for your additional observations. havent run it in a long time. the pid check was already in the above guide:

top (to get pid for homeassistant)

not sure if that should be emphasized a bit more.

anyways, cheers!

6

thanks,
Ive edited the main post a bit, so this is in the commands people can simply c&p

7

not sure if this is possible with all installation methods, but if the pgrep-utility is available you can query thd pid dynamically in the command for py-spy like:
py-spy dump --pid $(pgrep -u homeassistant hass )
here looking for processes named “hass” run by user homeassistant

Armin

8

thanks, I’ll give that a try.

but, how is that easier than checking top ?`

or, would this incorporate the correct pid directly into the other commands, is that what this does

9

yes, it puts the correct pid directly into the py-spy command ( or any other command).
pgrep -u homeassistant hass
just returns the pid, perhaps a little bit better than using top, as you do not have to know where in top to look for the pid of the process
using this with $(…) puts the output of the command directly as a parameter into any command in the shell

py-spy dump --pid $(pgrep -u homeassistant hass )
dumps all threads for the homeassistant process and you don’t need to know how to find the pid

Armin

1 Like
10

cool, I’ve updated the instructions above with this method. (left the original as reference, because as you said, not being sure this works for all platforms)

Great contribution, thanks!

11

I get “no such file or directory” for this command.

12

Added “wget” before the URL.

1 Like
13

Thank you. I think I may have solved my issue though. Will see if I need it in the next 24 hrs.

14

thanks,
out of curiosity: how could you edit my post?
is this because of it being a community guide anyone can edit?

15

Yes. Read the category description:

Community Guides

The Community Guides section is a place to share guides/tutorials with our community. Every post/topic in this section works like a Wiki and can be edited and improved by anybody.

Please note, guides provided in this section may be outdated/broken and are not supported by Home Assistant. Use these at your own risk.

1 Like
16

So that saves py_spy-0.3.14-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl in /config, now what?

I see no compressed file.

17

The whl-File is the compressed file

mkdir pyspy
cd pyspy
unzip ../py_spy-0.3.14-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl
cd py_spy-0.3.14.data/scripts

from the scripts folder you can execute pyspy with
./pyspy

Armin

1 Like
18

Thanks Armin.

I did not actually solve my issue and it occurred again just now. So just successfully ran py-spy for 120s.

Following Marius’s instructions it created an SVG file and dumped a bunch of info to my screen.

From memory the last time I did this it produced data files that Nick could interpret. I can’t find those anywhere. What should I send to Nick for analysis?

19

Perhaps you are looking for the “–format” parameter of the record command? Did you produce raw format?
py-spy record --help
I guess it depends on your problem? When I used py-spy I had a something leaking threads, used “pyspy dump” to monitor the threads and a command line entity to monitor the number of threads, my system was getting unstable after having 100s of threads created … at the end a custom integration was identified.

Armin

20

Maybe I was thinking of the Profiler integration?

21

Another option is to use gProfiler. It uses py spy also. Instructions are here: GitHub - Granulate/gprofiler: gProfiler is a system-wide profiler, combining multiple sampling profilers to produce unified visualization of what your CPU is spending time on.

You can generate different reports of the running threads, processes, …

Example:

flameshot_2024-02-05_10-40
flameshot_2024-02-05_10-401873×894 145 KB

22

I’m trying to diagnose high CPU issue on an RPi4.

I’ve disabled almost all integrations, and still seeing a solid ~65% cpu usage. (very stable - CPU usage stays at that level)

I have installed py-spy, but can’t find the PID for home assistant. top shows nothing with “homeassistant” or python or any other obvious process.

I note that for both top and htop, all the visible processes are ~0% CPU usage, yet I can see all four cores are running about 65% each. i.e I can’t see any processes actually using any CPU…yet the CPU is very busy

It appears as if whatever process is using all the CPU is hidden from me?

pgrep -u homeassistant shows unknown user.

I’m pretty lost here…I cant py-spy a process if I can’t see that process?

Here is the output from ps aux

PID USER TIME COMMAND
1 root 0:00 /package/admin/s6/command/s6-svscan -d4 – /run/service
17 root 0:00 s6-supervise s6-linux-init-shutdownd
19 root 0:00 /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -d3 -c /run/s6/basedir -g 3
26 root 0:00 s6-supervise s6rc-oneshot-runner
27 root 0:00 s6-supervise ttyd
28 root 0:00 s6-supervise sshd
29 root 0:00 s6-supervise s6rc-fdholder
37 root 0:00 /package/admin/s6/command/s6-ipcserverd -1 – /package/admin/s6/command/s6-ipcserver-acc
310 root 0:00 ttyd -d1 -i hassio --writable -p 62211 tmux -u new -A -s homeassistant zsh -l
313 root 0:00 sshd: /usr/sbin/sshd -D -e [listener] 0 of 10-100 startups
415 root 0:00 sshd: redacted [priv]
417 julz 0:00 sshd: redacted@pts/0
418 root 0:00 sudo -i
420 root 0:00 sudo -i
421 root 0:16 -zsh
1013 root 0:00 ps aux

23

Update: I couldn’t figure out how to see the processes directly from the terminal (I guess it’s running ina docker that is hidden from me, and I don’t know how to access).

But I installed Glances, and found it is the hassio_dns process that is using 162% cpu.

I checked my DNS entry, it was correct (8.8.8.8). So I changed it to point at my local DNS server, (that all my DHCP clients use), and that still had the same issue. I switched again to 1.1.1.1, same problem.

I note the dns logs have many lines like:
[ERROR] plugin/errors: 2 . NS: dial tcp 1.1.1.1:853: connect: connection refused

Anyhow, this is no longer a py-spy issue, so I’ll refrain from commenting further in this thread, and go try and figure out this DNS issue. Just leaving this info here in case it helps anyone else track down their CPU usage issues.

This thread looks promising regarding High CPU usage from Hassio DNS