Another option is to use gProfiler. It uses py spy also. Instructions are here: GitHub - Granulate/gprofiler: gProfiler is a system-wide profiler, combining multiple sampling profilers to produce unified visualization of what your CPU is spending time on.
You can generate different reports of the running threads, processes, …
Example:
I’m trying to diagnose high CPU issue on an RPi4.
I’ve disabled almost all integrations, and still seeing a solid ~65% cpu usage. (very stable - CPU usage stays at that level)
I have installed py-spy, but can’t find the PID for home assistant. top shows nothing with “homeassistant” or python or any other obvious process.
I note that for both top and htop, all the visible processes are ~0% CPU usage, yet I can see all four cores are running about 65% each. i.e I can’t see any processes actually using any CPU…yet the CPU is very busy
It appears as if whatever process is using all the CPU is hidden from me?
pgrep -u homeassistant shows unknown user.
I’m pretty lost here…I cant py-spy a process if I can’t see that process?
Here is the output from ps aux
PID USER TIME COMMAND
1 root 0:00 /package/admin/s6/command/s6-svscan -d4 – /run/service
17 root 0:00 s6-supervise s6-linux-init-shutdownd
19 root 0:00 /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -d3 -c /run/s6/basedir -g 3
26 root 0:00 s6-supervise s6rc-oneshot-runner
27 root 0:00 s6-supervise ttyd
28 root 0:00 s6-supervise sshd
29 root 0:00 s6-supervise s6rc-fdholder
37 root 0:00 /package/admin/s6/command/s6-ipcserverd -1 – /package/admin/s6/command/s6-ipcserver-acc
310 root 0:00 ttyd -d1 -i hassio --writable -p 62211 tmux -u new -A -s homeassistant zsh -l
313 root 0:00 sshd: /usr/sbin/sshd -D -e [listener] 0 of 10-100 startups
415 root 0:00 sshd: redacted [priv]
417 julz 0:00 sshd: redacted@pts/0
418 root 0:00 sudo -i
420 root 0:00 sudo -i
421 root 0:16 -zsh
1013 root 0:00 ps aux
Update: I couldn’t figure out how to see the processes directly from the terminal (I guess it’s running ina docker that is hidden from me, and I don’t know how to access).
But I installed Glances, and found it is the hassio_dns process that is using 162% cpu.
I checked my DNS entry, it was correct (8.8.8.8). So I changed it to point at my local DNS server, (that all my DHCP clients use), and that still had the same issue. I switched again to 1.1.1.1, same problem.
I note the dns logs have many lines like:
[ERROR] plugin/errors: 2 . NS: dial tcp 1.1.1.1:853: connect: connection refused
Anyhow, this is no longer a py-spy issue, so I’ll refrain from commenting further in this thread, and go try and figure out this DNS issue. Just leaving this info here in case it helps anyone else track down their CPU usage issues.
This thread looks promising regarding High CPU usage from Hassio DNS
You can build it manually from this PR fix line number for python 3.12 by penguin-wwy · Pull Request #666 · benfred/py-spy · GitHub
I tried building above mentioned py-spy PR for python3.12 myself (forked dev312 branch on GitHub, and build with GitHub Actions: simplify workflow push · TimSoethout/py-spy@dcdec48 · GitHub, see wheel.zip download at the bottom.)
I used the py_spy-0.3.14-py2.py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64 build for my Odroid C2.
This results in error when run in my homeassistant docker on hassio:
$ ./py-spy top --pid 67
thread 'Error: <unnamed>' panicked at 'Failed to get process executable name. Check that the process is running.called `Result::unwrap_err()` on an `Ok` value: ()
', Reason: src/sampler.rs:59No such file or directory: (os error 492
)note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Reason: No such file or directory (os error 2)
Any pointers?
(I’m debugging this to find out a memory leak that reboots my whole device every ~3h when memory get to ~90%.)