I have an automation that lets me know when the processor useage on my mini PC goes over 80%. It was triggering every hour on the hour so I looked at Glances at that time and Frigate CPU went from 170% to 438% - I know this isn’t accurate but I assume the increase is. System Monitor shows 100% CPU at top of the hour for 15 seconds while it’s around 40-50% otherwise. It’s only been since I upgraded Frigate to version 14 (which I LOVE btw). I can’t imagine this is normal - others would have noticed by now. System metrics in Frigate (Detectors, GPUs, CPU) don’t show any spikes.
I have HWACCEL enabled (I think) and use a Google Coral on a mini PC with N100 processor “bare bones”
I also have an N100 and a Google Coral and exactly the top of every hour the CPU hits 100% VERY briefly.
1 Like
Mine mysteriously stopped doing this. Hasn’t been over 67% for past 4 days.
It’s been 9 days - pretty much since I wrote this question - and it hasn’t triggered the automation that monitors CPU useage. I changed it so that CPU had to be over 85 percent for 3 seconds. Maybe an update or restart fixed it - or maybe it was just a fluke. Hoping yours is okay now too
I’m still experiencing this. I can’t figure out what is causing it. All Intergrations/add-ons are up to date. Anyone knows how to deep dive on how to monitor every running process CPU?
I confirmed via Glances addon that it was Frigate doing it. I know I disabled the Music Assistant addon but I think that was after the spikes stopped seemingly on their own. I don’t remember if I stopped HA on the mini PC rather than just restart HA - maybe try that since I think I probably did.
But my frigate CPU in the graph is not spiking. I think I have already tried what you suggested but will try again, just in case.
So it appears to be frigate although the system graph does not show hourly spikes on that add-on.
Do you have Glances addon installed? It will show which processes are highest for CPU and memory - everything running actually but you can sort high to low. It doesn’t keep a running total (graph) though. There is an addon called Grafana but I couldn’t get it setup correctly.
Glances shows “addon_ccab4aaf_frigate-fa” at 125% CPU now while Frigate says 41%. It was saying 438% when the CPU in System Monitor said 100. I wouldn’t have worried but it was also raising the CPU temp a lot. Looks like just us 3 and mine is okay now. Good luck
were you able to solve this? i’m having the same spikes on the hour, and when i disable Frigate 0.14.1 addon the spikes go away.
I haven’t solved it, sorry. The hourly spike is not causing a problem really, not that I can see. I have another issue where if my internal cameras are turned off, when it’s powered up again, the CPU goes 100%. The only way to bring it down again is re-start home assistant.
I haven’t had the problem - that I know of - in awhile. I have an automation to let me know if CPU goes over 85%. It does when I use ESPHome compiler but only for a minute or soMy only concern is I have a mini PC and not sure what is cooling it; whatever it is seems to do fine after a spike. I have no idea what I did and don’t think I did anything at all in Frigate
I encountered the same issue. After some debugging, I discovered that at the start of each hour, Frigate generates preview clips for each camera using ffmpeg. This causes a significant spike in CPU usage as the preview generation processes consume nearly all available CPU resources, leading to timeouts in the camera processing tasks.
Below is a snippet of the output from top, showing the ffmpeg preview generation processes, which are prefixed with ffmpeg -hide_banner -f concat:
top - 08:00:02 up 1 day, 18:04, 1 user, load average: 2.37, 2.12, 2.19 [IO: 0.20, 0.10, 0.15 CPU: 2.16, 2.01, 2.02]
Tasks: 508 total, 15 running, 490 sleeping, 0 stopped, 3 zombie
%Cpu(s): 82.9 us, 14.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 2.4 si, 0.0 st
GiB Mem : 17.418 total, 0.175 free, 3.060 used, 14.183 buff/cache
GiB Swap: 12.451 total, 12.443 free, 0.008 used. 11.560 avail Mem
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
16203 root 20 0 129.7m 25.7m 44.44 0.144 0:00.23 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
16231 root 20 0 33.6m 3.4m 16.67 0.019 0:00.03 R top -b -n1
19209 root 20 0 1013.0m 155.2m 16.67 0.870 2:05.29 S frigate.recording_manager
11901 root 15 -5 2563.5m 2.191g 11.11 12.58 250:19.44 S /usr/local/bin/qemu-system-x86_64 -name guest=afa59364-6fa2-480f-9fc8-7c98831165b4,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-afa5936+
16169 root 20 0 130.4m 26.1m 11.11 0.146 0:00.34 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
16174 root 20 0 130.3m 26.5m 11.11 0.148 0:00.26 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
16194 root 20 0 130.3m 26.2m 11.11 0.147 0:00.19 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
16217 root 20 0 129.8m 25.9m 11.11 0.145 0:00.14 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
16219 root 20 0 129.8m 25.5m 11.11 0.143 0:00.08 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
19242 root 20 0 1926.5m 172.5m 11.11 0.967 14:07.18 R frigate.output
7 root 20 0 0.0m 0.0m 5.556 0.000 8:05.90 R [rcu_sched]
9443 root 20 0 5960.3m 401.4m 5.556 2.251 370:16.55 R /usr/bin/node /server/node_modules/@scrypted/server/dist/scrypted-main.js child @scrypted/prebuffer-mixin
16167 root 20 0 130.0m 25.9m 5.556 0.145 0:00.40 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
16171 root 20 0 130.0m 25.6m 5.556 0.144 0:00.31 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
18836 root 20 0 1879.9m 324.8m 5.556 1.821 5:25.52 S python3 -u -m frigate
19420 root 20 0 1694.1m 133.0m 5.556 0.745 0:44.88 S frigate.capture:backyard
19437 root 20 0 452.4m 39.1m 5.556 0.219 1:06.87 S ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -user_agent FFmpeg +
top - 09:00:04 up 1 day, 19:04, 1 user, load average: 3.41, 2.32, 2.30 [IO: 0.09, 0.05, 0.01 CPU: 3.32, 2.26, 2.26]
Tasks: 508 total, 11 running, 494 sleeping, 0 stopped, 3 zombie
%Cpu(s): 81.0 us, 14.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 4.8 si, 0.0 st
GiB Mem : 17.418 total, 0.196 free, 3.082 used, 14.140 buff/cache
GiB Swap: 12.451 total, 12.444 free, 0.008 used. 11.561 avail Mem
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
21404 root 20 0 130.4m 26.7m 21.05 0.150 0:00.62 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
19242 root 20 0 1925.2m 175.4m 15.79 0.983 22:42.71 S frigate.output
21424 root 20 0 129.9m 25.5m 15.79 0.143 0:00.56 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
21431 root 20 0 130.4m 26.4m 15.79 0.148 0:00.44 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
21436 root 20 0 129.9m 25.9m 15.79 0.145 0:00.42 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
9443 root 20 0 5956.1m 398.3m 10.53 2.233 383:06.09 S /usr/bin/node /server/node_modules/@scrypted/server/dist/scrypted-main.js child @scrypted/prebuffer-mixin
21397 root 20 0 130.5m 26.8m 10.53 0.150 0:00.68 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
21401 root 20 0 130.0m 26.0m 10.53 0.146 0:00.57 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
21426 root 20 0 130.0m 26.0m 10.53 0.146 0:00.46 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
21438 root 20 0 129.9m 25.6m 10.53 0.143 0:00.38 R ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps+
21468 root 20 0 33.6m 3.5m 10.53 0.020 0:00.03 R top -b -n1
An example of the full command:
ffmpeg -hide_banner -f concat -y -protocol_whitelist pipe,file -safe 0 -threads 1 -i /dev/stdin -c:v libx264 -profile:v baseline -preset:v ultrafast -threads 1 -g 40 -bf 0 -b:v 9216 -fps_mode vfr -movflags +faststart -pix_fmt yuv420p /media/frigate/clips/previews/cam1/1735999767.38576-1736002800.140329.mp4
I believe the code that trigger the preview generation at each exact hour is in frigate/output/preview.py
Someone has already filed a Github issue here: [Support]: High CPU every hour (on the hour), presumably from Preview process · blakeblackshear/frigate · Discussion #14711 · GitHub
1 Like
Nice to know. It seems to have stopped now or maybe it’s so quick that I don’t notice it - System Monitor doesn’t either. Thanks for following this up