For some reason Frigate is acting inconsistently. This morning I noticed that the CPU usage of my server dropped off, the usage fell right around 2am. I assumed that it was Frigate since I’ve been having weird CPU usage issues with it. I went in an restarted the docker container for it and CPU usage cranked up to 80% and sat there. The docker and system were pretty much left alone all day yesterday but you can see that the system regularly sits at a load of ~5, 40% CPU usage until this morning.
15min load:
CPU percentage:
On further inspection it seems to be a single ffmpeg that consumes an order of magnitude more processing power than all the others. I tried turning off the detection for that camera and it doesn’t change. I tried restarting the docker container a few times and that one camera rlc-520-1 always consumes 500-700% CPU (this is because of top’s multi-core/multi-threading reporting), this is why my overall CPU usage is going to 80% in the HA chart. Take a look at my top output:
top - 07:23:44 up 30 days, 23:48, 2 users, load average: 15.86, 15.27, 13.20
Tasks: 473 total, 5 running, 468 sleeping, 0 stopped, 0 zombie
%Cpu0 : 17.5 us, 7.7 sy, 56.2 ni, 18.2 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu1 : 12.7 us, 7.9 sy, 62.3 ni, 16.8 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu2 : 14.4 us, 8.9 sy, 57.7 ni, 18.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 16.6 us, 10.8 sy, 51.7 ni, 20.6 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu4 : 14.8 us, 9.4 sy, 56.0 ni, 19.5 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu5 : 13.6 us, 6.8 sy, 61.7 ni, 18.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 15.8 us, 7.9 sy, 55.5 ni, 20.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 24.1 us, 10.2 sy, 38.8 ni, 13.3 id, 0.0 wa, 0.0 hi, 13.6 si, 0.0 st
%Cpu8 : 17.0 us, 13.6 sy, 51.7 ni, 17.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 19.7 us, 12.4 sy, 48.2 ni, 19.4 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu10 : 15.7 us, 8.0 sy, 55.9 ni, 20.1 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu11 : 19.3 us, 10.7 sy, 49.7 ni, 19.3 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
MiB Mem : 128803.2 total, 10534.3 free, 7460.0 used, 110808.8 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 119924.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2083338 root 20 0 5565688 250436 106636 R 615.2 0.2 10:46.56 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-520-1/bcs/channel0_sub.bcs?channel=0&stream=1&user=uname_pword&password=uname_pword -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:a copy /tmp/cache/front_door-%Y%m%d%H%M%S.m+
2083355 root 20 0 5563552 244164 107052 S 34.3 0.2 0:39.29 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-410-3/bcs/channel0_sub.bcs?channel=0&stream=1&user=uname_pword&password=uname_pword -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:a copy /tmp/cache/east_yard-%Y%m%d%H%M%S.mp+
2083216 root 20 0 2777808 135292 38632 S 29.7 0.1 0:19.36 python3 -u -m frigate
2083359 root 20 0 5462424 206092 106096 S 29.7 0.2 0:32.05 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-410-1/bcs/channel0_sub.bcs?channel=0&stream=1&user=uname_pword&password=uname_pword -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:a copy /tmp/cache/back_yard-%Y%m%d%H%M%S.mp+
2083366 root 20 0 5462032 204332 106024 S 28.7 0.2 0:32.93 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-410-2/bcs/channel0_sub.bcs?channel=0&stream=1&user=uname_pword&password=uname_pword -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:a copy /tmp/cache/out_the_back-%Y%m%d%H%M%S+
2083346 root 20 0 5563540 245608 106804 S 24.8 0.2 0:30.69 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-410-4/bcs/channel0_sub.bcs?channel=0&stream=1&user=uname_pword&password=uname_pword -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:a copy /tmp/cache/west_yard-%Y%m%d%H%M%S.mp+
2083368 root 20 0 5562608 246228 107264 S 24.1 0.2 0:29.00 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-511-1/bcs/channel0_sub.bcs?channel=0&stream=1&user=uname_pword&password=uname_pword -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:a copy /tmp/cache/garage-%Y%m%d%H%M%S.mp4 -+
2083377 root 20 0 5562340 243296 106668 S 22.4 0.2 0:26.32 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-e1-zoom-1/bcs/channel0_sub.bcs?channel=0&stream=1&user=uname_pword&password=uname_pword -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:a copy /tmp/cache/storage_room-%Y%m%d%H+
2083323 root 20 0 2171100 97352 13704 S 13.9 0.1 0:05.46 frigate.process:storage_room
2083339 root 20 0 5435936 167336 128660 S 9.9 0.1 0:13.07 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-410-3/bcs/channel0_main.bcs?channel=0&stream=0&user=uname_pword&password=uname_pword -r 8 -f rawvideo -pix_fmt yuv420p pipe:
2083354 root 20 0 5433648 162952 127216 S 9.9 0.1 0:12.16 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-410-2/bcs/channel0_main.bcs?channel=0&stream=0&user=uname_pword&password=uname_pword -r 8 -f rawvideo -pix_fmt yuv420p pipe:
2083333 root 20 0 5431348 162048 128892 S 9.6 0.1 0:11.83 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-410-4/bcs/channel0_main.bcs?channel=0&stream=0&user=uname_pword&password=uname_pword -r 8 -f rawvideo -pix_fmt yuv420p pipe:
2083328 root 20 0 5431396 161772 128616 R 9.2 0.1 0:11.92 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-520-1/bcs/channel0_main.bcs?channel=0&stream=0&user=uname_pword&password=uname_pword -r 8 -f rawvideo -pix_fmt yuv420p pipe:
2083345 root 20 0 5433552 164236 127312 S 9.2 0.1 0:12.12 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-410-1/bcs/channel0_main.bcs?channel=0&stream=0&user=uname_pword&password=uname_pword -r 8 -f rawvideo -pix_fmt yuv420p pipe:
2083358 root 20 0 5431396 160708 128112 R 9.2 0.1 0:11.91 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-511-1/bcs/channel0_main.bcs?channel=0&stream=0&user=uname_pword&password=uname_pword -r 8 -f rawvideo -pix_fmt yuv420p pipe:
2083360 root 20 0 1629708 96168 5072 S 8.9 0.1 0:10.69 python3 -u -m frigate
2083369 root 20 0 5435144 165648 128904 R 8.9 0.1 0:11.52 ffmpeg -hide_banner -loglevel warning -hwaccel cuda -c:v h264_cuvid -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts -fflags +discardcorrupt -use_wallclock_as_timestamps 1 -i rtmp://rlc-e1-zoom-1/bcs/channel0_main.bcs?channel=0&stream=0&user=uname_pword&password=uname_pword -r 8 -f rawvideo -pix_fmt yuv420p pipe:
2083324 root 20 0 1629708 96212 5136 S 8.6 0.1 0:10.64 python3 -u -m frigate
2083327 root 20 0 1629708 96028 4944 S 8.6 0.1 0:10.51 python3 -u -m frigate
2083331 root 20 0 1629708 96092 5016 S 8.6 0.1 0:10.75 python3 -u -m frigate
2083351 root 20 0 1629708 96208 5136 S 8.6 0.1 0:10.70 python3 -u -m frigate
2083335 root 20 0 1627136 95632 5008 S 8.3 0.1 0:10.23 python3 -u -m frigate
2083340 root 20 0 1627136 95788 5136 S 7.9 0.1 0:10.24 python3 -u -m frigate
1355 root 20 0 3033148 48160 24460 S 5.9 0.0 3170:36 /usr/bin/containerd
I tried turning up the logging using:
logger:
default: debug
but the output from the docker log doesn’t shed any light on this issue. As you can see I’m using the nvidia docker and I can see these processes in my nvidia-smi output. Any ideas?
P.S. That stream rlc-520-1 is of my front door which is completely static right now and probably has been for the last several hours. No shadows, no trees blowing in the wind, no people ringing the doorbell. So high CPU usage on that stream doesn’t add up.
UPDATE
My lovely wife wanted to try to help me solve the issue so I was explaining it to her and showing her some of the details. While doing that I noticed that I could watch the live stream for all cameras from the Frigate interface which meant that the camera’s streams were working. However, from Home Assistant all the cameras worked except for the Front Door (rlc-520-1). I think this means that Frigate’s re-broadcast stream for that camera was broken. I tried unplugging the PoE cord for it, didn’t help. I tried restating the docker container again didn’t help. I even recreated the container and that didn’t help either. 
Defeated, I was interested in seeing how much wattage it was consuming so I pulled up the Sense monitor and stopped the docker to see it in the “off” state (clicking “stop” in portainer). ~100W of additional power was consumed by the CPU crunching that camera (house consumption went from ~1550W to ~1450W). I let it sit “off” for about a minute, then clicked “start” from portainer. Frigate came up and the problem was gone. Now I can see all camera’s streams from HA again and all ffmpeg threads are consuming between 20-30% CPU.
Every time I had previously tried to restart the docker container (probably 5-6 times this morning), I hit the portainer “restart” button. This time, I instead clicked the “stop” button and waited about a minute, then clicked the “start” button and now it’s back to normal (15min load of ~4-5). I wish I could understand why the container is so inconsistent. 