Hey Blake, the change you made from 0.5.0-rc7 to 0.5.0 to specify the height of the debug view stretches my doorbell camera out of its natural aspect ratio. It isn’t a massive deal to me since this is just a debugging stream, but does represent a small regression on my end. Could you rewrite the code to preserve the natural resolution of the camera unless the height is specified rather than making a 360px height the default?
Is there prebuilt rpi4 docker anywhere ?
Looks great! I would like a copy of the dashboard so I can run it here, too.
I have updated the HA Addon for frigate so it uses 0.5.0 release.
You can add the following repository https://github.com/mories76/ha-addon-frigate
There is one major thing, the HA supervisor doesn’t have support for a larger shm-size than 64MB (yet)
I am running this on my celeron nuc without issues so far.
I don’t think anyone is using this addon, but if you do please let me know it this work for you as well or if something needs to be fixed.
@blakeblackshear Nice work !
4 Likes
hello
I would like a copy of the dashboard ,thank you
Howdy.
I’m running blakeblackshear/frigate:0.5.0 on an Intel NUC with a Coral USB C TPU.
I’m having an issue where even though the ffmpeg stream might be restarting correctly, It doesn’t look like the camera detection is kicking back in, or seeing the new stream. The debug view shows the camera frozen in time. Usually it’s only one camera, but this last time they both happened to stop within a half hour of each other.
This happens about every 12 hours where at least one of the cameras will freeze in Frigate.
On 0.4.0 they were rock solid. On most of the 0.5.0-rcX, I was having the freezing issues, but I thought it was the same issue as everyone else, and assumed the later RCs had fixed it. That turns out not to be the case for me. I’m hoping something might jump out to you in the logs or annotated monitoring below.
Here’s an image showing the monitoring (Times are in EST -05:00):
Here are the logs from the last few hours (Times are in UTC +00:00):
2020-03-05T23:22:28.417632427Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-03-05T23:26:48.884498134Z front_window: ffmpeg_process exited unexpectedly with 0
2020-03-05T23:26:48.884517394Z Terminating the existing ffmpeg process...
2020-03-05T23:26:48.884520202Z Waiting for ffmpeg to exit gracefully...
2020-03-05T23:26:48.884522509Z Creating ffmpeg process...
2020-03-05T23:26:48.884529454Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-03-05T23:34:19.475124741Z front_window: ffmpeg_process exited unexpectedly with 0
2020-03-05T23:34:19.475147210Z Terminating the existing ffmpeg process...
2020-03-05T23:34:19.475151836Z Waiting for ffmpeg to exit gracefully...
2020-03-05T23:34:19.475155220Z Creating ffmpeg process...
2020-03-05T23:34:19.475158684Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-03-05T23:39:59.990735187Z front_window: ffmpeg_process exited unexpectedly with 0
2020-03-05T23:39:59.990753544Z Terminating the existing ffmpeg process...
2020-03-05T23:39:59.990756298Z Waiting for ffmpeg to exit gracefully...
2020-03-05T23:39:59.990758829Z Creating ffmpeg process...
2020-03-05T23:39:59.990761325Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-03-05T23:52:40.678362433Z front_window: ffmpeg_process exited unexpectedly with 0
2020-03-05T23:52:40.678382344Z Terminating the existing ffmpeg process...
2020-03-05T23:52:40.678385165Z Waiting for ffmpeg to exit gracefully...
2020-03-05T23:52:40.678387428Z Creating ffmpeg process...
2020-03-05T23:52:40.678389906Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-03-05T23:52:57.333850405Z kitchen: ffmpeg_process exited unexpectedly with 0
2020-03-05T23:52:57.333873245Z Terminating the existing ffmpeg process...
2020-03-05T23:52:57.333903132Z Waiting for ffmpeg to exit gracefully...
2020-03-05T23:52:57.333908071Z Creating ffmpeg process...
2020-03-05T23:52:57.333911487Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.3:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
I don’t think this is the source of your issue, but you should remove the -vf mpdecimate from your output args. Frigate is now looking for changes on its own and filtering them out upstream will cause frigate to skip lots of frames (your skipped frames should stay near 0 most of the time). It also prevents frigate from establishing a good baseline for what the image looks like when there is no motion.
Next time this happens, can you check the tracked_objects_queue value at /debug/stats? I just experienced an issue for the first time today where the thread that processes that queue was stuck. When that happens, the coral is still running detection, but nothing is looking at the output and publishing to mqtt or updating the debug video feed.
1 Like
It’s obviously a bit opinionated in that there is a separate panel for each camera, so you would need to adjust and tweak for your own needs.
In addition, this is the home-assistant config block used for the data collection: https://github.com/billimek/home-assistant-config/blob/84d1083f5b2af39a030f311f01b74635d7b4b783/config/sensors.yaml#L536-L608
2 Likes
More testing and it looks like it may be a TCP timeout thing.
In frigate, I killed the stream. I then restarted the camera.
When ffmpeg restarted and the camera connected, the stream picked up and moved along.
I see the following which seems like the timeout value is off but I don’t know if that means it truly is off or it’s simply not set.
tcp 51944 0 e5c7d5da9709:48254 10.0.0.32:554 ESTABLISHED 1301/ffmpeg off (0.00/0/0)
Is there a way to kill the port from frigate (some kind of kill session command) or is that a connection timeout set on the camera?
So you are suggesting that the TCP connection isn’t being released even though the ffmpeg process has exited? Are you using netstat to check open connections? Maybe it’s possible that the OS is associating that connection with python rather than ffmpeg.
yes… I’m doing that with
netstat -op | grep ffmpeg
it’s showing both ffmpeg entries but I’m just pasted the one for the driveway camera.
Hopefully I can reproduce it on my side. Nice work.
I just checked and noticed both cameras my front_window camera stopped responding/is stuck.
The kitchen camera is kinda responding. But the debug view seems to take ages (as in 2-10 minutes) before it updates… even after manually reloading the page.
Here are the stats from the /debug/stats endpoint:
{
"coral": {
"detection_queue": 2016,
"detection_start": 0,
"fps": 3.7,
"inference_speed": 9.74
},
"front_window": {
"detection_fps": 3.7,
"fps": 10.1,
"skipped_fps": 0
},
"kitchen": {
"detection_fps": 0,
"fps": 0.1,
"skipped_fps": 0
},
"plasma_store_rc": null,
"tracked_objects_queue": 0
}
Does the value in detection queue under coral hold constant?
After 30 minutes, this is what the debug stats are showing:
So it looks like the detection_queue is climbing.
{
"coral": {
"detection_queue": 2203,
"detection_start": 0,
"fps": 3.7,
"inference_speed": 9.74
},
"front_window": {
"detection_fps": 3.7,
"fps": 10.1,
"skipped_fps": 0
},
"kitchen": {
"detection_fps": 0,
"fps": 0.1,
"skipped_fps": 0
},
"plasma_store_rc": null,
"tracked_objects_queue": 0
}
Seems like your detection process is stuck, but in a different place. I will create a dev image for you to test with tonight.
1 Like
When I kill the ffmpeg process with kill -9 PID, I see the TCP connection disappear from netstat immediately for my camera. Are you seeing it linger for a long time with the old PID?