hello
I would like a copy of the dashboard ,thank you
Howdy.
I’m running blakeblackshear/frigate:0.5.0 on an Intel NUC with a Coral USB C TPU.
I’m having an issue where even though the ffmpeg stream might be restarting correctly, It doesn’t look like the camera detection is kicking back in, or seeing the new stream. The debug view shows the camera frozen in time. Usually it’s only one camera, but this last time they both happened to stop within a half hour of each other.
This happens about every 12 hours where at least one of the cameras will freeze in Frigate.
On 0.4.0 they were rock solid. On most of the 0.5.0-rcX, I was having the freezing issues, but I thought it was the same issue as everyone else, and assumed the later RCs had fixed it. That turns out not to be the case for me. I’m hoping something might jump out to you in the logs or annotated monitoring below.
Here’s an image showing the monitoring (Times are in EST -05:00):
Here are the logs from the last few hours (Times are in UTC +00:00):
2020-03-05T23:22:28.417632427Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-03-05T23:26:48.884498134Z front_window: ffmpeg_process exited unexpectedly with 0
2020-03-05T23:26:48.884517394Z Terminating the existing ffmpeg process...
2020-03-05T23:26:48.884520202Z Waiting for ffmpeg to exit gracefully...
2020-03-05T23:26:48.884522509Z Creating ffmpeg process...
2020-03-05T23:26:48.884529454Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-03-05T23:34:19.475124741Z front_window: ffmpeg_process exited unexpectedly with 0
2020-03-05T23:34:19.475147210Z Terminating the existing ffmpeg process...
2020-03-05T23:34:19.475151836Z Waiting for ffmpeg to exit gracefully...
2020-03-05T23:34:19.475155220Z Creating ffmpeg process...
2020-03-05T23:34:19.475158684Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-03-05T23:39:59.990735187Z front_window: ffmpeg_process exited unexpectedly with 0
2020-03-05T23:39:59.990753544Z Terminating the existing ffmpeg process...
2020-03-05T23:39:59.990756298Z Waiting for ffmpeg to exit gracefully...
2020-03-05T23:39:59.990758829Z Creating ffmpeg process...
2020-03-05T23:39:59.990761325Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-03-05T23:52:40.678362433Z front_window: ffmpeg_process exited unexpectedly with 0
2020-03-05T23:52:40.678382344Z Terminating the existing ffmpeg process...
2020-03-05T23:52:40.678385165Z Waiting for ffmpeg to exit gracefully...
2020-03-05T23:52:40.678387428Z Creating ffmpeg process...
2020-03-05T23:52:40.678389906Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
2020-03-05T23:52:57.333850405Z kitchen: ffmpeg_process exited unexpectedly with 0
2020-03-05T23:52:57.333873245Z Terminating the existing ffmpeg process...
2020-03-05T23:52:57.333903132Z Waiting for ffmpeg to exit gracefully...
2020-03-05T23:52:57.333908071Z Creating ffmpeg process...
2020-03-05T23:52:57.333911487Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.3:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
I don’t think this is the source of your issue, but you should remove the -vf mpdecimate from your output args. Frigate is now looking for changes on its own and filtering them out upstream will cause frigate to skip lots of frames (your skipped frames should stay near 0 most of the time). It also prevents frigate from establishing a good baseline for what the image looks like when there is no motion.
Next time this happens, can you check the tracked_objects_queue value at /debug/stats? I just experienced an issue for the first time today where the thread that processes that queue was stuck. When that happens, the coral is still running detection, but nothing is looking at the output and publishing to mqtt or updating the debug video feed.
1 Like
It’s obviously a bit opinionated in that there is a separate panel for each camera, so you would need to adjust and tweak for your own needs.
In addition, this is the home-assistant config block used for the data collection: https://github.com/billimek/home-assistant-config/blob/84d1083f5b2af39a030f311f01b74635d7b4b783/config/sensors.yaml#L536-L608
2 Likes
More testing and it looks like it may be a TCP timeout thing.
In frigate, I killed the stream. I then restarted the camera.
When ffmpeg restarted and the camera connected, the stream picked up and moved along.
I see the following which seems like the timeout value is off but I don’t know if that means it truly is off or it’s simply not set.
tcp 51944 0 e5c7d5da9709:48254 10.0.0.32:554 ESTABLISHED 1301/ffmpeg off (0.00/0/0)
Is there a way to kill the port from frigate (some kind of kill session command) or is that a connection timeout set on the camera?
So you are suggesting that the TCP connection isn’t being released even though the ffmpeg process has exited? Are you using netstat to check open connections? Maybe it’s possible that the OS is associating that connection with python rather than ffmpeg.
yes… I’m doing that with
netstat -op | grep ffmpeg
it’s showing both ffmpeg entries but I’m just pasted the one for the driveway camera.
Hopefully I can reproduce it on my side. Nice work.
I just checked and noticed both cameras my front_window camera stopped responding/is stuck.
The kitchen camera is kinda responding. But the debug view seems to take ages (as in 2-10 minutes) before it updates… even after manually reloading the page.
Here are the stats from the /debug/stats endpoint:
{
"coral": {
"detection_queue": 2016,
"detection_start": 0,
"fps": 3.7,
"inference_speed": 9.74
},
"front_window": {
"detection_fps": 3.7,
"fps": 10.1,
"skipped_fps": 0
},
"kitchen": {
"detection_fps": 0,
"fps": 0.1,
"skipped_fps": 0
},
"plasma_store_rc": null,
"tracked_objects_queue": 0
}
Does the value in detection queue under coral hold constant?
After 30 minutes, this is what the debug stats are showing:
So it looks like the detection_queue is climbing.
{
"coral": {
"detection_queue": 2203,
"detection_start": 0,
"fps": 3.7,
"inference_speed": 9.74
},
"front_window": {
"detection_fps": 3.7,
"fps": 10.1,
"skipped_fps": 0
},
"kitchen": {
"detection_fps": 0,
"fps": 0.1,
"skipped_fps": 0
},
"plasma_store_rc": null,
"tracked_objects_queue": 0
}
Seems like your detection process is stuck, but in a different place. I will create a dev image for you to test with tonight.
1 Like
When I kill the ffmpeg process with kill -9 PID, I see the TCP connection disappear from netstat immediately for my camera. Are you seeing it linger for a long time with the old PID?
Thanks! I’ll keep you posted.
I tired running the latest dev image last night (confirmed with the hash against what was listed on DuckerHub). It looks like one of my cameras still stopped working last night. When I look at it, Frigate still seems to be detecting now without a jammed up queue. So that’s progress!
But the front_window ffmpeg stream stopped responding shortly after 2 am and never restarted.
The green light in the screenshot is when I pulled your latest dev image and started it up.
Here are the logs (in UTC). It looks like it was a pretty quiet night in the logs from the time that it first started up:
2020-03-07T01:40:49.744759216Z On connect called
2020-03-07T01:40:49.896175893Z /arrow/cpp/src/plasma/store.cc:1226: Allowing the Plasma store to use up to 0.4GB of memory.
2020-03-07T01:40:49.896192501Z /arrow/cpp/src/plasma/store.cc:1253: Starting object store with directory /dev/shm and huge page support disabled
2020-03-07T01:40:50.758495797Z Starting detection process: 29
2020-03-07T01:40:50.760509429Z Camera_process started for front_window: 30
2020-03-07T01:40:50.762050807Z Starting process for front_window: 30
2020-03-07T01:40:50.762337847Z ffprobe -v panic -show_error -show_streams -of json "rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword"
2020-03-07T01:40:50.762711458Z Camera_process started for kitchen: 31
2020-03-07T01:40:50.764457526Z Starting process for kitchen: 31
2020-03-07T01:40:50.764815766Z ffprobe -v panic -show_error -show_streams -of json "rtmp://192.168.10.1:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword"
2020-03-07T01:40:50.777726862Z * Serving Flask app "detect_objects" (lazy loading)
2020-03-07T01:40:50.777746230Z * Environment: production
2020-03-07T01:40:50.777750469Z WARNING: This is a development server. Do not use it in a production deployment.
2020-03-07T01:40:50.777780693Z Use a production WSGI server instead.
2020-03-07T01:40:50.777794254Z * Debug mode: off
2020-03-07T01:40:56.140539822Z {'streams': [{'index': 0, 'codec_name': 'aac', 'codec_long_name': 'AAC (Advanced Audio Coding)', 'profile': 'LC', 'codec_type': 'audio', 'codec_time_base': '1/16000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 'fltp', 'sample_rate': '16000', 'channels': 1, 'channel_layout': 'mono', 'bits_per_sample': 0, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/1000', 'start_pts': 499272321, 'start_time': '499272.321000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'High', 'codec_type': 'video', 'codec_time_base': '1/20', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 2560, 'height': 1440, 'coded_width': 2560, 'coded_height': 1440, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:1', 'pix_fmt': 'yuv420p', 'level': 51, 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'true', 'nal_length_size': '4', 'r_frame_rate': '30/1', 'avg_frame_rate': '10/1', 'time_base': '1/1000', 'start_pts': 499272285, 'start_time': '499272.285000', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
2020-03-07T01:40:56.148909376Z Creating ffmpeg process...
2020-03-07T01:40:56.148940531Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.1:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-07T01:40:56.620343048Z {'streams': [{'index': 0, 'codec_name': 'aac', 'codec_long_name': 'AAC (Advanced Audio Coding)', 'profile': 'LC', 'codec_type': 'audio', 'codec_time_base': '1/16000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 'fltp', 'sample_rate': '16000', 'channels': 1, 'channel_layout': 'mono', 'bits_per_sample': 0, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/1000', 'start_pts': 67265274, 'start_time': '67265.274000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'High', 'codec_type': 'video', 'codec_time_base': '1/20', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 2560, 'height': 1920, 'coded_width': 2560, 'coded_height': 1920, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:1', 'pix_fmt': 'yuv420p', 'level': 51, 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'true', 'nal_length_size': '4', 'r_frame_rate': '12/1', 'avg_frame_rate': '10/1', 'time_base': '1/1000', 'start_pts': 67265191, 'start_time': '67265.191000', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
2020-03-07T01:40:56.634381155Z Creating ffmpeg process...
2020-03-07T01:40:56.634412103Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
Lastly, here are the stats from the debug endpoint:
{
"coral": {
"detection_queue": 0,
"detection_start": 0,
"fps": 25.3,
"inference_speed": 9.07
},
"front_window": {
"detection_fps": 9.6,
"fps": 10,
"skipped_fps": 0
},
"kitchen": {
"detection_fps": 15.7,
"fps": 12.1,
"skipped_fps": 0
},
"plasma_store_rc": null,
"tracked_objects_queue": 0
}
Hi,
First thing first - HUGE thanks for this great system!
I’ve got the Coral usb stick yesterday so I’ve installed Frigate for the first time.
all seems to work ok for 1 camera, so I’ve added another 7 (so 8 1080p cameras in total).
I have a few issues.
- it seems that everything stopped working a while back.
i’m getting those message in repeate in the docker log:
ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://[email protected]:554/11 -f rawvideo -pix_fmt rgb24 pipe:
backshachen: ffmpeg_process exited unexpectedly with 0
Terminating the existing ffmpeg process...
Waiting for ffmpeg to exit gracefully...
Creating ffmpeg process...
ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://[email protected]:554/11 -f rawvideo -pix_fmt rgb24 pipe:
kvishback: ffmpeg_process exited unexpectedly with 0
Terminating the existing ffmpeg process...
Waiting for ffmpeg to exit gracefully...
Creating ffmpeg process...
ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://[email protected]:554/11 -f rawvideo -pix_fmt rgb24 pipe:
those are the stats:
and the debug:
coral: {
"detection_queue": 3761,
"detection_start": 0,
"fps": 0,
"inference_speed": 8.72
}
frontkvish: {
"detection_fps": 0,
"fps": 0.1,
"skipped_fps": 0
}
backkvish: {
"detection_fps": 0,
"fps": 0.1,
"skipped_fps": 0
}
kvishback: {
"detection_fps": 0,
"fps": 0.1,
"skipped_fps": 0
}
kvishfront: {
"detection_fps": 0,
"fps": 0.1,
"skipped_fps": 0
}
backshachen: {
"detection_fps": 0,
"fps": 0.1,
"skipped_fps": 0
}
shachenback: {
"detection_fps": 0,
"fps": 0.1,
"skipped_fps": 0
}
shachenfront: {
"detection_fps": 0,
"fps": 0.1,
"skipped_fps": 0
}
frontshachen: {
"detection_fps": 0,
"fps": 0.1,
"skipped_fps": 0
}
friendly_name: Frigate Debug
-
the other issue is with false detections, some examples:
https://i.imgur.com/TJV2LkR.jpg
hxxps://i.imgur.com/AWdSG3Y.jpg
hxxps://i.imgur.com/rnY3zhc.jpg
hxxps://i.imgur.com/Gvna0xo.jpg
hxxps://i.imgur.com/sGVMZEC.jpg -
I have one camera that I want to get detection for the full image, and another detection for a smaller part of the image (to turn on a nearby light automatically). do I need to setup 2 “cameras” from for same source to do this? one normally and one with a mask?
Thanks a lot again.
I’m not sure if this is it, so I’ll have to wait for it to die on it’s own to confirm, but it looks promising.
If I simply kill the ffmpeg process and let it restart, the parent is still running and the stream doesn’t continue.
Killing just this, does nothing
root 368 367 22 09:36 ? 00:00:05 ffmpeg -loglevel verbose -rtsp_transport tcp -i rtsp://username:[email protected]/live -f rawvideo -pix_fmt rgb24 pipe:
If I restart BOTH the ffmpeg AND the parent, it resumes.
UID PID PPID C STIME TTY TIME CMD
root 1 0 6 09:26 ? 00:00:38 python3.7 -u detect_objects.py
root 13 1 0 09:26 ? 00:00:04 /usr/local/bin/plasma_store -m 400000000 -s /tmp/plasma
root 19 1 0 09:26 ? 00:00:00 python3.7 -u detect_objects.py
root 21 1 47 09:26 ? 00:04:49 python3.7 -u detect_objects.py
root 25 21 74 09:26 ? 00:07:32 ffmpeg -loglevel verbose -rtsp_transport tcp -i rtsp://username:[email protected]:554/cam/realmonitor?channel=1&subtype=0 -f rawvideo -pix_fmt rgb
root 66 0 0 09:26 pts/0 00:00:00 /bin/bash
root 331 1 1 09:35 ? 00:00:01 [ffmpeg]
root 367 1 11 09:36 ? 00:00:03 python3.7 -u detect_objects.py
root 368 367 23 09:36 ? 00:00:06 ffmpeg -loglevel verbose -rtsp_transport tcp -i rtsp://username:[email protected]/live -f rawvideo -pix_fmt rgb24 pipe:
root 396 66 0 09:36 pts/0 00:00:00 ps -ef
The logs aren’t printing anything after 2am? And the video feed for front_window is frozen with an old timestamp?