I’m making some headway. It looks like the way I was checking to see if the detection process was stuck was too aggressive, and I was getting some false positives. I think that was causing the detection process to eventually be killed while it was in the middle of pulling something from the queue. That could leave a lingering lock on the queue object and prevent it from fetching anything when restarted. It seems consistent with the behavior I have seen. Testing a fix now. 
5 Likes
Just published 0.5.1-rc4, which I hope will fix the issue that causes the detection process to stop. Here is the change log:
- Resize preview according to camera’s aspect ratio
- Discard plasma store logs
- Fix check for detection process hanging
3 Likes
Love the work you’ve done! For me, on all versions, it seems that if there’s motion detected for too long, the object detection and the video feed from Frigate gets progressively further behind, until the feed from Frigate is maybe 15 minutes delayed from the actual video feed. Object detection still works - it just gets pretty choppy and delayed. Interestingly there’s not really any motion typically, but it detects a parked car as motion, for example, and keeps up the detection boxes on it.
Here’s the debug output from just a few minutes ago where it is doing this.
{"back_door":{"camera_fps":10.0,"detection_fps":0.0,"ffmpeg_pid":180,"frame_info":{"detect":1587914831.704363,"process":1587914831.499066,"read":1587914832.365871},"pid":116,"process_fps":4.8,"read_start":0.0,"skipped_fps":4.8},"coral":{"detection_start":0.0,"fps":9.0,"inference_speed":7.78,"pid":30},"driveway":{"camera_fps":10.3,"detection_fps":2.4,"ffmpeg_pid":93,"frame_info":{"detect":1587914829.133347,"process":1587914829.128108,"read":1587914832.339407},"pid":119,"process_fps":2.4,"read_start":0.0,"skipped_fps":7.8},"front_door":{"camera_fps":10.4,"detection_fps":0.0,"ffmpeg_pid":53,"frame_info":{"detect":1587914831.055296,"process":1587914830.826594,"read":1587914832.138246},"pid":117,"process_fps":4.8,"read_start":0.0,"skipped_fps":4.8},"front_yard":{"camera_fps":10.7,"detection_fps":6.6,"ffmpeg_pid":65,"frame_info":{"detect":1587914828.235891,"process":1587914827.38679,"read":1587914832.138669},"pid":118,"process_fps":2.4,"read_start":0.0,"skipped_fps":9.0},"garage":{"camera_fps":10.0,"detection_fps":0.0,"ffmpeg_pid":114,"frame_info":{"detect":1587914831.930109,"process":1587914831.283865,"read":1587914832.36541},"pid":120,"process_fps":4.7,"read_start":0.0,"skipped_fps":5.1},"plasma_store_rc":null}
Pulling video from Ubiquiti cameras at medium resolution, 10fps.
Running on a Coral and a fairly beefy host (16gb RAM, i7).
Any ideas?
Have you customized the ffmpeg parameters? Also, just to clarify, frigate isn’t detecting your car as motion. It continues to track objects that it has found until they disappear, even if they have stopped moving.
Ah, good to know.
Here are my ffmpeg parameters:
ffmpeg:
global_args:
- -hide_banner
- -loglevel
- panic
hwaccel_args:
- -hwaccel
- vaapi
- -hwaccel_device
- /dev/dri/renderD128
- -hwaccel_output_format
- yuv420p
input_args:
- -avoid_negative_ts
- make_zero
- -fflags
- nobuffer
- -flags
- low_delay
- -strict
- experimental
- -fflags
- +genpts+discardcorrupt
- -vsync
- drop
- -rtsp_transport
- tcp
- -stimeout
- '5000000'
- -use_wallclock_as_timestamps
- '1'
output_args:
- -f
- rawvideo
- -pix_fmt
- rgb24
1 Like
I don’t see anything that would be an issue in your ffmpeg params. Based on your /debug/stats output, frigate thinks it is showing you an image that is 5 seconds behind for front_yard. Was that output saved when it was 15 minutes behind? You can see that under frame_info. The read value is when frigate read the frame from ffmpeg, the detect value is the frame it is running detection on, and the process value is the time of the frame it is processing after detection.
Hi @blakeblackshear. I started running the latest RC when you posted it last (0.5.1-rc4). Thanks again for digging into this and attempted to debug our problems.
Unfortunately it looks like Frigate is still freezing up for my Reolink cameras (both different models). 
Both cameras stopped responding at the same time inside Frigate. The logs show they never restarted. The logs don’t seem to indicate any crashes. Here’s the entire log with no entries for today when they stopped working.
logs:
2020-04-25T17:08:22.251700217Z On connect called
2020-04-25T17:08:23.266945018Z ffprobe -v panic -show_error -show_streams -of json "rtmp://192.168.0.51:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword"
2020-04-25T17:08:23.267521749Z Starting detection process: 30
2020-04-25T17:08:28.105496983Z {'streams': [{'index': 0, 'codec_name': 'aac', 'codec_long_name': 'AAC (Advanced Audio Coding)', 'profile': 'LC', 'codec_type': 'audio', 'codec_time_base': '1/16000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 'fltp', 'sample_rate': '16000', 'channels': 1, 'channel_layout': 'mono', 'bits_per_sample': 0, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/1000', 'start_pts': 40140194, 'start_time': '40140.194000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'High', 'codec_type': 'video', 'codec_time_base': '1/18', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 2560, 'height': 1920, 'coded_width': 2560, 'coded_height': 1920, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:1', 'pix_fmt': 'yuv420p', 'level': 51, 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'true', 'nal_length_size': '4', 'r_frame_rate': '10/1', 'avg_frame_rate': '9/1', 'time_base': '1/1000', 'start_pts': 40140145, 'start_time': '40140.145000', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
2020-04-25T17:08:28.106092668Z Creating ffmpeg process...
2020-04-25T17:08:28.106111889Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.0.51:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-04-25T17:08:28.111635821Z ffprobe -v panic -show_error -show_streams -of json "rtmp://192.168.0.50:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword"
2020-04-25T17:08:33.795026589Z {'streams': [{'index': 0, 'codec_name': 'aac', 'codec_long_name': 'AAC (Advanced Audio Coding)', 'profile': 'LC', 'codec_type': 'audio', 'codec_time_base': '1/16000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 'fltp', 'sample_rate': '16000', 'channels': 1, 'channel_layout': 'mono', 'bits_per_sample': 0, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/1000', 'start_pts': 558571675, 'start_time': '558571.675000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'High', 'codec_type': 'video', 'codec_time_base': '1/22', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 2560, 'height': 1440, 'coded_width': 2560, 'coded_height': 1440, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:1', 'pix_fmt': 'yuv420p', 'level': 51, 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'true', 'nal_length_size': '4', 'r_frame_rate': '30/1', 'avg_frame_rate': '11/1', 'time_base': '1/1000', 'start_pts': 558571559, 'start_time': '558571.559000', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
2020-04-25T17:08:33.795328220Z Creating ffmpeg process...
2020-04-25T17:08:33.795336441Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.0.50:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-04-25T17:08:33.800437572Z Camera_process started for front_window: 52
2020-04-25T17:08:33.801123517Z Starting process for front_window: 52
2020-04-25T17:08:33.802488776Z Camera_process started for kitchen: 53
2020-04-25T17:08:33.803659286Z Starting process for kitchen: 53
2020-04-25T17:08:33.810045277Z * Serving Flask app "detect_objects" (lazy loading)
2020-04-25T17:08:33.810061082Z * Environment: production
2020-04-25T17:08:33.810063551Z WARNING: This is a development server. Do not use it in a production deployment.
2020-04-25T17:08:33.810066919Z Use a production WSGI server instead.
2020-04-25T17:08:33.810068925Z * Debug mode: off
/debug/stats/
{
"coral": {
"detection_start": 0,
"fps": 7.9,
"inference_speed": 7.72,
"pid": 30
},
"front_window": {
"camera_fps": 0,
"detection_fps": 4.6,
"ffmpeg_pid": 41,
"frame_info": {
"detect": 1587880762.259035,
"process": 1587880762.259035,
"read": 1587880762.259035
},
"pid": 52,
"process_fps": 5.1,
"read_start": 1587880762.271842,
"skipped_fps": 0
},
"kitchen": {
"camera_fps": 0,
"detection_fps": 3.3,
"ffmpeg_pid": 50,
"frame_info": {
"detect": 1587880737.855407,
"process": 1587880737.855407,
"read": 1587880737.886853
},
"pid": 53,
"process_fps": 6,
"read_start": 1587880737.867302,
"skipped_fps": 0
},
"plasma_store_rc": null
}
Debug output PID 50 (FFMPEG)
2020-04-26T18:05:24.340835104Z kitchen: ffmpeg didnt return a frame. something is wrong.
2020-04-26T18:05:24.341003117Z kitchen: ffmpeg process is not running. exiting capture thread...
2020-04-26T18:05:30.301681938Z Creating ffmpeg process...
2020-04-26T18:05:30.301735236Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.0.50:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
Debug output PID 53 (Kitchen Camera)
2020-04-26T18:05:50.346067806Z File "detect_objects.py", line 348, in <module>
2020-04-26T18:05:50.346158214Z main()
2020-04-26T18:05:50.346183440Z File "detect_objects.py", line 227, in main
2020-04-26T18:05:50.346204751Z camera_process['process'].start()
2020-04-26T18:05:50.346225774Z File "/usr/lib/python3.7/multiprocessing/process.py", line 112, in start
2020-04-26T18:05:50.346245616Z self._popen = self._Popen(self)
2020-04-26T18:05:50.346264390Z File "/usr/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
2020-04-26T18:05:50.346284233Z return _default_context.get_context().Process._Popen(process_obj)
2020-04-26T18:05:50.346303475Z File "/usr/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
2020-04-26T18:05:50.346323334Z return Popen(process_obj)
2020-04-26T18:05:50.346381611Z File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
2020-04-26T18:05:50.346400594Z self._launch(process_obj)
2020-04-26T18:05:50.346416410Z File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
2020-04-26T18:05:50.346433626Z code = process_obj._bootstrap()
2020-04-26T18:05:50.346449970Z File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
2020-04-26T18:05:50.346466987Z self.run()
2020-04-26T18:05:50.346482738Z File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
2020-04-26T18:05:50.346499530Z self._target(*self._args, **self._kwargs)
2020-04-26T18:05:50.346515655Z File "/opt/frigate/frigate/video.py", line 212, in track_camera
2020-04-26T18:05:50.346532947Z frame_time = frame_queue.get()
2020-04-26T18:05:50.346548948Z File "/usr/lib/python3.7/multiprocessing/queues.py", line 352, in get
2020-04-26T18:05:50.346565990Z res = self._reader.recv_bytes()
2020-04-26T18:05:50.346581956Z File "/usr/lib/python3.7/multiprocessing/connection.py", line 216, in recv_bytes
2020-04-26T18:05:50.346599136Z buf = self._recv_bytes(maxlength)
2020-04-26T18:05:50.346614991Z File "/usr/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
2020-04-26T18:05:50.346632241Z buf = self._recv(4)
2020-04-26T18:05:50.346647956Z File "/usr/lib/python3.7/multiprocessing/connection.py", line 379, in _recv
2020-04-26T18:05:50.346665110Z chunk = read(handle, remaining)
It seems that the kitchen camera started working as soon as I triggered the printstack for the PID… interesting…
Can you set one of them to log_level info instead of panic?
Just one or both?
New config:
ffmpeg:
global_args:
- -hide_banner
- -loglevel
- info
hwaccel_args:
- -hwaccel
- vaapi
- -hwaccel_device
- /dev/dri/renderD128
- -hwaccel_output_format
- yuv420p
input_args:
- -avoid_negative_ts
- make_zero
- -fflags
- nobuffer
- -flags
- low_delay
- -strict
- experimental
- -fflags
- +genpts+discardcorrupt
- -vsync
- drop
- -use_wallclock_as_timestamps
- '1'
output_args:
- -f
- rawvideo
- -pix_fmt
- rgb24
It will be easier to see what is happening with just one.
Done. I’ll report back with better logs the next time it stops working.
Hi,
Been running 0.5.0 for about 2 months. besides the freezes every few hours (easily solved by an automation that restarts the docker on sensor.frigate_coral_detection_queue above a value).
I still have a major issue with false detections.
I didn’t understand how to “teach” the system from the manual that you’ve previously referred me to.
I do have an idea - if you look at the image below you can see that it detects the dog in a red box as a dog and also as a person in a blue box. one on top of the other.
any way to have a rule that remove the “person” detection if both are in the same area?
Many thanks for the hard work!
@blakeblackshear What would you find useful to determine why the restarts are so prevalent with the Wyze camera but doesn’t seem to affect medium to higher end cameras?
My Wyze cam pid is at 5990 while my frontdoor camera is still at 25 which shows that the Wyze is rebooting a lot more.
Is the restart_delay still used and configurable?
If it’s a timeout thing, can frigate be used with a UDP stream?
That’s definitely something I could use as a heuristic to filter. Any chance you can capture a video clip that I can use for testing?
I ended removing the restart delay when fixing something else. From what I have seen on the Blue Iris forums, the Wyzecam’s just have a really unstable RTSP feed. I don’t know if anyone has a fix. You could try looking around for an example of ffmpeg parameters that are more stable.
That output was saved when it was running about three minutes behind (hard to see, but you can see in the screenshot if you want proof; the timestamp from Ubiquiti and then the timestamp from Frigate).
I restarted the docker container after posting (about ten hours ago) and it’s back to three minutes behind now again.
I’ll see if I can get another camera feed going from a different device and see if it has the same issue.
Hi all,
Just started implementing this. So far it’s been working as expected.
What is the best way to capture the best jpg and copy it into the www folder so I can include it in an iOS notification?
I tried checking the binary sensor for person detection and then using the camera snapshot service to get copy an image into the www folder. Although this works, it can end up sending an old image due to the delay in capturing the snapshot and sending the notification.
How are you guys achieving this?
Cheers.
Here’s 2 clips and images from this morning:
https://drive.google.com/drive/folders/1eUwkVrXH72LYK4Ht0w0ARWgz3IcIXTos
Captured with Blue Iris without re-encoding, so is should be the same stuff that frigate sees.
EDIT: added another clip from a different camera and 3 pics from that vid detection.
1 Like
I certainly believe you. The time that frigate displays on the image is the time that it captured the image from ffmpeg, not the time it finished processing. If that timestamp is old, then I think something is causing ffmpeg to get behind when capturing the video feed. Can you look at CPU usage and network I/O to see if something is maxed out?