I haven’t been able to reproduce this intentionally yet, but I did see it in real world once. It took 5 days for my detection queue to get in a bad state, and the only way to fix it was to completely restart the container. I have been running a modified version for 12 hours that randomly triggers failures of each camera and the detection process frequently. It is processing 4 simulated cameras at 50fps each (200fps total), and it hasn’t failed yet. I am going to try changing some of the failure handling in the detection process, but I won’t know if it works until others try it.
2 Likes
Hi @blakeblackshear. In response to your previous message above about capturing video. My wife and I were awakeNed last night due to an errant spider that was 96% a person. My sizes were not applicable because it was so close to the lens it looked much bigger than it was. It really looks nothing like a human. More of a moving white dot since the IR was reflecting off of it. I captured a video clip. Is there somewhere I can send it to you?
You can upload to google drive and send me a link as a private message.
I’ve tried both directions with the W and H values with no change.
doorway:
ffmpeg:
input: rtsp://viewer:[email protected]:554/Streaming/Channels/401
output_args:
- -vf
# - mpdecimate,rotate=PI/2
- "transpose=1"
- -f
- rawvideo
- -pix_fmt
- rgb24
# height: 480
# width: 640
height: 640
width: 480
take_frame: 1
fps: 5
snapshots:
show_timestamp: True
objects:
track:
- person
filters:
person:
min_area: 5000
max_area: 100000
threshold: 0.5
Your width and height values are indented one level too far. They should line up with take_frame.
Thank you, that was the ticket. Shall I assume this pic is the best I can hope for with the horizontal being stretched? At least I can see all of the top and bottom now. I tried narrowing the width but the interlacing problem returns. I’m assuming it’s an aspect ratio issue.
output_args:
- -vf
# - mpdecimate,rotate=PI/2
- "transpose=1"
- -f
- rawvideo
- -pix_fmt
- rgb24
height: 1920
width: 1080
For now, yes. It won’t impact the detection, but the preview is currently hard coded to 16:9. The next RC will fix that.
1 Like
Thank you for all your help. I really appreciate the work you’ve put into this awesome project.
I’m making some headway. It looks like the way I was checking to see if the detection process was stuck was too aggressive, and I was getting some false positives. I think that was causing the detection process to eventually be killed while it was in the middle of pulling something from the queue. That could leave a lingering lock on the queue object and prevent it from fetching anything when restarted. It seems consistent with the behavior I have seen. Testing a fix now. 
5 Likes
Just published 0.5.1-rc4, which I hope will fix the issue that causes the detection process to stop. Here is the change log:
- Resize preview according to camera’s aspect ratio
- Discard plasma store logs
- Fix check for detection process hanging
3 Likes
Love the work you’ve done! For me, on all versions, it seems that if there’s motion detected for too long, the object detection and the video feed from Frigate gets progressively further behind, until the feed from Frigate is maybe 15 minutes delayed from the actual video feed. Object detection still works - it just gets pretty choppy and delayed. Interestingly there’s not really any motion typically, but it detects a parked car as motion, for example, and keeps up the detection boxes on it.
Here’s the debug output from just a few minutes ago where it is doing this.
{"back_door":{"camera_fps":10.0,"detection_fps":0.0,"ffmpeg_pid":180,"frame_info":{"detect":1587914831.704363,"process":1587914831.499066,"read":1587914832.365871},"pid":116,"process_fps":4.8,"read_start":0.0,"skipped_fps":4.8},"coral":{"detection_start":0.0,"fps":9.0,"inference_speed":7.78,"pid":30},"driveway":{"camera_fps":10.3,"detection_fps":2.4,"ffmpeg_pid":93,"frame_info":{"detect":1587914829.133347,"process":1587914829.128108,"read":1587914832.339407},"pid":119,"process_fps":2.4,"read_start":0.0,"skipped_fps":7.8},"front_door":{"camera_fps":10.4,"detection_fps":0.0,"ffmpeg_pid":53,"frame_info":{"detect":1587914831.055296,"process":1587914830.826594,"read":1587914832.138246},"pid":117,"process_fps":4.8,"read_start":0.0,"skipped_fps":4.8},"front_yard":{"camera_fps":10.7,"detection_fps":6.6,"ffmpeg_pid":65,"frame_info":{"detect":1587914828.235891,"process":1587914827.38679,"read":1587914832.138669},"pid":118,"process_fps":2.4,"read_start":0.0,"skipped_fps":9.0},"garage":{"camera_fps":10.0,"detection_fps":0.0,"ffmpeg_pid":114,"frame_info":{"detect":1587914831.930109,"process":1587914831.283865,"read":1587914832.36541},"pid":120,"process_fps":4.7,"read_start":0.0,"skipped_fps":5.1},"plasma_store_rc":null}
Pulling video from Ubiquiti cameras at medium resolution, 10fps.
Running on a Coral and a fairly beefy host (16gb RAM, i7).
Any ideas?
Have you customized the ffmpeg parameters? Also, just to clarify, frigate isn’t detecting your car as motion. It continues to track objects that it has found until they disappear, even if they have stopped moving.
Ah, good to know.
Here are my ffmpeg parameters:
ffmpeg:
global_args:
- -hide_banner
- -loglevel
- panic
hwaccel_args:
- -hwaccel
- vaapi
- -hwaccel_device
- /dev/dri/renderD128
- -hwaccel_output_format
- yuv420p
input_args:
- -avoid_negative_ts
- make_zero
- -fflags
- nobuffer
- -flags
- low_delay
- -strict
- experimental
- -fflags
- +genpts+discardcorrupt
- -vsync
- drop
- -rtsp_transport
- tcp
- -stimeout
- '5000000'
- -use_wallclock_as_timestamps
- '1'
output_args:
- -f
- rawvideo
- -pix_fmt
- rgb24
1 Like
I don’t see anything that would be an issue in your ffmpeg params. Based on your /debug/stats output, frigate thinks it is showing you an image that is 5 seconds behind for front_yard. Was that output saved when it was 15 minutes behind? You can see that under frame_info. The read value is when frigate read the frame from ffmpeg, the detect value is the frame it is running detection on, and the process value is the time of the frame it is processing after detection.
Hi @blakeblackshear. I started running the latest RC when you posted it last (0.5.1-rc4). Thanks again for digging into this and attempted to debug our problems.
Unfortunately it looks like Frigate is still freezing up for my Reolink cameras (both different models). 
Both cameras stopped responding at the same time inside Frigate. The logs show they never restarted. The logs don’t seem to indicate any crashes. Here’s the entire log with no entries for today when they stopped working.
logs:
2020-04-25T17:08:22.251700217Z On connect called
2020-04-25T17:08:23.266945018Z ffprobe -v panic -show_error -show_streams -of json "rtmp://192.168.0.51:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword"
2020-04-25T17:08:23.267521749Z Starting detection process: 30
2020-04-25T17:08:28.105496983Z {'streams': [{'index': 0, 'codec_name': 'aac', 'codec_long_name': 'AAC (Advanced Audio Coding)', 'profile': 'LC', 'codec_type': 'audio', 'codec_time_base': '1/16000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 'fltp', 'sample_rate': '16000', 'channels': 1, 'channel_layout': 'mono', 'bits_per_sample': 0, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/1000', 'start_pts': 40140194, 'start_time': '40140.194000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'High', 'codec_type': 'video', 'codec_time_base': '1/18', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 2560, 'height': 1920, 'coded_width': 2560, 'coded_height': 1920, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:1', 'pix_fmt': 'yuv420p', 'level': 51, 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'true', 'nal_length_size': '4', 'r_frame_rate': '10/1', 'avg_frame_rate': '9/1', 'time_base': '1/1000', 'start_pts': 40140145, 'start_time': '40140.145000', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
2020-04-25T17:08:28.106092668Z Creating ffmpeg process...
2020-04-25T17:08:28.106111889Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.0.51:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-04-25T17:08:28.111635821Z ffprobe -v panic -show_error -show_streams -of json "rtmp://192.168.0.50:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword"
2020-04-25T17:08:33.795026589Z {'streams': [{'index': 0, 'codec_name': 'aac', 'codec_long_name': 'AAC (Advanced Audio Coding)', 'profile': 'LC', 'codec_type': 'audio', 'codec_time_base': '1/16000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 'fltp', 'sample_rate': '16000', 'channels': 1, 'channel_layout': 'mono', 'bits_per_sample': 0, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/1000', 'start_pts': 558571675, 'start_time': '558571.675000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'High', 'codec_type': 'video', 'codec_time_base': '1/22', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 2560, 'height': 1440, 'coded_width': 2560, 'coded_height': 1440, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:1', 'pix_fmt': 'yuv420p', 'level': 51, 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'true', 'nal_length_size': '4', 'r_frame_rate': '30/1', 'avg_frame_rate': '11/1', 'time_base': '1/1000', 'start_pts': 558571559, 'start_time': '558571.559000', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
2020-04-25T17:08:33.795328220Z Creating ffmpeg process...
2020-04-25T17:08:33.795336441Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.0.50:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-04-25T17:08:33.800437572Z Camera_process started for front_window: 52
2020-04-25T17:08:33.801123517Z Starting process for front_window: 52
2020-04-25T17:08:33.802488776Z Camera_process started for kitchen: 53
2020-04-25T17:08:33.803659286Z Starting process for kitchen: 53
2020-04-25T17:08:33.810045277Z * Serving Flask app "detect_objects" (lazy loading)
2020-04-25T17:08:33.810061082Z * Environment: production
2020-04-25T17:08:33.810063551Z WARNING: This is a development server. Do not use it in a production deployment.
2020-04-25T17:08:33.810066919Z Use a production WSGI server instead.
2020-04-25T17:08:33.810068925Z * Debug mode: off
/debug/stats/
{
"coral": {
"detection_start": 0,
"fps": 7.9,
"inference_speed": 7.72,
"pid": 30
},
"front_window": {
"camera_fps": 0,
"detection_fps": 4.6,
"ffmpeg_pid": 41,
"frame_info": {
"detect": 1587880762.259035,
"process": 1587880762.259035,
"read": 1587880762.259035
},
"pid": 52,
"process_fps": 5.1,
"read_start": 1587880762.271842,
"skipped_fps": 0
},
"kitchen": {
"camera_fps": 0,
"detection_fps": 3.3,
"ffmpeg_pid": 50,
"frame_info": {
"detect": 1587880737.855407,
"process": 1587880737.855407,
"read": 1587880737.886853
},
"pid": 53,
"process_fps": 6,
"read_start": 1587880737.867302,
"skipped_fps": 0
},
"plasma_store_rc": null
}
Debug output PID 50 (FFMPEG)
2020-04-26T18:05:24.340835104Z kitchen: ffmpeg didnt return a frame. something is wrong.
2020-04-26T18:05:24.341003117Z kitchen: ffmpeg process is not running. exiting capture thread...
2020-04-26T18:05:30.301681938Z Creating ffmpeg process...
2020-04-26T18:05:30.301735236Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.0.50:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
Debug output PID 53 (Kitchen Camera)
2020-04-26T18:05:50.346067806Z File "detect_objects.py", line 348, in <module>
2020-04-26T18:05:50.346158214Z main()
2020-04-26T18:05:50.346183440Z File "detect_objects.py", line 227, in main
2020-04-26T18:05:50.346204751Z camera_process['process'].start()
2020-04-26T18:05:50.346225774Z File "/usr/lib/python3.7/multiprocessing/process.py", line 112, in start
2020-04-26T18:05:50.346245616Z self._popen = self._Popen(self)
2020-04-26T18:05:50.346264390Z File "/usr/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
2020-04-26T18:05:50.346284233Z return _default_context.get_context().Process._Popen(process_obj)
2020-04-26T18:05:50.346303475Z File "/usr/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
2020-04-26T18:05:50.346323334Z return Popen(process_obj)
2020-04-26T18:05:50.346381611Z File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
2020-04-26T18:05:50.346400594Z self._launch(process_obj)
2020-04-26T18:05:50.346416410Z File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
2020-04-26T18:05:50.346433626Z code = process_obj._bootstrap()
2020-04-26T18:05:50.346449970Z File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
2020-04-26T18:05:50.346466987Z self.run()
2020-04-26T18:05:50.346482738Z File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
2020-04-26T18:05:50.346499530Z self._target(*self._args, **self._kwargs)
2020-04-26T18:05:50.346515655Z File "/opt/frigate/frigate/video.py", line 212, in track_camera
2020-04-26T18:05:50.346532947Z frame_time = frame_queue.get()
2020-04-26T18:05:50.346548948Z File "/usr/lib/python3.7/multiprocessing/queues.py", line 352, in get
2020-04-26T18:05:50.346565990Z res = self._reader.recv_bytes()
2020-04-26T18:05:50.346581956Z File "/usr/lib/python3.7/multiprocessing/connection.py", line 216, in recv_bytes
2020-04-26T18:05:50.346599136Z buf = self._recv_bytes(maxlength)
2020-04-26T18:05:50.346614991Z File "/usr/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
2020-04-26T18:05:50.346632241Z buf = self._recv(4)
2020-04-26T18:05:50.346647956Z File "/usr/lib/python3.7/multiprocessing/connection.py", line 379, in _recv
2020-04-26T18:05:50.346665110Z chunk = read(handle, remaining)
It seems that the kitchen camera started working as soon as I triggered the printstack for the PID… interesting…
Can you set one of them to log_level info instead of panic?
Just one or both?
New config:
ffmpeg:
global_args:
- -hide_banner
- -loglevel
- info
hwaccel_args:
- -hwaccel
- vaapi
- -hwaccel_device
- /dev/dri/renderD128
- -hwaccel_output_format
- yuv420p
input_args:
- -avoid_negative_ts
- make_zero
- -fflags
- nobuffer
- -flags
- low_delay
- -strict
- experimental
- -fflags
- +genpts+discardcorrupt
- -vsync
- drop
- -use_wallclock_as_timestamps
- '1'
output_args:
- -f
- rawvideo
- -pix_fmt
- rgb24
It will be easier to see what is happening with just one.
Done. I’ll report back with better logs the next time it stops working.