Hey @dave122. The cv error was fixed by changing the frigate launch command I had in my config to use python3.7. The second error was resolved by removing an incorrect attempt at passing sham-size. I’m actually not doing it at all now, but it seems to be ok with 3 cameras.
Awesome! The base one is 25 from microcenter. Another $10 contributor and we should be good.
I’m seeing these messages too
running 0.5.0 final
frigate-new | /arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 29 objects to free up 80201472 bytes. The number of bytes in use (before this eviction) is 398512640.
frigate-new | /arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 29 objects to free up 80201472 bytes. The number of bytes in use (before this eviction) is 398512640.
frigate-new | /arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 29 objects to free up 80201472 bytes. The number of bytes in use (before this eviction) is 398512640.
frigate-new | /arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 29 objects to free up 80201472 bytes. The number of bytes in use (before this eviction) is 398512640.
ffmpeg string from logs
frigate-new | ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.x.y:8554/unicast -f rawvideo -pix_fmt rgb24 pipe:
I also had my queue fill up and repeated restarts. I’ll gather that data if it happens again.
1 Like
BTW, I also noticed that this one is less resilient when the laptop where it is running goes to sleep and then wakes up later again.
I know that’s not usually something you want to happen (rather just run it permanently), but in my case the power utility is experiencing a lot of problems and power goes down fairly often. The notebook has a fairly ok battery but I’ve set it to go to sleep after 5 mins on battery power.
When the power comes back on, I start up the laptop again using a Wakeup on Lan switch in HA.
Frigate used to resume fine when the power came back on, but on 0.5.0 it usually fails to start again.
I got my Wyze cam today. Thanks to those who chipped in for one. The good news is that I can easily reproduce the issue. It does appear to be related the the amount of motion. I am guessing the Wyze cam is struggling to compute the P/B differential frames when enough of the image is changing. I just need to find the ffmpeg parameters to get it to skip the corrupt frames and move on.
5 Likes
Interesting! I have root access to my wyze if you have any questions about it. (dafang hacks)
With an older dev image (from Sunday night?); It ran for ~4 days before it stopped working (on both cameras stopped responding around the same time). That’s some pretty great progress! I’ll pull down the latest DEV image and restart it. In the meantime, here were my findings from that run.
I can see from the logs the ffmpeg stream restarted quite a few times before Frigate stopped processing today. If I poke at the debug stream for each camera (before restarting the container), I can see it’s still occasionally picking a new frame. But maybe 1 frame every 10-30 minutes or so. Maybe a queue issue again? Or should my shm_size be larger than 1g? Any clues into what I can help debug further?
I can see from some of the container stats that it’s not maxing out any resources that I can tell…
Here’s a shot of the metrics for the cameras and coral:
I ran the /debug/print_stack?pid=X on both camera procs. They’re at the bottom of the last log.
/debug/stats
{
"coral": {
"detection_start": 0,
"fps": 0,
"inference_speed": 10.11,
"pid": 18341
},
"front_window": {
"detection_fps": 0,
"ffmpeg_pid": 22414,
"fps": 0.1,
"pid": 30,
"read_start": 0,
"skipped_fps": 0
},
"kitchen": {
"detection_fps": 0,
"ffmpeg_pid": 22736,
"fps": 0.1,
"pid": 31,
"read_start": 0,
"skipped_fps": 0
},
"plasma_store_rc": null
}
/debug/stack
File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap self._bootstrap_inner()
File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run()
File "/opt/frigate/frigate/object_processing.py", line 52, in run camera, frame_time, tracked_objects = self.tracked_objects_queue.get()
File "/usr/lib/python3.7/multiprocessing/queues.py", line 352, in get res = self._reader.recv_bytes()
File "/usr/lib/python3.7/multiprocessing/connection.py", line 216, in recv_bytes buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4)
File "/usr/lib/python3.7/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining)
Full Container Logs
2020-03-09T00:00:28.684583310Z On connect called
2020-03-09T00:00:28.845744739Z /arrow/cpp/src/plasma/store.cc:1226: Allowing the Plasma store to use up to 0.4GB of memory.
2020-03-09T00:00:28.845761221Z /arrow/cpp/src/plasma/store.cc:1253: Starting object store with directory /dev/shm and huge page support disabled
2020-03-09T00:00:29.695895741Z Starting detection process: 29
2020-03-09T00:00:29.697530812Z Camera_process started for front_window: 30
2020-03-09T00:00:29.698378376Z Starting process for front_window: 30
2020-03-09T00:00:29.698625786Z ffprobe -v panic -show_error -show_streams -of json "rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword"
2020-03-09T00:00:29.699590846Z Camera_process started for kitchen: 31
2020-03-09T00:00:29.700426523Z Starting process for kitchen: 31
2020-03-09T00:00:29.700668634Z ffprobe -v panic -show_error -show_streams -of json "rtmp://192.168.10.1:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword"
2020-03-09T00:00:29.706250467Z * Serving Flask app "detect_objects" (lazy loading)
2020-03-09T00:00:29.706268214Z * Environment: production
2020-03-09T00:00:29.706271023Z WARNING: This is a development server. Do not use it in a production deployment.
2020-03-09T00:00:29.706273213Z Use a production WSGI server instead.
2020-03-09T00:00:29.706275199Z * Debug mode: off
2020-03-09T00:00:34.528751525Z {'streams': [{'index': 0, 'codec_name': 'aac', 'codec_long_name': 'AAC (Advanced Audio Coding)', 'profile': 'LC', 'codec_type': 'audio', 'codec_time_base': '1/16000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 'fltp', 'sample_rate': '16000', 'channels': 1, 'channel_layout': 'mono', 'bits_per_sample': 0, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/1000', 'start_pts': 666052217, 'start_time': '666052.217000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'High', 'codec_type': 'video', 'codec_time_base': '1/24', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 2560, 'height': 1440, 'coded_width': 2560, 'coded_height': 1440, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:1', 'pix_fmt': 'yuv420p', 'level': 51, 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'true', 'nal_length_size': '4', 'r_frame_rate': '30/1', 'avg_frame_rate': '12/1', 'time_base': '1/1000', 'start_pts': 666052168, 'start_time': '666052.168000', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
2020-03-09T00:00:34.542840043Z Creating ffmpeg process...
2020-03-09T00:00:34.542907409Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.1:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-09T00:00:35.240962697Z {'streams': [{'index': 0, 'codec_name': 'aac', 'codec_long_name': 'AAC (Advanced Audio Coding)', 'profile': 'LC', 'codec_type': 'audio', 'codec_time_base': '1/16000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 'fltp', 'sample_rate': '16000', 'channels': 1, 'channel_layout': 'mono', 'bits_per_sample': 0, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/1000', 'start_pts': 147645780, 'start_time': '147645.780000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'High', 'codec_type': 'video', 'codec_time_base': '1/20', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 2560, 'height': 1920, 'coded_width': 2560, 'coded_height': 1920, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:1', 'pix_fmt': 'yuv420p', 'level': 51, 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'true', 'nal_length_size': '4', 'r_frame_rate': '10/1', 'avg_frame_rate': '10/1', 'time_base': '1/1000', 'start_pts': 147645726, 'start_time': '147645.726000', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
2020-03-09T00:00:35.244983861Z Creating ffmpeg process...
2020-03-09T00:00:35.245000680Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-09T05:59:57.509423722Z Process for front_window has been reading from ffmpeg for over 10 seconds long. Killing ffmpeg...
2020-03-09T06:00:07.520516639Z Unable to kill ffmpeg with pid 62
2020-03-09T06:00:07.527794890Z front_window: ffmpeg_process didnt return any bytes
2020-03-09T06:00:07.527811521Z front_window: ffmpeg_process exited unexpectedly with -9
2020-03-09T06:00:07.527814438Z Letting front_window rest for 0 seconds before restarting...
2020-03-09T06:00:07.527816815Z Terminating the existing ffmpeg process...
2020-03-09T06:00:07.527819169Z Waiting for ffmpeg to exit gracefully...
2020-03-09T06:00:07.528567081Z Creating ffmpeg process...
2020-03-09T06:00:07.528574740Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-09T06:00:17.542675373Z front_window: ffmpeg_process didnt return any bytes
2020-03-09T06:00:17.542809947Z front_window: ffmpeg_process exited unexpectedly with 1
2020-03-09T06:00:17.542887125Z Letting front_window rest for 0 seconds before restarting...
2020-03-09T06:00:17.542915923Z Terminating the existing ffmpeg process...
2020-03-09T06:00:17.543092004Z Waiting for ffmpeg to exit gracefully...
2020-03-09T06:00:17.543920859Z Creating ffmpeg process...
2020-03-09T06:00:17.543982515Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-09T06:00:27.566587571Z front_window: ffmpeg_process didnt return any bytes
2020-03-09T06:00:27.566654413Z front_window: ffmpeg_process exited unexpectedly with 1
2020-03-09T06:00:27.566662114Z Letting front_window rest for 0 seconds before restarting...
2020-03-09T06:00:27.566669393Z Terminating the existing ffmpeg process...
2020-03-09T06:00:27.566673917Z Waiting for ffmpeg to exit gracefully...
2020-03-09T06:00:27.567068987Z Creating ffmpeg process...
2020-03-09T06:00:27.567082561Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
##################
## RETRIES TRUNCATED HERE TO FIT UNDER POST CHARACTER LIMIT
##################
2020-03-13T17:27:36.728691222Z kitchen: ffmpeg_process didnt return any bytes
2020-03-13T17:27:36.728713864Z kitchen: ffmpeg_process exited unexpectedly with 0
2020-03-13T17:27:36.728716824Z Letting kitchen rest for 0 seconds before restarting...
2020-03-13T17:27:36.728719122Z Terminating the existing ffmpeg process...
2020-03-13T17:27:36.728721379Z Waiting for ffmpeg to exit gracefully...
2020-03-13T17:27:36.729262360Z Creating ffmpeg process...
2020-03-13T17:27:36.729268169Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.1:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-13T17:40:27.801771592Z kitchen: ffmpeg_process didnt return any bytes
2020-03-13T17:40:27.801790471Z kitchen: ffmpeg_process exited unexpectedly with 0
2020-03-13T17:40:27.801793197Z Letting kitchen rest for 0 seconds before restarting...
2020-03-13T17:40:27.801795437Z Terminating the existing ffmpeg process...
2020-03-13T17:40:27.801797718Z Waiting for ffmpeg to exit gracefully...
2020-03-13T17:40:27.802360809Z Creating ffmpeg process...
2020-03-13T17:40:27.802371151Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.1:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-13T17:40:47.623146333Z front_window: ffmpeg_process didnt return any bytes
2020-03-13T17:40:47.623163565Z front_window: ffmpeg_process exited unexpectedly with 0
2020-03-13T17:40:47.623166492Z Letting front_window rest for 0 seconds before restarting...
2020-03-13T17:40:47.623168993Z Terminating the existing ffmpeg process...
2020-03-13T17:40:47.623171441Z Waiting for ffmpeg to exit gracefully...
2020-03-13T17:40:47.623787863Z Creating ffmpeg process...
2020-03-13T17:40:47.623797735Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-13T17:52:08.695270471Z front_window: ffmpeg_process exited unexpectedly with 0
2020-03-13T17:52:08.695302177Z Letting front_window rest for 0 seconds before restarting...
2020-03-13T17:52:08.695307373Z Terminating the existing ffmpeg process...
2020-03-13T17:52:08.695311441Z Waiting for ffmpeg to exit gracefully...
2020-03-13T17:52:08.696290440Z Creating ffmpeg process...
2020-03-13T17:52:08.696305924Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-13T17:58:09.416486269Z kitchen: ffmpeg_process didnt return any bytes
2020-03-13T17:58:09.416507264Z kitchen: ffmpeg_process exited unexpectedly with 0
2020-03-13T17:58:09.416510142Z Letting kitchen rest for 0 seconds before restarting...
2020-03-13T17:58:09.416512448Z Terminating the existing ffmpeg process...
2020-03-13T17:58:09.416514954Z Waiting for ffmpeg to exit gracefully...
2020-03-13T17:58:09.417035131Z Creating ffmpeg process...
2020-03-13T17:58:09.417046508Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.1:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-13T18:03:49.785937612Z front_window: ffmpeg_process didnt return any bytes
2020-03-13T18:03:49.785965868Z front_window: ffmpeg_process exited unexpectedly with 0
2020-03-13T18:03:49.785973755Z Letting front_window rest for 0 seconds before restarting...
2020-03-13T18:03:49.785980200Z Terminating the existing ffmpeg process...
2020-03-13T18:03:49.785985779Z Waiting for ffmpeg to exit gracefully...
2020-03-13T18:03:49.786859275Z Creating ffmpeg process...
2020-03-13T18:03:49.786873776Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-13T18:16:41.753900823Z File "/usr/lib/python3.7/threading.py", line 890, in _bootstrap
2020-03-13T18:16:41.753933432Z self._bootstrap_inner()
2020-03-13T18:16:41.753942900Z File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
2020-03-13T18:16:41.753945291Z self.run()
2020-03-13T18:16:41.753947265Z File "detect_objects.py", line 82, in run
2020-03-13T18:16:41.753949250Z self.tflite_process.start_or_restart()
2020-03-13T18:16:41.753951185Z File "/opt/frigate/frigate/edgetpu.py", line 109, in start_or_restart
2020-03-13T18:16:41.753953121Z self.detect_process.start()
2020-03-13T18:16:41.753954971Z File "/usr/lib/python3.7/multiprocessing/process.py", line 112, in start
2020-03-13T18:16:41.753956926Z self._popen = self._Popen(self)
2020-03-13T18:16:41.753958782Z File "/usr/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
2020-03-13T18:16:41.753960706Z return _default_context.get_context().Process._Popen(process_obj)
2020-03-13T18:16:41.753962634Z File "/usr/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
2020-03-13T18:16:41.753964564Z return Popen(process_obj)
2020-03-13T18:16:41.753966390Z File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
2020-03-13T18:16:41.753968330Z self._launch(process_obj)
2020-03-13T18:16:41.753970189Z File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
2020-03-13T18:16:41.753972816Z code = process_obj._bootstrap()
2020-03-13T18:16:41.753974688Z File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
2020-03-13T18:16:41.753976630Z self.run()
2020-03-13T18:16:41.753978481Z File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
2020-03-13T18:16:41.753980387Z self._target(*self._args, **self._kwargs)
2020-03-13T18:16:41.753982223Z File "/opt/frigate/frigate/edgetpu.py", line 72, in run_detector
2020-03-13T18:16:41.753984133Z object_id_str = detection_queue.get()
2020-03-13T18:16:41.753986022Z File "/usr/lib/python3.7/multiprocessing/queues.py", line 351, in get
2020-03-13T18:16:41.753987969Z with self._rlock:
2020-03-13T18:16:41.753989813Z File "/usr/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
2020-03-13T18:16:41.753991777Z return self._semlock.__enter__()
2020-03-13T18:16:53.562741671Z File "detect_objects.py", line 302, in <module>
2020-03-13T18:16:53.562785601Z main()
2020-03-13T18:16:53.562795009Z File "detect_objects.py", line 187, in main
2020-03-13T18:16:53.562802910Z camera_process['process'].start()
2020-03-13T18:16:53.562810763Z File "/usr/lib/python3.7/multiprocessing/process.py", line 112, in start
2020-03-13T18:16:53.562818806Z self._popen = self._Popen(self)
2020-03-13T18:16:53.562826000Z File "/usr/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
2020-03-13T18:16:53.562852620Z return _default_context.get_context().Process._Popen(process_obj)
2020-03-13T18:16:53.562861038Z File "/usr/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
2020-03-13T18:16:53.562868751Z return Popen(process_obj)
2020-03-13T18:16:53.562875833Z File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
2020-03-13T18:16:53.562883289Z self._launch(process_obj)
2020-03-13T18:16:53.562890467Z File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
2020-03-13T18:16:53.562897976Z code = process_obj._bootstrap()
2020-03-13T18:16:53.562905066Z File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
2020-03-13T18:16:53.562912643Z self.run()
2020-03-13T18:16:53.562919762Z File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
2020-03-13T18:16:53.562927252Z self._target(*self._args, **self._kwargs)
2020-03-13T18:16:53.562934498Z File "/opt/frigate/frigate/video.py", line 296, in track_camera
2020-03-13T18:16:53.562942013Z region_detections = object_detector.detect(tensor_input)
2020-03-13T18:16:53.562949310Z File "/opt/frigate/frigate/edgetpu.py", line 126, in detect
2020-03-13T18:16:53.562956875Z self.detection_queue.put(now)
2020-03-13T18:16:53.562963948Z File "/usr/lib/python3.7/multiprocessing/queues.py", line 364, in put
2020-03-13T18:16:53.562971560Z self._writer.send_bytes(obj)
2020-03-13T18:16:53.562978777Z File "/usr/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
2020-03-13T18:16:53.562986312Z self._send_bytes(m[offset:offset + size])
2020-03-13T18:16:53.562993532Z File "/usr/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
2020-03-13T18:16:53.563001035Z self._send(header + buf)
2020-03-13T18:16:53.563008090Z File "/usr/lib/python3.7/multiprocessing/connection.py", line 368, in _send
2020-03-13T18:16:53.563015587Z n = write(self._handle, buf)
For my issue (wyze+dafang on 0.5.0 w/ coral)
In addition to the not enough space logs I get regular pretty regular restarts but things still seem to work. However, after a day or so there will be a catastrophic error, the queue will build, ffmpeg continues to restart, detection/processing no longer reliably works.The debug stream is either frozen, very lagged. or not ‘streaming’ anymore The behavior may be based on how ffmpeg is behaving at any particular time.
At the timestamp the queue starts to build I have this log message:
frigate-new | 2020-03-13T13:29:36.211272424Z /arrow/cpp/src/plasma/store.cc:738: Disconnecting client on fd 6
context:
frigate-new | 2020-03-13T10:34:41.733546488Z Waiting for ffmpeg to exit gracefully...
frigate-new | 2020-03-13T10:34:41.733551618Z Creating ffmpeg process...
frigate-new | 2020-03-13T10:34:41.733557083Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.1.185:8554/unicast -f rawvideo -pix_fmt rgb24 pipe:
frigate-new | 2020-03-13T10:35:08.839818315Z front: ffmpeg_process is still running but didnt return any bytes
frigate-new | 2020-03-13T10:35:08.839861314Z front: ffmpeg_process exited unexpectedly with 0
frigate-new | 2020-03-13T10:35:08.839872063Z Terminating the existing ffmpeg process...
frigate-new | 2020-03-13T10:35:08.839884660Z Waiting for ffmpeg to exit gracefully...
frigate-new | 2020-03-13T10:35:08.839892406Z Creating ffmpeg process...
frigate-new | 2020-03-13T10:35:08.839908095Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.1.185:8554/unicast -f rawvideo -pix_fmt rgb24 pipe:
frigate-new | 2020-03-13T11:38:54.441626279Z front: ffmpeg_process is still running but didnt return any bytes
frigate-new | 2020-03-13T11:38:54.441690945Z front: ffmpeg_process is still running but didnt return any bytes
frigate-new | 2020-03-13T11:38:54.441753221Z front: ffmpeg_process exited unexpectedly with 0
frigate-new | 2020-03-13T11:38:54.441774972Z Terminating the existing ffmpeg process...
frigate-new | 2020-03-13T11:38:54.441783875Z Waiting for ffmpeg to exit gracefully...
frigate-new | 2020-03-13T11:38:54.441789992Z Creating ffmpeg process...
frigate-new | 2020-03-13T11:38:54.441803368Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.1.185:8554/unicast -f rawvideo -pix_fmt rgb24 pipe:
frigate-new | 2020-03-13T11:58:01.344993425Z front: ffmpeg_process is still running but didnt return any bytes
frigate-new | 2020-03-13T11:58:01.345053328Z front: ffmpeg_process exited unexpectedly with 0
frigate-new | 2020-03-13T11:58:01.345062966Z Terminating the existing ffmpeg process...
frigate-new | 2020-03-13T11:58:01.345069237Z Waiting for ffmpeg to exit gracefully...
frigate-new | 2020-03-13T11:58:01.345078811Z Creating ffmpeg process...
frigate-new | 2020-03-13T11:58:01.345100311Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.1.185:8554/unicast -f rawvideo -pix_fmt rgb24 pipe:
frigate-new | 2020-03-13T12:39:23.136307216Z front: ffmpeg_process is still running but didnt return any bytes
frigate-new | 2020-03-13T12:39:23.136397392Z front: ffmpeg_process is still running but didnt return any bytes
frigate-new | 2020-03-13T12:39:23.136533062Z front: ffmpeg_process exited unexpectedly with 0
frigate-new | 2020-03-13T12:39:23.136566508Z Terminating the existing ffmpeg process...
frigate-new | 2020-03-13T12:39:23.136586930Z Waiting for ffmpeg to exit gracefully...
frigate-new | 2020-03-13T12:39:23.136606690Z Creating ffmpeg process...
frigate-new | 2020-03-13T12:39:23.136639169Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.1.185:8554/unicast -f rawvideo -pix_fmt rgb24 pipe:
frigate-new | 2020-03-13T13:29:36.199481805Z Detection appears to be stuck. Restarting detection process
frigate-new | 2020-03-13T13:29:36.206900948Z Waiting for detection process to exit gracefully...
frigate-new | 2020-03-13T13:29:36.211272424Z /arrow/cpp/src/plasma/store.cc:738: Disconnecting client on fd 6
frigate-new | 2020-03-13T13:29:36.216170002Z Starting detection process: 8097
frigate-new | 2020-03-13T13:33:38.289053672Z front: ffmpeg_process is still running but didnt return any bytes
frigate-new | 2020-03-13T13:33:38.289094072Z front: ffmpeg_process exited unexpectedly with 0
frigate-new | 2020-03-13T13:33:38.289104869Z Terminating the existing ffmpeg process...
frigate-new | 2020-03-13T13:33:38.289128555Z Waiting for ffmpeg to exit gracefully...
frigate-new | 2020-03-13T13:33:38.289136535Z Creating ffmpeg process...
frigate-new | 2020-03-13T13:33:38.289148278Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.1.185:8554/unicast -f rawvideo -pix_fmt rgb24 pipe:1.447910833Z front: ffmpeg_process exited unexpectedly with 0
frigate-new | 2020-03-13T13:34:21.447918522Z Terminating the existing ffmpeg process...
frigate-new | 2020-03-13T13:34:21.447926361Z Waiting for ffmpeg to exit gracefully...
frigate-new | 2020-03-13T13:34:21.447934599Z Creating ffmpeg process...
frigate-new | 2020-03-13T13:34:21.447943098Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.1.185:8554/unicast -f rawvideo -pix_fmt rgb24 pipe:8.099048001Z front: ffmpeg_process exited unexpectedly with 0
frigate-new | 2020-03-13T13:34:58.099077459Z Terminating the existing ffmpeg process...
frigate-new | 2020-03-13T13:34:58.099279305Z Waiting for ffmpeg to exit gracefully...
frigate-new | 2020-03-13T13:34:58.099338000Z Creating ffmpeg process...
frigate-new | 2020-03-13T13:34:58.099362711Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.1.185:8554/unicast -f rawvideo -pix_fmt rgb24 pipe:
frigate-new | 2020-03-13T13:35:26.812440396Z front: ffmpeg_process is still running but didnt return any bytes
frigate-new | 2020-03-13T13:35:26.812483777Z front: ffmpeg_process exited unexpectedly with 0
frigate-new | 2020-03-13T13:35:26.812495340Z Terminating the existing ffmpeg process...
frigate-new | 2020-03-13T13:35:26.812502500Z Waiting for ffmpeg to exit gracefully...
frigate-new | 2020-03-13T13:35:26.815324365Z Creating ffmpeg process...
frigate-new | 2020-03-13T13:35:26.815349264Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://192.168.1.185:8554/unicast -f rawvideo -pix_fmt rgb24 pipe:
frigate-new | 2020-03-13T13:36:05.379324205Z front: ffmpeg_process is still running but didnt return any bytes
I’ll hold of on restarting the container for a bit in case you any any info from within the container.
That particular issue I have found and solved already. Should be pushing a release sometime tomorrow.
2 Likes
Do you see any benefit from running the defang hack instead of the Wyze rtsp firmware?
I like that its open source and less risk of phoning home somewhere. It was available before the beta rtsp firmware so I never tried the beta.
I’m not sure new cameras can be hacked either.
Just pushed 0.5.1-rc1.
Changes:
- Switch to SimpleQueue instead of Queue (can no longer report queue sizes)
- Handle rare situations where plasma store fails
- Make detection queue more resilient
- Removed SharedArray as a dependency
Note that this does not fix issues related to Wyzecam RTSP feeds.
2 Likes
Which parameters should we be using with the Wyze cameras?
Here’s what I’m currently using
ffmpeg -hide_banner -loglevel info -rtsp_transport tcp -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://user:[email protected]/live -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
the new version died after 10 minutes
2020-03-13T21:53:28.182350286Z Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 2304x1296 [SAR 1:1 DAR 16:9], q=2-31, 1433272 kb/s, 20 fps, 20 tbn, 20 tbc
2020-03-13T21:53:28.182413760Z Metadata:
2020-03-13T21:53:28.182457840Z encoder : Lavc57.107.100 rawvideo
2020-03-13T21:53:28.182503482Z [rawvideo @ 0x563faf08a3c0] Encoder did not produce proper pts, making some up.
frame= 886 fps= 20 q=-0.0 size= 7750728kB time=00:00:41.66 bitrate=746496.0kbits/s speed=0.928x 496.5kbits/s speed=0.513x
frame= 1765 fps= 14 q=-0.0 size= 7624125kB time=00:01:23.66 bitrate=746496.0kbits/s speed=0.94x
frame= 2642 fps= 20 q=-0.0 size=23112216kB time=00:02:12.10 bitrate=14332720kbits/s speed=0.942x s speed=0.989x
frame= 2512 fps= 20 q=-0.0 size=30854196kB time=00:02:56.35 bitrate=1433272.3kbits/s speed=0.994x
frame= 3134 fps= 14 q=-0.0 size=19039050kB time=00:03:40.40 bitrate=1433272.3kbits/s speed=0.995x s speed=0.942x
frame= 5300 fps= 20 q=-0.0 size=46364400kB time=00:04:25.00 bitrate=1433272.3kbits/s speed=0.996x
frame= 6185 fps= 20 q=-0.0 size=26723925kB time=00:04:53.26 bitrate=746496.0kbits/s speed=0.946x
frame= 5025 fps= 14 q=-0.0 size=30526875kB time=00:05:35.00 bitrate=1433272.3kbits/s speed=0.996x
frame= 5874 fps= 20 q=-0.0 size=68881752kB time=00:06:33.70 bitrate=1433272.3kbits/s speed=0.992x
frame= 8775 fps= 20 q=-0.0 size=76763700kB time=00:07:18.40 bitrate=746496.0kbits/s speed=0.945x its/s speed=0.945x
frame= 9624 fps= 20 q=-0.0 size=84190752kB time=00:08:01.20 bitrate=1433272.3kbits/s speed=0.997x
frame=10494 fps= 20 q=-0.0 size=91801550kB time=00:08:15.86 bitrate=746496.0kbits/s speed=0.944x
frame=11469 fps= 20 q=-0.0 size=100330812kB time=00:09:33.45 bitrate=1433272.3kbits/s speed=0.998x
2020-03-13T22:03:23.175894940Z Process for driveway has been reading from ffmpeg for over 10 seconds long. Killing ffmpeg…
2020-03-13T22:03:33.185914228Z Unable to kill ffmpeg with pid 28
2020-03-13T22:03:33.193637223Z driveway: ffmpeg_process didnt return any bytes
2020-03-13T22:03:33.193815923Z driveway: ffmpeg_process exited unexpectedly with -9
2020-03-13T22:03:33.193869629Z Letting driveway rest for 10 seconds before restarting…
2020-03-13T22:03:43.203704769Z Terminating the existing ffmpeg process…
2020-03-13T22:03:43.204366067Z Waiting for ffmpeg to exit gracefully…
2020-03-13T22:03:43.205291595Z Creating ffmpeg process…
2020-03-13T22:03:43.205459595Z ffmpeg -hide_banner -loglevel info -rtsp_transport tcp -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://user:[email protected]/live -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
Guessed Channel Layout for Input Stream #0.1 : mono:10:17.00 bitrate=1433272.3kbits/s speed=0.999x
2020-03-13T22:03:45.064596869Z Input #0, rtsp, from ‘rtsp://user:[email protected]/live’:
You definitely don’t want mpdecimate in there anymore since 0.5.0. Note my comment that 0.5.1 doesn’t address the Wyze camera issues. Just wanted to get these fixes out before digging in on the Wyze issues.
Bit of a docker noob here. When I add a 2nd camera to the docker compose file I get a repetitive message “On connect called” and nothing more. Would it be possible for someone with a multi cam setup to provide a config.yml and docker-comose.yml?
Just updated, working great so far for me with my 3 cameras @blakeblackshear.
Noted something else earlier on this thread, but you may have missed it… My doorbell camera is not 16:9 and so the debugging mjpeg stream for it no longer displays it in the correct aspect ratio because of the changes you made after 0.5.0-rc7 to be able to specify height. Think you might be able to get the aspect ratio from ffprobe or not alter the aspect ratio at all in your code?
ffprobe reports the aspect ratio of my doorbell camera as:
'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:1'
Interesting…
You don’t want to add another container to your compose file. You add the second camera to the one config file. The message you are seeing is because you have multiple frigate containers.
did the debug telemetry change?
{"coral":{"detection_start":0.0,"fps":10.0,"inference_speed":8.94,"pid":22},"front":{"detection_fps":10.0,"ffmpeg_pid":28,"fps":10.1,"pid":23,"read_start":0.0,"skipped_fps":0.0},"plasma_stor`e_rc":null}
Not seeing queue depth
It’s due to the *Switch to SimpleQueue instead of Queue (can no longer report queue sizes)