It seems the processes are failing quicker when there is more motion. People are outside playing in our yard and when it fails, I’m restarting both the ffmpeg process and parent process
How frequently is it restarting? The logs will give me a better idea of where it is happening.
I have camera stream @ 2560x1920 (rtsp), but the stream that generate frigate on localhost:5000/cameraname is in smaller resolution. How can I define the output stream resolution?
You can define the output resolution and fps by adding ?h=<frame_height>&fps=<desired_fps> to the end of the url.
Perfect, thank you!
I noticed both ffmpeg processes (both cameras) are being restarted
2020-03-08T18:19:31.911257771Z ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg developers
2020-03-08T18:19:31.911479712Z built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
2020-03-08T18:19:31.912263911Z libavutil 55. 78.100 / 55. 78.100
2020-03-08T18:19:31.912459213Z libavcodec 57.107.100 / 57.107.100
2020-03-08T18:19:31.912658834Z libavfilter 6.107.100 / 6.107.100
2020-03-08T18:19:31.912717714Z libavresample 3. 7. 0 / 3. 7. 0
2020-03-08T18:19:31.912771901Z libswscale 4. 8.100 / 4. 8.100
2020-03-08T18:19:31.912860366Z libswresample 2. 9.100 / 2. 9.100
2020-03-08T18:19:31.946728219Z [rtsp @ 0x562215206500] SDP:
2020-03-08T18:19:31.947059368Z v=0
2020-03-08T18:19:31.952773445Z [rtsp @ 0x562215206500] setting jitter buffer size to 0
2020-03-08T18:19:32.081720296Z Last message repeated 1 times
2020-03-08T18:19:32.673704074Z [h264 @ 0x562215209f40] Reinit context to 1920x1088, pix_fmt: yuv420p
2020-03-08T18:19:33.865541490Z Guessed Channel Layout for Input Stream #0.1 : mono
2020-03-08T18:19:33.865768207Z Input #0, rtsp, from ‘rtsp://user:[email protected]/live’:
2020-03-08T18:19:33.866626113Z Stream #0:0: Video: h264 (Main), 1 reference frame, yuv420p(progressive, left), 1920x1080 (1920x1088), 15 fps, 15 tbr, 90k tbn, 30 tbc
2020-03-08T18:19:33.866878163Z Stream #0:1: Audio: pcm_alaw, 8000 Hz, mono, s16, 64 kb/s
2020-03-08T18:19:33.867867293Z Stream mapping:
2020-03-08T18:19:33.868061503Z Stream #0:0 → #0:0 (h264 (native) → rawvideo (native))
2020-03-08T18:19:33.870763792Z [h264 @ 0x56221522a2a0] Reinit context to 1920x1088, pix_fmt: yuv420p
2020-03-08T18:19:33.957608557Z [graph 0 input from stream 0:0 @ 0x562215352b80] w:1920 h:1080 pixfmt:yuv420p tb:1/90000 fr:15/1 sar:0/1 sws_param:flags=2
2020-03-08T18:19:33.958708063Z [auto_scaler_0 @ 0x562215681100] w:iw h:ih flags:‘bicubic’ interl:0
2020-03-08T18:19:33.959524900Z [format @ 0x56221567e680] auto-inserting filter ‘auto_scaler_0’ between the filter ‘Parsed_null_0’ and the filter ‘format’
2020-03-08T18:19:33.961482608Z [auto_scaler_0 @ 0x562215681100] w:1920 h:1080 fmt:yuv420p sar:0/1 → w:1920 h:1080 fmt:rgb24 sar:0/1 flags:0x4
2020-03-08T18:19:33.974596104Z Output #0, rawvideo, to ‘pipe:’:
2020-03-08T18:19:33.975272491Z Metadata:
2020-03-08T18:19:33.976064521Z title : Session streamed by “wyze”
2020-03-08T18:19:33.976865998Z comment : live
2020-03-08T18:19:33.977497484Z encoder : Lavf57.83.100
2020-03-08T18:19:33.978677013Z Stream #0:0: Video: rawvideo, 1 reference frame (RGB[24] / 0x18424752), rgb24(left), 1920x1080, q=2-31, 746496 kb/s, 15 fps, 15 tbn, 15 tbc
2020-03-08T18:19:33.979212912Z Metadata:
2020-03-08T18:19:33.979818976Z encoder : Lavc57.107.100 rawvideo
2020-03-08T18:19:33.980389212Z *** 6 dup!
*** dropping frame 7 from stream 0 at ts 1 fps=0.9 q=-0.0 size= 42525kB time=00:00:00.46 bitrate=746495.5kbits/s dup=6 drop=0 speed=0.06x
2020-03-08T18:19:41.681514704Z *** dropping frame 7 from stream 0 at ts 2
2020-03-08T18:19:41.696207307Z *** dropping frame 7 from stream 0 at ts 3
2020-03-08T18:19:41.707551819Z *** dropping frame 7 from stream 0 at ts 4
[NULL @ 0x5594febc1180] missing picture in access unit with size 26=1433272.3kbits/s dup=1263 drop=6 speed=0.476x p=6 drop=4 speed=0.217x
2020-03-08T18:20:03.097527829Z [h264 @ 0x5594fedf5860] no frame!
Error while decoding stream #0:0: Invalid data found when processing inputtime=00:01:21.05 bitrate=1433272.3kbits/s dup=1263 drop=6 speed=0.476x
2020-03-08T18:20:03.534113849Z No more output streams to write to, finishing.
2020-03-08T18:20:03.534686400Z frame= 1624 fps=9.5 q=-0.0 Lsize=14206752kB time=00:01:21.20 bitrate=1433272.3kbits/s dup=1263 drop=6 speed=0.476x
2020-03-08T18:20:03.535303504Z video:14206752kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
2020-03-08T18:20:03.535843829Z Input file #0 (rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0):
2020-03-08T18:20:03.536747113Z Input stream #0:0 (video): 368 packets read (11764758 bytes); 367 frames decoded;
2020-03-08T18:20:03.537306325Z Input stream #0:1 (audio): 1317 packets read (328143 bytes);
2020-03-08T18:20:03.537750612Z Total: 1685 packets (12092901 bytes) demuxed
2020-03-08T18:20:03.538167815Z Output file #0 (pipe:):
2020-03-08T18:20:03.538800835Z Output stream #0:0 (video): 1624 frames encoded; 1624 packets muxed (14547714048 bytes);
2020-03-08T18:20:03.539226443Z Total: 1624 packets (14547714048 bytes) muxed
2020-03-08T18:20:03.604116424Z frontdoor: ffmpeg_process exited unexpectedly with 0
2020-03-08T18:20:03.604315283Z Letting frontdoor rest for 6 seconds before restarting…
2020-03-08T18:20:09.610223038Z Terminating the existing ffmpeg process…
2020-03-08T18:20:09.610920323Z Waiting for ffmpeg to exit gracefully…
2020-03-08T18:20:09.613530411Z Creating ffmpeg process…
2020-03-08T18:20:09.613752253Z ffmpeg -loglevel verbose -rtsp_transport tcp -i rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0 -f rawvideo -pix_fmt rgb24 pipe:
2020-03-08T18:20:09.776604054Z ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg developers
2020-03-08T18:20:09.776815428Z built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
2020-03-08T18:20:13.522779771Z [rtsp @ 0x55e0f9fa1500] SDP:
2020-03-08T18:20:13.523442024Z v=0
2020-03-08T18:20:13.524671267Z a=rtpmap:97 MPEG4-GENERIC/16000
2020-03-08T18:20:13.524728089Z a=fmtp:97 streamtype=5;profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3;config=1408
2020-03-08T18:20:13.524815716Z a=recvonly
2020-03-08T18:20:13.524873978Z
2020-03-08T18:20:13.579952408Z [rtsp @ 0x55e0f9fa1500] setting jitter buffer size to 0
2020-03-08T18:20:14.383423841Z Last message repeated 1 times
2020-03-08T18:20:15.151696833Z [h264 @ 0x55e0f9fa5180] Reinit context to 2304x1296, pix_fmt: yuvj420p
2020-03-08T18:20:15.626040000Z Input #0, rtsp, from ‘rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0’:
2020-03-08T18:20:15.626267941Z Metadata:
2020-03-08T18:20:15.626710963Z Stream #0:0: Video: h264 (High), 1 reference frame, yuvj420p(pc, bt709, progressive, left), 2304x1296 [SAR 1:1 DAR 16:9], 20 fps, 20 tbr, 90k tbn, 40 tbc
2020-03-08T18:20:15.626914256Z Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp
2020-03-08T18:20:15.627581055Z Stream mapping:
2020-03-08T18:20:15.627773949Z Stream #0:0 → #0:0 (h264 (native) → rawvideo (native))
2020-03-08T18:20:15.630565801Z [h264 @ 0x55e0f9fd2e40] Reinit context to 2304x1296, pix_fmt: yuvj420p
2020-03-08T18:20:15.694935340Z [graph 0 input from stream 0:0 @ 0x55e0f9fd2020] w:2304 h:1296 pixfmt:yuvj420p tb:1/90000 fr:20/1 sar:1/1 sws_param:flags=2
2020-03-08T18:20:15.695357906Z [auto_scaler_0 @ 0x55e0fa5fbae0] w:iw h:ih flags:‘bicubic’ interl:0
2020-03-08T18:20:15.695558572Z [format @ 0x55e0f9fd3300] auto-inserting filter ‘auto_scaler_0’ between the filter ‘Parsed_null_0’ and the filter ‘format’
2020-03-08T18:20:15.696786896Z [auto_scaler_0 @ 0x55e0fa5fbae0] w:2304 h:1296 fmt:yuvj420p sar:1/1 → w:2304 h:1296 fmt:rgb24 sar:1/1 flags:0x4
2020-03-08T18:20:15.710391966Z Output #0, rawvideo, to ‘pipe:’:
2020-03-08T18:20:15.710614563Z Metadata:
2020-03-08T18:20:15.711071137Z Stream #0:0: Video: rawvideo, 1 reference frame (RGB[24] / 0x18424752), rgb24(left), 2304x1296 [SAR 1:1 DAR 16:9], q=2-31, 1433272 kb/s, 20 fps, 20 tbn, 20 tbc
2020-03-08T18:20:15.711300332Z Metadata:
*** dropping frame 9 from stream 0 at ts 3 fps=1.9 q=-0.0 size= 78732kB time=00:00:00.45 bitrate=1433272.3kbits/s dup=8 drop=0 speed=0.0955x
2020-03-08T18:20:20.393729258Z *** dropping frame 9 from stream 0 at ts 4
2020-03-08T18:20:20.407087303Z *** dropping frame 9 from stream 0 at ts 5
2020-03-08T18:20:20.416509105Z *** dropping frame 9 from stream 0 at ts 6
2020-03-08T18:20:26.515334070Z Process for driveway has been reading from ffmpeg for over 10 seconds long. Killing ffmpeg…
*** 6 dup!0 fps= 10 q=-0.0 size= 1749600kB time=00:00:10.00 bitrate=1433272.3kbits/s dup=8 drop=4 speed=0.505x s dup=8 drop=4 speed=0.143x
*** dropping frame 210 from stream 0 at ts 208= 10 q=-0.0 size= 1828332kB time=00:00:10.45 bitrate=1433272.3kbits/s dup=14 drop=4 speed=0.511x
2020-03-08T18:20:36.462307235Z *** dropping frame 211 from stream 0 at ts 208
2020-03-08T18:20:36.462479147Z Last message repeated 1 times
2020-03-08T18:20:36.462548311Z *** dropping frame 211 from stream 0 at ts 209
2020-03-08T18:20:36.462628067Z Last message repeated 1 times
2020-03-08T18:20:36.462687992Z *** dropping frame 212 from stream 0 at ts 210
2020-03-08T18:20:36.559986122Z Unable to kill ffmpeg with pid 7870
2020-03-08T18:20:36.571236242Z driveway: ffmpeg_process didnt return any bytes
2020-03-08T18:20:36.571504780Z driveway: ffmpeg_process didnt return any bytes
2020-03-08T18:20:36.571609125Z driveway: ffmpeg_process didnt return any bytes
2020-03-08T18:20:36.571682733Z driveway: ffmpeg_process didnt return any bytes
2020-03-08T18:20:36.572013881Z driveway: ffmpeg_process exited unexpectedly with -9
2020-03-08T18:20:36.572229056Z Letting driveway rest for 5 seconds before restarting…
*** 22 dup! fps= 11 q=-0.0 size= 2256984kB time=00:00:12.90 bitrate=1433272.3kbits/s dup=14 drop=10 speed=0.533x dup=14 drop=10 speed=0.513x
2020-03-08T18:20:41.537273110Z *** 39 dup!
2020-03-08T18:20:41.576828349Z Terminating the existing ffmpeg process…
2020-03-08T18:20:41.577033779Z Waiting for ffmpeg to exit gracefully…
2020-03-08T18:20:41.578706631Z Creating ffmpeg process…
2020-03-08T18:20:41.578903918Z ffmpeg -loglevel verbose -rtsp_transport tcp -i rtsp://g6ev8v43:[email protected]/live -f rawvideo -pix_fmt rgb24 pipe:
2020-03-08T18:20:41.755326902Z ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg developers
2020-03-08T18:20:41.755539427Z built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
2020-03-08T18:20:41.756231544Z libavutil 55. 78.100 / 55. 78.100
2020-03-08T18:20:41.756617613Z libavcodec 57.107.100 / 57.107.100
2020-03-08T18:20:41.756751470Z libavformat 57. 83.100 / 57. 83.100
2020-03-08T18:20:41.756811315Z libavdevice 57. 10.100 / 57. 10.100
2020-03-08T18:20:41.756867621Z libavfilter 6.107.100 / 6.107.100
2020-03-08T18:20:41.756922323Z libavresample 3. 7. 0 / 3. 7. 0
2020-03-08T18:20:41.757046092Z libswresample 2. 9.100 / 2. 9.100
2020-03-08T18:20:41.757102547Z libpostproc 54. 7.100 / 54. 7.100
2020-03-08T18:20:41.771360446Z [rtsp @ 0x55573ad2b500] SDP:
2020-03-08T18:20:41.771555952Z v=0
2020-03-08T18:20:41.776048962Z [rtsp @ 0x55573ad2b500] setting jitter buffer size to 0
2020-03-08T18:20:41.916984530Z Last message repeated 1 times
2020-03-08T18:20:41.917195771Z [h264 @ 0x55573ad2ef40] Reinit context to 1920x1088, pix_fmt: yuv420p
2020-03-08T18:20:43.672065308Z Guessed Channel Layout for Input Stream #0.1 : mono
2020-0
The issues seem to be when it is detecting movement. There is no longer anyone playing in the front yard and it’s been stable.
However, when it freezes, the ffmpeg process restarts… doesn’t detect anything and goes into the restart loop. The fix is to restart both processes
The movement is the section in the screenshot in the red box
ffmpeg is reporting lots of errors with your rtsp stream. It looks like you have removed most of the default input parameters. I would try running ffmpeg yourself inside the container and see if it fails and exits on its own. I am guessing it will. I am just reading the output of ffmpeg, so if it is exiting or failing, frigate will too. You should be able to run it with:
ffmpeg -loglevel verbose -rtsp_transport tcp -i "rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0" -f rawvideo -pix_fmt rgb24 pipe: > /dev/null
while changing the logging level, I must have introduced a rogue character that was messing up the yaml.
I replaced a few lines and am back to what seems to be the default
This is what I now have
ffmpeg -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://username:[email protected]/live -f rawvideo -pix_fmt rgb24 pipe:
Thanks for your constant attention to improving this!
I just noticed one of my cameras (front_window/192.168.10.2) stopped responding so I printed out the stacktrace for the PID from it using blakeblackshear/frigate:dev@55199bd1f8538d2e9ed6e94d4dcd8efd1ffb51c450f53bec7f23bbfb647a931e. (not the latest image as of this afternoon, but I’m updating it after this post)
2020-03-07T23:17:14.236366404Z On connect called
2020-03-07T23:17:14.392107744Z /arrow/cpp/src/plasma/store.cc:1226: Allowing the Plasma store to use up to 0.4GB of memory.
2020-03-07T23:17:14.392141529Z /arrow/cpp/src/plasma/store.cc:1253: Starting object store with directory /dev/shm and huge page support disabled
2020-03-07T23:17:15.249934695Z Starting detection process: 29
2020-03-07T23:17:15.251402781Z Camera_process started for front_window: 30
2020-03-07T23:17:15.252405762Z Starting process for front_window: 30
2020-03-07T23:17:15.252652535Z ffprobe -v panic -show_error -show_streams -of json "rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword"
2020-03-07T23:17:15.252934714Z Camera_process started for kitchen: 31
2020-03-07T23:17:15.254040712Z Starting process for kitchen: 31
2020-03-07T23:17:15.254332654Z ffprobe -v panic -show_error -show_streams -of json "rtmp://192.168.10.1:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword"
2020-03-07T23:17:15.262955883Z * Serving Flask app "detect_objects" (lazy loading)
2020-03-07T23:17:15.262978652Z * Environment: production
2020-03-07T23:17:15.262984140Z WARNING: This is a development server. Do not use it in a production deployment.
2020-03-07T23:17:15.262988709Z Use a production WSGI server instead.
2020-03-07T23:17:15.262992507Z * Debug mode: off
2020-03-07T23:17:20.107441208Z {'streams': [{'index': 0, 'codec_name': 'aac', 'codec_long_name': 'AAC (Advanced Audio Coding)', 'profile': 'LC', 'codec_type': 'audio', 'codec_time_base': '1/16000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 'fltp', 'sample_rate': '16000', 'channels': 1, 'channel_layout': 'mono', 'bits_per_sample': 0, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/1000', 'start_pts': 58650299, 'start_time': '58650.299000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'High', 'codec_type': 'video', 'codec_time_base': '1/20', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 2560, 'height': 1920, 'coded_width': 2560, 'coded_height': 1920, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:1', 'pix_fmt': 'yuv420p', 'level': 51, 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'true', 'nal_length_size': '4', 'r_frame_rate': '10/1', 'avg_frame_rate': '10/1', 'time_base': '1/1000', 'start_pts': 58650251, 'start_time': '58650.251000', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
2020-03-07T23:17:20.121022523Z Creating ffmpeg process...
2020-03-07T23:17:20.121077454Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-07T23:17:23.267058117Z {'streams': [{'index': 0, 'codec_name': 'aac', 'codec_long_name': 'AAC (Advanced Audio Coding)', 'profile': 'LC', 'codec_type': 'audio', 'codec_time_base': '1/16000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 'fltp', 'sample_rate': '16000', 'channels': 1, 'channel_layout': 'mono', 'bits_per_sample': 0, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/1000', 'start_pts': 577060152, 'start_time': '577060.152000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'High', 'codec_type': 'video', 'codec_time_base': '1/24', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 2560, 'height': 1440, 'coded_width': 2560, 'coded_height': 1440, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:1', 'pix_fmt': 'yuv420p', 'level': 51, 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'true', 'nal_length_size': '4', 'r_frame_rate': '30/1', 'avg_frame_rate': '12/1', 'time_base': '1/1000', 'start_pts': 577060113, 'start_time': '577060.113000', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
2020-03-07T23:17:23.280863350Z Creating ffmpeg process...
2020-03-07T23:17:23.280922257Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.1:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-08T23:19:35.758101852Z Detection appears to be stuck. Restarting detection process
2020-03-08T23:19:35.758235679Z Waiting for detection process to exit gracefully...
2020-03-08T23:19:35.767778331Z /arrow/cpp/src/plasma/store.cc:738: Disconnecting client on fd 6
2020-03-08T23:19:35.773427845Z Starting detection process: 1754
2020-03-08T23:22:26.702416709Z kitchen: ffmpeg_process didnt return any bytes
2020-03-08T23:22:26.702437548Z kitchen: ffmpeg_process exited unexpectedly with 0
2020-03-08T23:22:26.702440380Z Letting kitchen rest for 0 seconds before restarting...
2020-03-08T23:22:26.702442785Z Terminating the existing ffmpeg process...
2020-03-08T23:22:26.702445481Z Waiting for ffmpeg to exit gracefully...
2020-03-08T23:22:26.702990395Z Creating ffmpeg process...
2020-03-08T23:22:26.702995555Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.1:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-08T23:30:10.069975134Z kitchen: ffmpeg_process didnt return any bytes
2020-03-08T23:30:10.070005973Z kitchen: ffmpeg_process exited unexpectedly with 0
2020-03-08T23:30:10.070009333Z Letting kitchen rest for 0 seconds before restarting...
2020-03-08T23:30:10.070011669Z Terminating the existing ffmpeg process...
2020-03-08T23:30:10.070027690Z Waiting for ffmpeg to exit gracefully...
2020-03-08T23:30:10.070561339Z Creating ffmpeg process...
2020-03-08T23:30:10.070572930Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.1:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-08T23:31:59.483940936Z front_window: ffmpeg_process didnt return any bytes
2020-03-08T23:31:59.483961420Z front_window: ffmpeg_process exited unexpectedly with 0
2020-03-08T23:31:59.483964904Z Letting front_window rest for 0 seconds before restarting...
2020-03-08T23:31:59.483970880Z Terminating the existing ffmpeg process...
2020-03-08T23:31:59.483973579Z Waiting for ffmpeg to exit gracefully...
2020-03-08T23:31:59.484698124Z Creating ffmpeg process...
2020-03-08T23:31:59.484712068Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-08T23:36:40.677701376Z kitchen: ffmpeg_process didnt return any bytes
2020-03-08T23:36:40.677720263Z kitchen: ffmpeg_process exited unexpectedly with 0
2020-03-08T23:36:40.677723275Z Letting kitchen rest for 0 seconds before restarting...
2020-03-08T23:36:40.677725537Z Terminating the existing ffmpeg process...
2020-03-08T23:36:40.677727774Z Waiting for ffmpeg to exit gracefully...
2020-03-08T23:36:40.678253155Z Creating ffmpeg process...
2020-03-08T23:36:40.678264084Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.1:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-08T23:41:30.547997807Z front_window: ffmpeg_process exited unexpectedly with 0
2020-03-08T23:41:30.548063807Z Letting front_window rest for 0 seconds before restarting...
2020-03-08T23:41:30.548077278Z Terminating the existing ffmpeg process...
2020-03-08T23:41:30.548087835Z Waiting for ffmpeg to exit gracefully...
2020-03-08T23:41:30.550769559Z Creating ffmpeg process...
2020-03-08T23:41:30.550873469Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -use_wallclock_as_timestamps 1 -i rtmp://192.168.10.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=admin&password=fakepassword -f rawvideo -pix_fmt rgb24 pipe:
2020-03-08T23:49:21.338644644Z File "detect_objects.py", line 278, in <module>
2020-03-08T23:49:21.338723095Z main()
2020-03-08T23:49:21.338733079Z File "detect_objects.py", line 165, in main
2020-03-08T23:49:21.338740059Z camera_process['process'].start()
2020-03-08T23:49:21.338747572Z File "/usr/lib/python3.7/multiprocessing/process.py", line 112, in start
2020-03-08T23:49:21.338755742Z self._popen = self._Popen(self)
2020-03-08T23:49:21.338762750Z File "/usr/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
2020-03-08T23:49:21.338770765Z return _default_context.get_context().Process._Popen(process_obj)
2020-03-08T23:49:21.338778264Z File "/usr/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
2020-03-08T23:49:21.338785125Z return Popen(process_obj)
2020-03-08T23:49:21.338792784Z File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
2020-03-08T23:49:21.338800368Z self._launch(process_obj)
2020-03-08T23:49:21.338923787Z File "/usr/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
2020-03-08T23:49:21.338935880Z code = process_obj._bootstrap()
2020-03-08T23:49:21.338943375Z File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
2020-03-08T23:49:21.338950829Z self.run()
2020-03-08T23:49:21.338958455Z File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
2020-03-08T23:49:21.338965329Z self._target(*self._args, **self._kwargs)
2020-03-08T23:49:21.338972388Z File "/opt/frigate/frigate/video.py", line 293, in track_camera
2020-03-08T23:49:21.338979546Z region_detections = object_detector.detect(tensor_input)
2020-03-08T23:49:21.338986493Z File "/opt/frigate/frigate/edgetpu.py", line 127, in detect
2020-03-08T23:49:21.338993645Z raw_detections = self.plasma_client.get(object_id_detections, timeout_ms=10000)
I noticed that you put up a new dev image from today so I’ll pull it to see if that helps with anything.
That stacktrace seems to indicate that it recovered. It looks like the ffmpeg process restarted at 23:41:30 and you dumped the stacktrace at 23:49:21. Was the timestamp in the debug feed prior to 23:41:30? Also, next time, grab the output of /debug/stack and the dump the stacktrace from the coral pid too.
The hardest part is that I can’t reproduce this with my cameras. They are all PoE Dahua cameras. If someone wants to buy me a Wyze camera, I am happy to flash it with the RTSP firmware and use it for testing going forward.
1 Like
Mine crashed out again some time this morning after running for 5 days on 0.5.0.
Edit: Oh and the plasma store gave a return code of -7 (first time that happened).
Logs:
2020-03-09T05:27:46.916331635Z Detection appears to be stuck. Restarting detection process
2020-03-09T05:27:46.988035821Z Waiting for detection process to exit gracefully...
2020-03-09T05:27:46.933879367Z /arrow/cpp/src/plasma/store.cc:738: Disconnecting client on fd 6
2020-03-09T05:27:47.102647696Z Starting detection process: 13190
2020-03-09T05:45:03.343198978Z Process Process-2:
2020-03-09T05:45:03.834962553Z Process Process-3:
2020-03-09T05:45:05.215854968Z Traceback (most recent call last):
2020-03-09T05:45:05.215976851Z Traceback (most recent call last):
2020-03-09T05:45:05.216366962Z File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
2020-03-09T05:45:05.216426083Z self.run()
2020-03-09T05:45:05.216448177Z File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
2020-03-09T05:45:05.216468048Z self.run()
2020-03-09T05:45:05.216501989Z File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
2020-03-09T05:45:05.216547104Z self._target(*self._args, **self._kwargs)
2020-03-09T05:45:05.216620544Z File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
2020-03-09T05:45:05.216637930Z self._target(*self._args, **self._kwargs)
2020-03-09T05:45:05.216648838Z File "/opt/frigate/frigate/video.py", line 373, in track_camera
2020-03-09T05:45:05.216659810Z plasma_client.put(frame, plasma.ObjectID(object_id))
2020-03-09T05:45:05.216670146Z File "/opt/frigate/frigate/video.py", line 373, in track_camera
2020-03-09T05:45:05.216680637Z plasma_client.put(frame, plasma.ObjectID(object_id))
2020-03-09T05:45:05.216691134Z File "pyarrow/_plasma.pyx", line 536, in pyarrow._plasma.PlasmaClient.put
2020-03-09T05:45:05.216701912Z File "pyarrow/_plasma.pyx", line 536, in pyarrow._plasma.PlasmaClient.put
2020-03-09T05:45:05.216712459Z File "pyarrow/_plasma.pyx", line 364, in pyarrow._plasma.PlasmaClient.create
2020-03-09T05:45:05.216722963Z File "pyarrow/_plasma.pyx", line 364, in pyarrow._plasma.PlasmaClient.create
2020-03-09T05:45:05.216736814Z File "pyarrow/_plasma.pyx", line 291, in pyarrow._plasma.plasma_check_status
2020-03-09T05:45:05.216753749Z File "pyarrow/_plasma.pyx", line 291, in pyarrow._plasma.plasma_check_status
2020-03-09T05:45:05.216771272Z File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
2020-03-09T05:45:05.216789457Z File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
2020-03-09T05:45:05.234100381Z OSError: Connection reset by peer
2020-03-09T05:45:05.240699997Z OSError: Encountered unexpected EOF
2020-03-09T05:45:07.863896598Z Process for front is not alive. Starting again...
2020-03-09T05:45:07.870432369Z Camera_process started for front: 13232
2020-03-09T05:45:07.874356220Z Starting process for front: 13232
2020-03-09T05:45:07.887191392Z Process for cars is not alive. Starting again...
2020-03-09T05:45:07.892589561Z Camera_process started for cars: 13233
2020-03-09T05:45:07.892751897Z Process for cat_cam is not alive. Starting again...
2020-03-09T05:45:07.896138519Z Starting process for cars: 13233
2020-03-09T05:45:07.897713847Z Camera_process started for cat_cam: 13234
2020-03-09T05:45:07.901778431Z Starting process for cat_cam: 13234
2020-03-09T05:45:07.944359542Z ffprobe -v panic -show_error -show_streams -of json "rtsp://<redacted>:554/Streaming/Channels/2/preview"
2020-03-09T05:45:07.944419427Z ffprobe -v panic -show_error -show_streams -of json "rtsp://<redacted>:554/onvif1"ffprobe -v panic -show_error -show_streams -of json "rtsp://<redacted>:554/Streaming/Channels/2/preview"
2020-03-09T05:45:07.944438060Z
2020-03-09T05:45:08.146233076Z Exception in thread Thread-3:
2020-03-09T05:45:08.146337006Z Traceback (most recent call last):
2020-03-09T05:45:08.146387019Z File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
2020-03-09T05:45:08.146427364Z self.run()
2020-03-09T05:45:08.146513699Z File "/opt/frigate/frigate/object_processing.py", line 65, in run
2020-03-09T05:45:08.146560933Z current_frame = self.plasma_client.get(object_id, timeout_ms=0)
2020-03-09T05:45:08.146598488Z File "pyarrow/_plasma.pyx", line 580, in pyarrow._plasma.PlasmaClient.get
2020-03-09T05:45:08.146633561Z File "pyarrow/_plasma.pyx", line 568, in pyarrow._plasma.PlasmaClient.get
2020-03-09T05:45:08.146673852Z File "pyarrow/_plasma.pyx", line 428, in pyarrow._plasma.PlasmaClient.get_buffers
2020-03-09T05:45:08.146710145Z File "pyarrow/_plasma.pyx", line 322, in pyarrow._plasma.PlasmaClient._get_object_buffers
2020-03-09T05:45:08.146743971Z File "pyarrow/_plasma.pyx", line 291, in pyarrow._plasma.plasma_check_status
2020-03-09T05:45:08.146776251Z File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
2020-03-09T05:45:08.146792505Z OSError: Broken pipe
2020-03-09T05:45:08.146805601Z
2020-03-09T05:45:10.879859976Z {'streams': [{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'Baseline', 'codec_type': 'video', 'codec_time_base': '0/2', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 1280, 'height': 720, 'coded_width': 1280, 'coded_height': 720, 'has_b_frames': 0, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:1', 'pix_fmt': 'yuv420p', 'level': 10, 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '15/1', 'avg_frame_rate': '0/0', 'time_base': '1/90000', 'start_pts': 5940, 'start_time': '0.066000', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'pcm_alaw', 'codec_long_name': 'PCM A-law / G.711 A-law', 'codec_type': 'audio', 'codec_time_base': '1/8000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 's16', 'sample_rate': '8000', 'channels': 1, 'bits_per_sample': 8, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/8000', 'start_pts': 0, 'start_time': '0.000000', 'bit_rate': '64000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
2020-03-09T05:45:11.009262525Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 20 more times
2020-03-09T05:45:11.409684722Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 19 more times
2020-03-09T05:45:11.809991193Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 18 more times
2020-03-09T05:45:12.210286948Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 17 more times
2020-03-09T05:45:12.610600703Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 16 more times
2020-03-09T05:45:13.010888863Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 15 more times
2020-03-09T05:45:13.411179148Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 14 more times
2020-03-09T05:45:13.811487474Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 13 more times
2020-03-09T05:45:14.211754426Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 12 more times
2020-03-09T05:45:14.612121681Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 11 more times
2020-03-09T05:45:15.012394953Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 10 more times
2020-03-09T05:45:15.342001212Z {'streams': [{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'Main', 'codec_type': 'video', 'codec_time_base': '1/8', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 640, 'height': 480, 'coded_width': 640, 'coded_height': 480, 'has_b_frames': 0, 'sample_aspect_ratio': '4:3', 'display_aspect_ratio': '16:9', 'pix_fmt': 'yuvj420p', 'level': 22, 'color_range': 'pc', 'color_space': 'bt709', 'color_transfer': 'bt709', 'color_primaries': 'bt709', 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '25/1', 'avg_frame_rate': '4/1', 'time_base': '1/90000', 'start_pts': 132300, 'start_time': '1.470000', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
2020-03-09T05:45:15.346692181Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 20 more times
2020-03-09T05:45:15.412638482Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 9 more times
2020-03-09T05:45:15.614484680Z {'streams': [{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'Main', 'codec_type': 'video', 'codec_time_base': '1/8', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 640, 'height': 480, 'coded_width': 640, 'coded_height': 480, 'has_b_frames': 0, 'sample_aspect_ratio': '4:3', 'display_aspect_ratio': '16:9', 'pix_fmt': 'yuvj420p', 'level': 22, 'color_range': 'pc', 'color_space': 'bt709', 'color_transfer': 'bt709', 'color_primaries': 'bt709', 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '25/1', 'avg_frame_rate': '4/1', 'time_base': '1/90000', 'start_pts': 135900, 'start_time': '1.510000', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
2020-03-09T05:45:15.617566140Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 20 more times
2020-03-09T05:45:15.747150312Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 19 more times
2020-03-09T05:45:15.813018533Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 8 more times
2020-03-09T05:45:16.018063594Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 19 more times
2020-03-09T05:45:16.147523791Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 18 more times
2020-03-09T05:45:16.213302970Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 7 more times
2020-03-09T05:45:16.418335907Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 18 more times
2020-03-09T05:45:16.547864359Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 17 more times
2020-03-09T05:45:16.613613868Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 6 more times
2020-03-09T05:45:16.818684480Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 17 more times
2020-03-09T05:45:16.948141805Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 16 more times
2020-03-09T05:45:17.013860693Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 5 more times
2020-03-09T05:45:17.219005691Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 16 more times
2020-03-09T05:45:17.348513346Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 15 more times
2020-03-09T05:45:17.414204963Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 4 more times
2020-03-09T05:45:17.619336010Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 15 more times
2020-03-09T05:45:17.748805482Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 14 more times
2020-03-09T05:45:17.814523719Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 3 more times
2020-03-09T05:45:18.019602460Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 14 more times
2020-03-09T05:45:18.149147861Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 13 more times
2020-03-09T05:45:18.214778023Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 2 more times
2020-03-09T05:45:18.419994693Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 13 more times
2020-03-09T05:45:18.549413290Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 12 more times
2020-03-09T05:45:18.615051065Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 1 more times
2020-03-09T05:45:18.820224457Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 12 more times
2020-03-09T05:45:18.949694657Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 11 more times
2020-03-09T05:45:19.037541832Z Process Process-50:
2020-03-09T05:45:19.071178928Z Traceback (most recent call last):
2020-03-09T05:45:19.071244634Z File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
2020-03-09T05:45:19.071271571Z self.run()
2020-03-09T05:45:19.071289943Z File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
2020-03-09T05:45:19.071309557Z self._target(*self._args, **self._kwargs)
2020-03-09T05:45:19.071327291Z File "/opt/frigate/frigate/video.py", line 175, in track_camera
2020-03-09T05:45:19.071345051Z object_detector = RemoteObjectDetector(name, '/labelmap.txt', detection_queue)
2020-03-09T05:45:19.071363326Z File "/opt/frigate/frigate/edgetpu.py", line 115, in __init__
2020-03-09T05:45:19.071381551Z self.plasma_client = plasma.connect("/tmp/plasma")
2020-03-09T05:45:19.071406517Z File "pyarrow/_plasma.pyx", line 850, in pyarrow._plasma.connect
2020-03-09T05:45:19.071418571Z File "pyarrow/_plasma.pyx", line 291, in pyarrow._plasma.plasma_check_status
2020-03-09T05:45:19.071429727Z File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
2020-03-09T05:45:19.071440524Z OSError: Could not connect to socket /tmp/plasma
2020-03-09T05:45:19.220554511Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 11 more times
2020-03-09T05:45:19.350033755Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 10 more times
2020-03-09T05:45:19.620888495Z /arrow/cpp/src/plasma/io.cc:168: Connection to IPC socket failed for pathname /tmp/plasma, retrying 10 more times
I downloaded the latest dev image now and also removed the mpdecimate command from the ffmpeg output args. Will see how it goes!
Could you add an option so we can choose to close both processes?
Ideally, it shouldn’t happen but it would be nice to have the option since losing the context would be preferred to having the process freeze up.
Interesting. I can monitor the plasma store and restart if it exits too. I haven’t seen that either. Seems like the cause of your failure.
I could, but it really shouldn’t be necessary and it will likely leave a bunch of abandoned ffmpeg processes that cause other issues. Your logs seem to indicate that the process is not actually stuck because it is restarting ffmpeg. I still want to understand why it doesn’t recover.
Hey ender7, would you be able to tell me how you got past that error? I have the same one on the hass.io build Thanks.
Wow always something new.
Thanks Blake. Currently running the dev image now for a couple of hours with good success, will let you know if it breaks.
Leaving out mpdecimate from the ffmpeg arguments does seem improve motion capturing, especially on one camera where I’m looking for cats (smaller objects and lower quality camera).
I throw in $10 towards a wyze if anyone else wants to contribute.
I was about to post the same thing 
1 Like