Does the detection_start value under coral change after each restart?
Yep, it resets after the process is restarted after getting stuck. I converted the detection_start time from epoch and it represents when the motion happens.
Strange. It seems like there is something specific about the data being sent to tensorflow for your unifi camera that causes it to get stuck. Looks like there might be a new version of tensorflow lite available. A bit of a long shot, but can you try the rc7 image I just pushed up?
1 Like
Yes its working now! Very slow and choppy…the process still gets stuck but it is working now. Thank you!
Currently up for around 17 hours on RC5 with 2 cams, so apparently it hasn’t been necessary yet for the restart-function when detection hangs 
Will update to rc7 now.
hello
Still stopping with the Rc5
the log show :
2020-03-01T19:42:36.626845636Z Creating ffmpeg process...
2020-03-01T19:42:36.626912092Z ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_dela:
2020-03-01T19:42:46.638561503Z Avant: ffmpeg_process exited unexpectedly with 1
2020-03-01T19:42:46.638612584Z Terminating the existing ffmpeg process...
2020-03-01T19:42:46.638661177Z Waiting for ffmpeg to exit gracefully...
2020-03-01T19:42:46.638712355Z Creating ffmpeg process...
2020-03-01T19:42:46.638764157Z ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_dela:
2020-03-01T20:34:38.717698333Z Detection appears to be stuck. Restarting detection process
2020-03-01T20:35:38.765144093Z Detection appears to be stuck. Restarting detection process
2020-03-01T20:36:38.788718990Z Detection appears to be stuck. Restarting detection process
2020-03-01T20:37:38.837164403Z Detection appears to be stuck. Restarting detection process
all the night with a the end ::
2020-03-02T10:31:14.745257447Z Detection appears to be stuck. Restarting detection process
2020-03-02T10:32:14.803618756Z Detection appears to be stuck. Restarting detection process
2020-03-02T10:32:51.549046605Z /arrow/cpp/src/plasma/store.cc:738: Disconnecting client on fd 7
Updated to Rc7 now ,testing
thank you .
8 hours on RC7 so far without any issues. I would have had multiple freezes with RC4 or earlier, so a definite thumbs-up from me so far.
You should be able to open a shell and use standard docker commands to watch the log… Also the log usually works for me on synology until the container starts getting unstable, then it starts to crap out, so that might be what you are seeing as well.
How are you watching the logs?
I can get into the container … “docker exec -it frigate bash” but I can’t find any logs.
Try
docker logs -f --timestamps frigate
CTRL C to stop following, omit the -f to not follow at all
1 Like
@blakeblackshear. Now that I have better access to my logs… thanks @danfulton
Here’s are some logs from my latest issue
2020-03-02T19:57:48.695845376Z encoder : Lavc57.107.100 rawvideo
2020-03-02T19:57:48.699721735Z [rawvideo @ 0x55895c545e20] Encoder did not produce proper pts, making some up.
2020-03-02T19:57:48.923766295Z [graph 0 input from stream 0:0 @ 0x5647cdc6a6c0] w:2304 h:1296 pixfmt:yuvj420p tb:1/90000 fr:241/12 sar:1/1 sws_param:flags=2
2020-03-02T19:57:48.923954332Z [auto_scaler_0 @ 0x5647cdc12580] w:iw h:ih flags:‘bicubic’ interl:0
2020-03-02T19:57:48.924010201Z [format @ 0x5647cdc0fae0] auto-inserting filter ‘auto_scaler_0’ between the filter ‘Parsed_null_0’ and the filter ‘format’
2020-03-02T19:57:48.924067293Z [swscaler @ 0x5647cdd72b00] deprecated pixel format used, make sure you did set range correctly
2020-03-02T19:57:48.924846278Z [auto_scaler_0 @ 0x5647cdc12580] w:2304 h:1296 fmt:yuvj420p sar:1/1 → w:2304 h:1296 fmt:rgb24 sar:1/1 flags:0x4
2020-03-02T19:57:48.937693854Z Output #0, rawvideo, to ‘pipe:’:
2020-03-02T19:57:48.937859573Z Metadata:
2020-03-02T19:57:48.937910733Z title : Media Server
2020-03-02T19:57:48.937957945Z encoder : Lavf57.83.100
2020-03-02T19:57:48.938002904Z Stream #0:0: Video: rawvideo, 1 reference frame (RGB[24] / 0x18424752), rgb24(left), 2304x1296 [SAR 1:1 DAR 16:9], q=2-31, 1439244 kb/s, 20.08 fps, 20.08 tbn, 20.08 tbc
2020-03-02T19:57:48.938071835Z Metadata:
2020-03-02T19:57:48.938116437Z encoder : Lavc57.107.100 rawvideo
2020-03-02T19:57:48.944371507Z [rawvideo @ 0x5647cdbf8c80] Encoder did not produce proper pts, making some up.
frame= 648 fps= 14 q=-0.0 size= 3936600kB time=00:00:43.66 bitrate=1439244.3kbits/s speed=0.976x 39230.7kbits/s speed=0.0472x
frame= 1281 fps= 14 q=-0.0 size= 7782075kB time=00:01:24.92 bitrate=750643.2kbits/s speed=0.9545x
frame= 1924 fps= 14 q=-0.0 size=23243436kB time=00:02:12.29 bitrate=1439244.3kbits/s speed=0.989x
frame= 2544 fps= 14 q=-0.0 size=30985416kB time=00:02:56.36 bitrate=1439244.3kbits/s speed=0.991x
frame= 4427 fps= 20 q=-0.0 size=38727396kB time=00:03:40.43 bitrate=750643.2kbits/s speed=0.942x 7x
frame= 5315 fps= 20 q=-0.0 size=46908825kB time=00:04:10.01 bitrate=750643.2kbits/s speed=0.94x
frame= 4415 fps= 14 q=-0.0 size=26851324kB time=00:05:09.36 bitrate=1439244.3kbits/s speed=0.993x
frame= 7111 fps= 20 q=-0.0 size=62207028kB time=00:05:54.07 bitrate=1439244.3kbits/sspeed=0.94x
frame= 8009 fps= 20 q=-0.0 size=70499925kB time=00:06:16.50 bitrate=750643.2kbits/s speed=0.939x
frame= 6309 fps= 14 q=-0.0 size=38327175kB time=00:06:58.27 bitrate=750643.2kbits/ss speed=0.994x
frame= 6935 fps= 14 q=-0.0 size=85555440kB time=00:08:06.97 bitrate=1439244.3kbits/s speed=0.994x
frame= 7562 fps= 14 q=-0.0 size=45939150kB time=00:08:21.33 bitrate=1439244.3kbits/s speed=0.994x 0.939x
frame=11561 fps= 20 q=-0.0 size=49845375kB time=00:09:03.97 bitrate=750643.2kbits/s speed=0.94x
frame=12445 fps= 20 q=-0.0 size=108868860kB time=00:10:19.66 bitrate=1439244.3kbits/s speed=0.994x d=0.994x
frame=13319 fps= 20 q=-0.0 size=116514612kB time=00:11:4.86 bitrate=750643.2kbits/s speed=0.938x
frame=10035 fps= 20 q=-0.0 size=123985404kB time=00:11:45.70 bitrate=1439244.3kbits/s speed=0.994x
frame=15041 fps= 20 q=-0.0 size=131578668kB time=00:12:28.92 bitrate=14343.2kbits/s speed=0.937x speed=0.937x
frame=15905 fps= 20 q=-0.0 size=68343750kB time=00:12:25.85 bitrate=750643.2kbits/s speed=0.937x
frame=11869 fps= 14 q=-0.0 size=72104175kB time=00:13:06.89 bitrate=750643.2kbits/s spepeed=0.995x
frame=17634 fps= 20 q=-0.0 size=154262232kB ime=00:13:47.53 bitrate=750643.2kbits/s speed=0.938x
frame=13095 fps= 14 q=-0.0 size=79552125kB time=00:14:28.17 bitrate=750643.2kbits/s speed=0.938x
frame=19358 fps= 20 q=-0.0 size=169343784kB time=00:16:03.88 itrate=750643.2kbits/s speed=0.938x 9244.3kbits/s speed=0.995x
frame=20217 fps= 20 q=-0.0 size=86933250kB time=00:15:48.72 bitrate=750643.2kbits/s speed=0.938x
frame=21078 fps= 20 q=-0.0 size=184390344kB time=00:17:29.52 bitrate=1439244.3kbs/s speed=0.938x =0.995x
frame=21938 fps= 20 q=-0.0 size=19191365kB time=00:17:09.67 bitrate=750643.2kbits/s speed=0.938x
frame=22789 fps= 20 q=-0.0 size=199358172kB time=00:18:54.72 bitrate=1439244.3kbits/s speed=0.995x
frame=23647 fps= 20 q=-0.0 size=101780550kB time=00:18:30.76 bitrate=750643.2kbits/s speed=0.939x
frame=17357 fps= 14 q=-0.0 size=105443775kB time=00:19:10.74 bitrate=750643.2kbits/s speed=0.939x 0.995x
frame=17966 fps= 14 q=-0.0 size=109143450kB time=00:19:51.11 bitrate=750643.2kbits/s speeed=0.995x 3.2kbits/s speed=0.939x
frame=18558 fps= 14 q=-0.0 size=112739850kB time=00:20:30.36 bitrate=7539244.3kbits/s speed=0.995x
frame=27034 fps= 20 q=-0.0 size=236493432kB time=00:22:10.80 bitrate=750643.2kbits/s speed=0.939x
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 396956672.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 396956672.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 394219520.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 396956672.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 394219520.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 394219520.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 396956672.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 396956672.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 394219520.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 394219520.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 394219520.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 394219520.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 394219520.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 391482368.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 394219520.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 396956672.
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 13 objects to free up 80880384 bytes. The number of bytes in use (before this eviction) is 391482368.
Out of curiosity, has anyone managed to get hardware acceleration working on a Synology yet?
rc7 is still working great for me. It has handled multiple ffmpeg restarts and a detection process restart on its own.
Yes, I’m seeing a performance improvement with the Coral USB device. I don’t think they issues above are related. before I had the device i was getting about 1 fps and the video was very choppy
What is this referring to?
There is not enough space to create this object, so evicting X objects to free up XXXXXXXX bytes
is this the memory specified with the –shm-size= arg?
I have a weird issue, I am getting a default black square on the debug url when there is no frame detected:
frame = object_processor.get_current_frame(camera_name)
if frame is None:
frame = np.zeros((720,1280,3), np.uint8)
It’s happening randomly but it can be reproduced by deleting the container and recreating it.
The container seems to find my camera feed 1 out of 10 tries.
So I think it is happening because the ffprobe is hanging:
def get_frame_shape(source):
print(ffprobe_cmd)
p = sp.Popen(ffprobe_cmd, stdout=sp.PIPE, shell=True)
(output, err) = p.communicate()
p_status = p.wait()
info = json.loads(output)
print(info)
The print(ffprobe_cmd) command works, but not the print(info) - it hangs in between.
When the command hang I can see the ffprobe process staying open in the container:
root@5458115157f2:/opt/frigate# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 12.9 2.3 750900 93296 ? Ssl 00:26 0:02 python3.7 -u detect_objects.py
root 13 3.4 0.0 428300 3936 ? Sl 00:26 0:00 /usr/local/bin/plasma_store -m 400000000 -s /tmp/plasma
root 17 0.3 1.9 696916 77628 ? Sl 00:26 0:00 python3.7 -u detect_objects.py
root 18 0.0 1.4 603180 56576 ? S 00:26 0:00 python3.7 -u detect_objects.py
root 23 0.0 0.0 4624 848 ? S 00:26 0:00 /bin/sh -c ffprobe -v panic -show_error -show_streams -of json "rtsp://user:[email protected]:554/h264Preview_01_sub"
root 35 1.4 0.9 514028 37864 ? S 00:26 0:00 ffprobe -v panic -show_error -show_streams -of json rtsp://user:[email protected]:554/h264Preview_01_sub
I tried the ffprobe manually and it worked :-/
Partially. That means there isn’t enough room in the cache for the object it is trying to store. It should be cleaning up it’s own cache. What is your detection queue size when you get those messages?
I missed that you added those extra items.
I’ve added detection_queue and detection_start to template.yaml and will let you know when it fails again.
I wonder if ffprobe has a timeout parameter. Frigate will bypass ffprobe if you specify width and height parameters in the camera config. See if it works reliability that way.