Local realtime person detection for RTSP cameras

[blakeblackshear]

The not enough space messages are not necessarily a problem. They are an indication that there are more frames in cache and frigate isn’t processing them fast enough. In that situation, the cache frees up some space automatically to make room for the newer frames. Unless your video feeds stop completely, it should continue just fine. If it happens continuously, then I would recommend lowering the fps or using take_frame to reduce the fps.

For the queue size, I had to switch to a different type of queue that doesn’t deadlock with multi processing. This queue does not have a way to measure the queue size.

[mr-onion]

Thanks for the quick response.

It did indeed stop working unfortunately. The message was being repeated in the log every X seconds for two hours until I restarted it. I believe the debug video stream was still running (not frozen) before I restarted, but there were no persons detected when there should have been.

I only have four low res 640x480 streams running at 8fps so I don’t think it should be hitting any limits. I did have a fifth camera until a squirrel chewed through the cable, but even then it ran fine 24x7 on previous versions.

I’ll try and do some more debugging if it does it again, and then reduce the fps to 5, to see if it makes a difference.

Many thanks

[surge919]

I’ve seen references to reconnect flags… for example… -reconnect 1 -reconnect_at_eof 1 -reconnect_streamed 1 -reconnect_delay_max 2000… but these references seem to be relatively old.

I’ve also seen references to 4.x versions coming in while searching which seems to possibly address some of the issues I’ve been seeing. Have you tested 4.x?

[blakeblackshear]

I have, but it broke hw acceleration on some setups.

[blakeblackshear]

You shouldn’t need to reduce the fps at that resolution. I will look and see how it would be possible for the debug stream to keep working, but detection to stop.

[mr-onion]

Thanks

Not sure if this is of any help. The FPS on each camera started going very erratic at the same time the plasma messages started, and both the coral fps and inference value stopped updating from the same time. Where it goes back to normal at about 1445 is after I restarted it.

[surge919]

I’ll keep looking for clues. Do you have any suggestions as to what else I should look for or change in the config?

[blakeblackshear]

I would still suggest running the ffmpeg command from my earlier post directly to see if it exits too. I have an idea for something that might help in the code.

[surge919]

OK, I’ll try it again.

This is what you posted earlier in the thread

ffmpeg -loglevel verbose -rtsp_transport tcp -i rtsp://user:[email protected]/live -f rawvideo -pix_fmt rgb24 pipe: > /dev/null

Here’s what is shown with a ps in the container when it’s run normally.

ffmpeg -loglevel info -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://user:[email protected]/live -f rawvideo -pix_fmt rgb24 pipe: > /dev/null

Once I comment out the driveway camera, which do you want me to run?

[blakeblackshear]

The second one.

[surge919]

Here’s the full run. From start to failure

root@5a79ef3d62ad:/opt/frigate# ffmpeg -loglevel info -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://user:[email protected]/live -f rawvideo -pix_fmt rgb24 pipe: > /dev/null
ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Guessed Channel Layout for Input Stream #0.1 : mono
Input #0, rtsp, from 'rtsp://user:[email protected]/live':
  Metadata:
    title           : Session streamed by "wyze"
    comment         : live
  Duration: N/A, start: 1584469562.473567, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuv420p(progressive), 1920x1080, 15 fps, 15 tbr, 90k tbn, 30 tbc
    Stream #0:1: Audio: pcm_alaw, 8000 Hz, mono, s16, 64 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))
Press [q] to stop, [?] for help
Output #0, rawvideo, to 'pipe:':      0kB time=-577014:32:22.77 bitrate=  -0.0kbits/s speed=N/A
  Metadata:
    title           : Session streamed by "wyze"
    comment         : live
    encoder         : Lavf57.83.100
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 1920x1080, q=2-31, 746496 kb/s, 15 fps, 15 tbn, 15 tbc
    Metadata:
      encoder         : Lavc57.107.100 rawvideo
[rawvideo @ 0x564e5320fcc0] Encoder did not produce proper pts, making some up.
[h264 @ 0x564e531f8380] error while decoding MB 104 47, bytestream -546496.0kbits/s speed=0.962x
frame= 1524 fps=9.0 q=-0.0 Lsize= 9258300kB time=00:01:41.60 bitrate=746496.0kbits/s speed=0.597x
video:9258300kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
root@5a79ef3d62ad:/opt/frigate#

[hasshoolio]

Regarding your first paragraph - would you expect the values in the log message to change?

For my issue the same values repeat over and over, even across restarts. It’s always:

frigate-new | /arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 29 objects to free up 80201472 bytes. The number of bytes in use (before this eviction) is 398512640.

[blakeblackshear]

They should be the same because your frame size is always the same.

[blakeblackshear]

Try running that same command in the 0.3.0 container or whichever version was stable for you previously?

[surge919]

The version I stated using was 0.5.0-rc1 since I didn’t have a Coral device.

I’ve started that ffmpeg command in the container.

How will the version of frigate make a difference? If the issue is the ffmpeg parameters it will fail on both versions.
This seems to be an issue with how the parent process deals with the issue since in all cases, when the issue happens, a restart of the parent process fixes it.

[blakeblackshear]

I was hoping to confirm that ffmpeg does behave the same in both container versions. I wasn’t restarting the parent process in 0.3.0, so I am wondering if it wasn’t failing as frequently for some reason. I would also expect it to behave the same, so I don’t yet see how 0.3.0 would have been more stable.

[surge919]

Here’s the output from the latest run
This was run directly from the container as mentioned.

root@c37cec3a7e0d:/opt/frigate# ffmpeg -loglevel info -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://user:[email protected]/live -f rawvideo -pix_fmt rgb24 pipe: > /dev/null
ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100
Guessed Channel Layout for Input Stream #0.1 : mono
Input #0, rtsp, from 'rtsp://user:[email protected]/live':
  Metadata:
    title           : Session streamed by "wyze"
    comment         : live
  Duration: N/A, start: 1584478300.503033, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuv420p(progressive), 1920x1080, 15 fps, 15 tbr, 90k tbn, 30 tbc
    Stream #0:1: Audio: pcm_alaw, 8000 Hz, mono, s16, 64 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))
Press [q] to stop, [?] for help
Output #0, rawvideo, to 'pipe:':      0kB time=-577014:32:22.77 bitrate=  -0.0kbits/s speed=N/A
  Metadata:
    title           : Session streamed by "wyze"
    comment         : live
    encoder         : Lavf57.83.100
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 1920x1080, q=2-31, 746496 kb/s, 15 fps, 15 tbn, 15 tbc
    Metadata:
      encoder         : Lavc57.107.100 rawvideo
[rawvideo @ 0x55820e9412c0] Encoder did not produce proper pts, making some up.
[rtsp @ 0x55820e91c740] CSeq 41 expected, 0 received.:00.80 bitrate=746496.0kbits/s speed=0.961x
frame=16216 fps= 14 q=-0.0 Lsize=98512200kB time=00:18:01.06 bitrate=746496.0kbits/s speed=0.951x
video:98512200kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
root@c37cec3a7e0d:/opt/frigate#

[surge919]

I know you said it works fine with your Wyze cam, but how much movement is it detecting?

Mine’s been steady on the last version until it detected motion. Once that happened, it spit out the following.

frame=25487 fps= 20 q=-0.0 size=222960276kB time=00:21:14.35 bitrate=746496.0kbits/s speed=0.976x
frame=26650 fps= 20 q=-0.0 size=233055468kB time=00:22:12.05 bitrate=1433272.3kbits/s speed=0.999x
[h264 @ 0x562c1905a800] error while decoding MB 60 37, bytestream -5=1433272.3kbits/s speed=0.995x    speed=0.999x
No more output streams to write to, finishing.me=00:21:11.26 bitrate=746496.0kbits/s speed=0.938x     33272.3kbits/s speed=0.995x
2020-03-17T23:34:25.322845196Z frame=19069 fps= 14 q=-0.0 Lsize=115844175kB time=00:21:11.26 bitrate=746496.0kbits/s speed=0.938x
2020-03-17T23:34:25.322994018Z video:115844175kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
2020-03-17T23:34:25.323071763Z Input file #0 (rtsp://user:[email protected]/live):
2020-03-17T23:34:25.323138281Z   Input stream #0:0 (video): 19077 packets read (214656098 bytes); 19069 frames decoded;
2020-03-17T23:34:25.323207167Z   Input stream #0:1 (audio): 8130 packets read (8672000 bytes);
2020-03-17T23:34:25.323268875Z   Total: 27207 packets (223328098 bytes) demuxed
2020-03-17T23:34:25.323330178Z Output file #0 (pipe:):
2020-03-17T23:34:25.323390964Z   Output stream #0:0 (video): 19069 frames encoded; 19069 packets muxed (118624435200 bytes);
2020-03-17T23:34:25.323454524Z   Total: 19069 packets (118624435200 bytes) muxed
2020-03-17T23:34:25.484162199Z driveway: ffmpeg didnt return a frame. something is wrong.
2020-03-17T23:34:25.484408805Z driveway: ffmpeg didnt return a frame. something is wrong.
2020-03-17T23:34:25.484616281Z driveway: ffmpeg didnt return a frame. something is wrong.
.
.
.
.
2020-03-17T23:34:25.506845818Z driveway: ffmpeg didnt return a frame. something is wrong.
2020-03-17T23:34:25.506909140Z driveway: ffmpeg process is not running. exiting capture thread...
2020-03-17T23:34:29.314020657Z Creating ffmpeg process...
2020-03-17T23:34:29.314687256Z ffmpeg -loglevel verbose -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://user:[email protected]/live -f rawvideo -pix_fmt rgb24 pipe:
ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg developers/s speed=0.996x    33272.3kbits/s speed=0.995x
2020-03-17T23:34:30.356934128Z   built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
2020-03-17T23:34:30.357053379Z   configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
2020-03-17T23:34:30.357258607Z   libavutil      55. 78.100 / 55. 78.100
2020-03-17T23:34:30.357336767Z   libavcodec     57.107.100 / 57.107.100

[bk55 (brian)]

After several hours, 0.5.1rc2, keep showing following message and it won’t recover:

/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 87 objects to free up 80246016 bytes. The number of bytes in use (before this eviction) is 399657472.

I have 2 cameras and it usually affects one of them. CPU usage increases and so does the fps and detection. The debug output shows grey/blue rapidly flashing image.

[bk55 (brian)]

Correction : after about 20 minutes, it did recover after the last occurrence.