Local realtime person detection for RTSP cameras

This line is saying that your RTSP url returned a 404 Not Found. Are you sure you have the right url?

Can you build the ffmpeg-subprocess branch and run benchmark.py? I am curious to see how the RPi4 performs. Hoping mine ships today.

Average inference time: 12.464006978034973
pi@raspberrypi:~/frigate-0.2.0-beta $

Looks like one of my sub streams died on one of the cameras, weird. Looks like all is well on the Pi4, good inference times too!

Not bad. It is probably fine if you are only running a limited # of regions. When I get mine, I will work on hwaccel for the Pi.

1 Like

Okay, so my test with one landscape orientation 1080p camera with a mask is coming to a close after a little more than a day. The docker container is pretty much completely unresponsive and the stream output from frigate is only updating a frame about once every few minutes. Hereā€™s what I could get from the terminal, it wonā€™t give me a full log. Itā€™s just repeating this. You can compare to my last post which shows what it looks like when I have just started it and itā€™s working.

    encoder         : Lavf57.83.100                                             
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 1920x1080, q=2-3
1, 199065 kb/s, 4 fps, 4 tbn, 4 tbc                                             
    Metadata:                                                                   
      encoder         : Lavc57.107.100 rawvideo                                 
last frame is more than 2 seconds old, restarting camera capture...             
Terminating the existing capture process...                                     
Creating a new capture process...                                               
Starting a new capture process...                                               
ffmpeg -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict ex
perimental -fflags +genpts -rtsp_transport tcp -stimeout 5000000 -use_wallclock_
as_timestamps 1 -i rtsp://uname:[email protected]:554
/Sms=4.unicast -f rawvideo -pix_fmt rgb24 pipe:                                 
ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg develop
ers                                                                             
  built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)                                     
  configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hard
ened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu -
-enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-g
nutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enab
le-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-li
bfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lam
e --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus -
-enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --ena
ble-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libthe
ora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack -
-enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libz
mq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 -
-enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --en
able-frei0r --enable-libopencv --enable-libx264 --enable-shared                 
  libavutil      55. 78.100 / 55. 78.100                                        
  libavcodec     57.107.100 / 57.107.100                                        
  libavformat    57. 83.100 / 57. 83.100                                        
  libavdevice    57. 10.100 / 57. 10.100                                        
  libavfilter     6.107.100 /  6.107.100                                        
  libavresample   3.  7.  0 /  3.  7.  0                                        
  libswscale      4.  8.100 /  4.  8.100                                                                             
  libswresample   2.  9.100 /  2.  9.100                                        
  libpostproc    54.  7.100 / 54.  7.100                                        
Input #0, rtsp, from 'rtsp://uname:[email protected]:
554/Sms=4.unicast':                                                                                                                         
  Metadata:                                                                     
    title           : Session streamed by "ssrtspd"                             
    comment         : 1,4_high.unicast                                          
  Duration: N/A, start: 1563394347.384278, bitrate: N/A                         
    Stream #0:0: Video: h264 (Main), yuv420p(progressive), 1920x1080, 4 fps, 4 t
br, 90k tbn, 8 tbc                                                              
Stream mapping:                                                                 
  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))                      
Press [q] to stop, [?] for help                                                 
frame=    0 fps=0.0 q=0.0 size=       0kB time=-577014:32:22.77 bitrate=  -0.0kb
frame=    0 fps=0.0 q=0.0 size=       0kB time=-577014:32:22.77 bitrate=  -0.0kb
frame=    0 fps=0.0 q=0.0 size=       0kB time=-577014:32:22.77 bitrate=  -0.0kb
frame=    0 fps=0.0 q=0.0 size=       0kB time=-577014:32:22.77 bitrate=  -0.0kb
Output #0, rawvideo, to 'pipe:':                                                
  Metadata:                                                                     
    title           : Session streamed by "ssrtspd"                             
    comment         : 1,4_high.unicast                                          
    encoder         : Lavf57.83.100                                             
    Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 1920x1080, q=2-3
1, 199065 kb/s, 4 fps, 4 tbn, 4 tbc                                             
    Metadata:                                                                   
      encoder         : Lavc57.107.100 rawvideo                                 

Are you able to keep it going for a couple days or more against a hikvision camera? If so, could you post your video settings?

Can you run htop to see if a particular process is using lots of CPU or memory in the container? Something has to be working hard.

Iā€™ve already stopped it and started the old one with the low res streams and no masks againā€¦ Iā€™m curious how long it will run. It was going pretty well last timeā€¦ Iā€™ll try to look at htop if it bogs down again, but synology does include a little process monitor which looks like this, and itā€™s never shown anything unusual when it starts to bogā€¦

Also CPU and memory have never really grown much from where it starts (this is on the smaller feeds and old version so not the same, but it doesnā€™t move muchā€¦ At least not nearly enough to challenge resources on the synologyā€¦ I donā€™t have a resource cap set for frigate.

Was there anything of use in the logs?

The rtsp url you are using does not seem to match anything I can find about your camera. Are you not connecting directly to the camera?

I changed the RTSP Url because I donā€™t want to publish that information on the internetā€¦ I am currently taking the feed from synology surveillance station, not from the camera because it looks like my camera canā€™t serve up 2 1080p feeds at onceā€¦ Also it seems better to just have synology feed it to another process running on the same host so as to avoid having to go out on the network. Suffice it to say it works well enough to run for a 12+ hours making bounding boxes, so Iā€™m quite sure itā€™s a working URL.

But Iā€™ve gone back to using the subfeed from the camera for nowā€¦

Would you be willing to try with frigate as the only consumer of that 1080p feed for a bit? That would tell us if having the synology as a middleman is the issue. Also, what is that feature called? I donā€™t have a synology.

So I have had the Pi up for roughly a day since I restarted it last night, one of seven feeds is down and didnā€™t recover. There are some ā€œlast frame is more than 2 seconds oldā€¦ā€ messages and ā€œOpening the RTSP URLā€ā€¦

Iā€™m using substreams and pulling the video via the Hikvision NVRā€¦

The feature is called ā€œshare stream pathā€.

I wouldnā€™t mind trying it directly against the 1080p stream of the camera, though I have done it many times already against the 704x568 substream of the cameraā€¦ In most cases except the last when I put it to fixed bitrate ending in the same result as with the synologyā€¦ Also I tried it against the 720p feed with the same resultā€¦

For now, Iā€™m going to see how long I can keep it up against the fixed bitrate substream with no maskā€¦ I want to see if thereā€™s one scenario in which it works properlyā€¦

In my experience, the ā€œlast frame is more than 2 seconds oldā€ is the beginning of the endā€¦ They will become more and more frequent over the space of a few hours until you can no longer open the frigate streamā€¦ So I suspect if you run it for longer you will start to see the same problemsā€¦ Iā€™m interested to hear your results.

I wonder if the process isnā€™t actually exiting, and it is creating a duplicate connection. Over time the number of connections would probably slow things down like that.

If restarting the container fixes it, that almost has to be the problem.

I think I know what to try. Will try and push up a new image tonight.

Ok. I completely revamped the way that I am running the ffmpeg subprocess so that I can make sure it is correctly killed in the event frames stop coming in. I tested it several different ways, and I really think this will solve the issues you are having. Updated image is pushed to blakeblackshear/frigate:0.2.0-beta.

2 Likes

That sounds very promising, will test it soon! Thanks!

When is your PI slated to arrive?