Local realtime person detection for RTSP cameras

Hi, can you recommend hardware for 16 x 4Mpixel Hikvision cameras? Extremely interesting project, i would try it, how many Corals do i need to buy and what type of NUC should do the job?

It depends on what resolution you want to analyze with frigate, what FPS, and how much motion you typically see. My recommendation is 5fps, and the resolution for each camera should be set so that the smallest person you want to detect is no more than 300px tall. Any resolutions higher than than will not gain any accuracy, and may lower accuracy. The number of corals depends on how much motion you generally have on your cameras. A setup with 100 cameras that rarely have motion will be fine with a single coral vs a setup with 5 cameras on a busy street may need more. You can also be more or less aggressive with masking to avoid motion where possible.

HD is the stream i usually stream into the NVR. 5fps is totally fine as you wrote. This case what NUC would be good? Does your soft work with Jetson NANO. Nano is better for this, as i see https://www.youtube.com/watch?v=RgTXqP8yZ_Y

It doesn’t work on the Nano currently. Looking at the comments, it looks like Coral actually outperforms the Nano.

A mix-up in that test result table, coral dev is 20.7 ms and NVIDIA Jetson nano is 72 ms. The audio is correct if you listen to it, but the column order is wrong. Thank you for the pointing it out

yes, hope the guys speech is correct. OK for me no issue to buy coral as i dont use any hardware yet. one question, if for some reason 1 x USB coral is not enough, can i just add plug more into the NUC? Does your software support multiple corals? Some people having issue darkmode detection, can you improve it? Reason i am asking i would use this in a big warehouse, it would not be a hobby installation, so i will be responsible for it

It doesn’t support multiple Corals yet, but it will in the future. Accuracy at night can certainly be improved and it is a need for home installations too. I can’t make any promises as this is a hobby project for me that I provide for free and work on in my very limited spare time.

extremely needed such project like this, if no multiple coral support yet i am afraid a bit that my 16 cams will be way too much for 1 unit. I keep watching this forum if anyone use it with more than 10 cams. Similar project is shinobi.video but this use multiple expensive GPUs, thats why i dont understand how it is possible to do it with a 1x coral

Hi Black, this really is an excellent piece of software. I now have it running on a Pi 4. I connected one Reolink 511 camera bu§t have had to run on sub channel (640x480) as using the default (2304x1296) slowed down to the point it could not keep up…it works ok on the 640x480 detecting people down to 1000 pixel sq area…would adding a Coral help? Thanks Tim

1 Like

sorry i meant Blake!

You can just run 4 instances on 4 VMs for now. Map only 1 USB hub with 1 coral through to each VM… Haven’t heard of anyone doing it, but I think it would have a good chance of working.

as i heard vmware esxi dropping or already dropped USB pass thru support, and also you may not be also to pass several USB stick to differnt VMs. So, this is not a good route even it seems to be working now.

1 Like

I think someone figured it out on proxmox as well (or does that use esxi as well).

i suffer with vmware esxi at work and i know free version has limitation on USB pass-true, it worked year ago fine, but they want make money with it. Never use proxmox, but as a linux guy i would not even use virtualization for this, lot more relaible if you just throw a USB stick on the Ubuntu itself and run the or build the soft on it.

For reference I run 7 1080p cameras at 5fps on this NUC: https://www.amazon.com/gp/product/B01MSZLO9P/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

It only uses a single core normally and the Coral almost never hits capacity.

1 Like

you should make a little youtube demo video about it, many thanks and congrats on this job, really awesome!! OK, i probably give this a try, worst case i wont dump on all of my cameras.

Yeah CPU isn’t an issue even for 16 cameras, an i3 NUC should be fine. But I think you would hit the wall of a single coral with 16 cameras… Might be able to squeak by on 2 corals though.

What is the spec on your processor and memory size?

thats why I asked the developer to add the possibility for multiple corals in the same NUC.

@blakeblackshear Thanks for adding the zone and clips function. I like that I can now send small video clips in my Telegram notification.

I am seeing a lot of following errors after about 14hrs of running new 0.6.0-rc1 container. Any suggestion how to solve or troubleshoot further?

2020-09-01T02:34:25.301397116Z [h264 @ 0x5587643e3f20] error while decoding MB 0 1, bytestream -53
2020-09-01T02:34:25.303484355Z [h264 @ 0x5587643e3f20] top block unavailable for requested intra mode
2020-09-01T02:34:25.303502270Z [h264 @ 0x5587643e3f20] error while decoding MB 10 0, bytestream 187210
2020-09-01T02:34:25.310045887Z [h264 @ 0x5587643e3f20] error while decoding MB 28 9, bytestream -19
2020-09-01T02:34:28.214358357Z No frames received from back in 5 seconds. Exiting ffmpeg...
2020-09-01T02:34:28.214435720Z Waiting for ffmpeg to exit gracefully...
2020-09-01T02:34:28.222781353Z back: ffmpeg didnt return a frame. something is wrong.
2020-09-01T02:34:28.223220288Z Exception in thread back:
2020-09-01T02:34:28.223243734Z Traceback (most recent call last):
2020-09-01T02:34:28.223249870Z   File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
2020-09-01T02:34:28.223255824Z     self.run()
2020-09-01T02:34:28.223261970Z   File "/opt/frigate/frigate/video.py", line 147, in run
2020-09-01T02:34:28.223270860Z     frame_bytes = self.ffmpeg_process.stdout.read(self.frame_size)
2020-09-01T02:34:28.223279258Z ValueError: read of closed file
2020-09-01T02:34:28.223320392Z 
2020-09-01T02:34:35.378316811Z [h264 @ 0x55e70ef34f20] top block unavailable for requested intra mode -1
2020-09-01T02:34:35.378336489Z [h264 @ 0x55e70ef34f20] error while decoding MB 20 0, bytestream 187030
2020-09-01T02:34:35.380326981Z [h264 @ 0x55e70ef34f20] error while decoding MB 9 6, bytestream -9
2020-09-01T02:34:35.383299920Z [h264 @ 0x55e70ef34f20] error while decoding MB 7 6, bytestream -5
2020-09-01T02:34:35.386373630Z [h264 @ 0x55e70ef34f20] error while decoding MB 13 5, bytestream -12
2020-09-01T02:34:38.229866119Z Creating ffmpeg process...
2020-09-01T02:34:38.229896729Z ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD129 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -r 7 -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://user:[email protected]:554//h264Preview_01_sub -f rawvideo -pix_fmt rgb24 pipe:
2020-09-01T02:34:45.461540563Z [h264 @ 0x56287167af20] error while decoding MB 8 0, bytestream -25
2020-09-01T02:34:45.463629424Z [h264 @ 0x56287167af20] top block unavailable for requested intra mode -1
2020-09-01T02:34:45.463640957Z [h264 @ 0x56287167af20] error while decoding MB 9 0, bytestream 190651
2020-09-01T02:34:45.466596341Z [h264 @ 0x56287167af20] error while decoding MB 4 13, bytestream -5
2020-09-01T02:34:55.549341284Z [h264 @ 0x56151af0df20] error while decoding MB 9 1, bytestream -15
2020-09-01T02:34:55.551639753Z [h264 @ 0x56151af0df20] top block unavailable for requested intra mode -1
2020-09-01T02:34:55.551715124Z [h264 @ 0x56151af0df20] error while decoding MB 9 0, bytestream 184979
2020-09-01T02:34:55.553267728Z [h264 @ 0x56151af0df20] left block unavailable for requested intra mode
2020-09-01T02:34:55.553276628Z [h264 @ 0x56151af0df20] error while decoding MB 0 28, bytestream 2358
2020-09-01T02:35:05.647217338Z [h264 @ 0x563a83429f20] error while decoding MB 2 22, bytestream -35
2020-09-01T02:35:05.648789288Z [h264 @ 0x563a83429f20] top block unavailable for requested intra mode
2020-09-01T02:35:05.648800527Z [h264 @ 0x563a83429f20] error while decoding MB 10 0, bytestream 157675
2020-09-01T02:35:15.729444316Z [h264 @ 0x5597f3bfaf20] error while decoding MB 0 11, bytestream -27
2020-09-01T02:35:15.732454361Z [h264 @ 0x5597f3bfaf20] top block unavailable for requested intra mode
2020-09-01T02:35:15.732483504Z [h264 @ 0x5597f3bfaf20] error while decoding MB 10 0, bytestream 175524
2020-09-01T02:35:25.821894617Z [h264 @ 0x557ad8876f20] error while decoding MB 11 4, bytestream -7
2020-09-01T02:35:25.824446681Z [h264 @ 0x557ad8876f20] top block unavailable for requested intra mode
2020-09-01T02:35:25.824458776Z [h264 @ 0x557ad8876f20] error while decoding MB 10 0, bytestream 167659
2020-09-01T02:35:35.904574829Z [h264 @ 0x5563da000f20] error while decoding MB 1 15, bytestream -63

Hi, I have Ubuntu 18.04 and look what i keep getting when i run the dockerfile:

  Downloading https://files.pythonhosted.org/packages/2f/b4/2ddaaecc332e6ddafb7726abb6139955a99282afe5f370930890bb572707/opencv-python-headless-4.4.0.42.tar.gz (88.9MB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-1an_3_dg/opencv-python-headless/setup.py", line 9, in <module>
        import skbuild
    ModuleNotFoundError: No module named 'skbuild'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-1an_3_dg/opencv-python-headless/