Local realtime person detection for RTSP cameras

Hello everyone,

I have been looking into somehow integrating person detection into my security cameras and tying it all Lynn to home assistant for notifications. This seems like a dream come true. Thank you Blake!!

I am thinking of using a celeron based intel nuc to run frigate on with the Mini PCIe TPUs
I have a few questions.

Will frigate run with the PCIe accelerator card or must it be the usb one?

Is installing simply spinning up a docker?

Can I use frigate as a live monitor of my cams to see the rtsp streams?

Thank you all in advance for your feedback. I canā€™t wait to get started.

Hey, Frigate can run on RTX 2080 with GPU support added in this PR:


Would be great if you try it and share the performance you get.
1 Like

Awesome man, thanks for the heads up!!!

Looking at this it looks like all I need to do is run the container and pull out my coral. It should fall back to GPU automatically?

Object detection can run now on Nvidia CUDA GPUs. The logic of device selection is preserved: Edge TPU is detected and loaded first, if Coral isn't available, then GPU is determined, lastly, if no GPU, the program falls back to CPU.

Do you know what version container I need to run? Ill test right now. Im currently running 0.5.1-rc2

thanks again

Yes, just pull out the Coral in order to switch Frigate to GPU.
The container for GPU is not yet at Docker Hub, as it is just a PR, not a release or candidate. Wait till Blake reviews it and pushes the image. But the sources of the PR contain two Dockerfiles, you can pull the sources, build the containers yourself and run it. Make sure your Docker uses the NVIDIA Container Toolkit and recognises --gpus command line flag.

I have docker setup with nvidia already. Im using 19.03 and my daemon.json has:

{
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m"
    },
    "storage-driver": "overlay2",
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

using docker command

sudo docker run --rm \
--privileged \
--shm-size=1024m \
-v gpus=all \
-v /opt/frigate:/config:ro \
-v /etc/localtime:/etc/localtime:ro \
-p 5000:5000 \
blakeblackshear/frigate:0.5.1-rc2

But it doesnt try for gpu, it sees the coral is gone and goes strraight to cpuā€¦I feel like Im using the wrong version because Ive had this one running like a month. I didnt see any new versions though.

Here is the log (redacted stream urls)

da@deep:/opt$ sudo docker run --rm \
> --privileged \
> --shm-size=1024m \
> -v gpus=all \
> -v /opt/frigate:/config:ro \
> -v /etc/localtime:/etc/localtime:ro \
> -p 5000:5000 \
> blakeblackshear/frigate:0.5.1-rc2
On connect called
/arrow/cpp/src/plasma/store.cc:1226: Allowing the Plasma store to use up to 0.4GB of memory.
/arrow/cpp/src/plasma/store.cc:1253: Starting object store with directory /dev/shm and huge page support disabled
ffprobe -v panic -show_error -show_streams -of json "rtsp://10.2.2.2:7447/xxx"
Starting detection process: 38
No EdgeTPU detected. Falling back to CPU.
{'streams': [{'index': 0, 'codec_name': 'aac', 'codec_long_name': 'AAC (Advanced Audio Coding)', 'profile': 'LC', 'codec_type': 'audio', 'codec_time_base': '1/48000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 'fltp', 'sample_rate': '48000', 'channels': 1, 'channel_layout': 'mono', 'bits_per_sample': 0, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/48000', 'start_pts': 0, 'start_time': '0.000000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'Main', 'codec_type': 'video', 'codec_time_base': '1/30', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 1024, 'height': 576, 'coded_width': 1024, 'coded_height': 576, 'has_b_frames': 0, 'sample_aspect_ratio': '1:1', 'display_aspect_ratio': '16:9', 'pix_fmt': 'yuvj420p', 'level': 31, 'color_range': 'pc', 'color_space': 'bt709', 'color_transfer': 'bt709', 'color_primaries': 'bt709', 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '25/1', 'avg_frame_rate': '15/1', 'time_base': '1/90000', 'start_pts': 48001, 'start_time': '0.533344', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
Creating ffmpeg process...
ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://10.2.2.2:7447/xxx -f rawvideo -pix_fmt rgb24 pipe:
ffprobe -v panic -show_error -show_streams -of json "rtsp://10.2.2.2:7447/xxx"
drive: ffmpeg didnt return a frame. something is wrong.
drive: ffmpeg didnt return a frame. something is wrong.
drive: ffmpeg process is not running. exiting capture thread...
{'streams': [{'index': 0, 'codec_name': 'aac', 'codec_long_name': 'AAC (Advanced Audio Coding)', 'profile': 'LC', 'codec_type': 'audio', 'codec_time_base': '1/48000', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'sample_fmt': 'fltp', 'sample_rate': '48000', 'channels': 1, 'channel_layout': 'mono', 'bits_per_sample': 0, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/48000', 'start_pts': 0, 'start_time': '0.000000', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}, {'index': 1, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'Main', 'codec_type': 'video', 'codec_time_base': '1/30', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 1024, 'height': 576, 'coded_width': 1024, 'coded_height': 576, 'has_b_frames': 0, 'sample_aspect_ratio': '1:1', 'display_aspect_ratio': '16:9', 'pix_fmt': 'yuvj420p', 'level': 31, 'color_range': 'pc', 'color_space': 'bt709', 'color_transfer': 'bt709', 'color_primaries': 'bt709', 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'false', 'nal_length_size': '0', 'r_frame_rate': '25/1', 'avg_frame_rate': '15/1', 'time_base': '1/90000', 'start_pts': 33600, 'start_time': '0.373333', 'bits_per_raw_sample': '8', 'disposition': {'default': 0, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}}]}
Creating ffmpeg process...
ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://10.2.2.2:7447/xxx -f rawvideo -pix_fmt rgb24 pipe:
Camera_process started for drive: 49
Starting process for drive: 49
Camera_process started for front: 50
Starting process for front: 50
 * Serving Flask app "detect_objects" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
front: ffmpeg didnt return a frame. something is wrong.
front: ffmpeg didnt return a frame. something is wrong.
front: ffmpeg process is not running. exiting capture thread...
Creating ffmpeg process...
ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://10.2.2.2:7447/xxx -f rawvideo -pix_fmt rgb24 pipe:
Creating ffmpeg process...
ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://10.2.2.2:7447/xxx -f rawvideo -pix_fmt rgb24 pipe:
^C^CTraceback (most recent call last):
  File "detect_objects.py", line 337, in <module>
    main()
  File "detect_objects.py", line 332, in main
    object_processor.join()
  File "/usr/lib/python3.7/threading.py", line 1044, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.7/threading.py", line 1060, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt
^CException ignored in: <module 'threading' from '/usr/lib/python3.7/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 1307, in _shutdown
    lock.acquire()
KeyboardInterrupt

Not sure if this will help but incaseā€¦

da@deep:~$ nvidia-smi
Thu Apr  9 08:36:15 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2080    On   | 00000000:07:00.0 Off |                  N/A |
| 24%   24C    P8     3W / 245W |     57MiB /  7973MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1620      G   /usr/lib/xorg/Xorg                            38MiB |
|    0      1880      G   /usr/bin/gnome-shell                          17MiB |
+-----------------------------------------------------------------------------+
da@deep:~$

BTW - didnt make any changes to my config file, its the same as it was for the coral

Ah sorry I jumped the gun. Thanks for the info Iā€™ll keep my eye out and let you know

The image is not yet at Docker Hub, so one needs to build it.

  1. Pull the sources from https://github.com/asmirnou/frigate/tree/gpu_support
  2. cd frigate
  3. git checkout gpu_support
  4. docker build . -f Dockerfile -t frigate
  5. docker build . -f Dockerfile.gpu -t frigate-gpu
  6. and only then
sudo docker run --rm \
--gpus all \
--privileged \
--shm-size=1024m \
-v /opt/frigate:/config:ro \
-v /etc/localtime:/etc/localtime:ro \
-p 5000:5000 \
frigate-gpu
1 Like

your the man, thanks dude! Its up and running and it is using my gpu. How should I test? I have 3 cameras now but could add a couple moreā€¦Im also mainly using unifi cameras set at 1024x578 something like that set to 15fps. Should I make it 1080/30fps

I do see inference speed normally sat at 7.5 and now is 3.2

To estimate the maximum throughput just share inference time from
http://localhost:5000/debug/stats
the maximum FPS can be calculated as 1000/inference time. If it less than 10 ms, then Frigate on GPU can process 100+ FPS.

Cool, then RTX 2080 is two times faster than Coral :slight_smile:

make sure Im reading it rightā€¦

{"coral":{"detection_start":0.0,"fps":0.0,"inference_speed":3.2,"pid":36},"drive":{"detection_fps":0.0,"ffmpeg_pid":123,"fps":15.0,"pid":78,"read_start":1586423390.79056,"skipped_fps":0.0},"front":{"detection_fps":0.0,"ffmpeg_pid":126,"fps":15.0,"pid":79,"read_start":1586423390.747784,"skipped_fps":0.0},"garage":{"detection_fps":0.0,"ffmpeg_pid":76,"fps":10.2,"pid":80,"read_start":1586423390.525449,"skipped_fps":0.0},"plasma_store_rc":null}

Awesomeā€¦thank you again!!!

its barely using the cardā€¦

da@deep:~$ nvidia-smi
Thu Apr  9 09:13:08 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2080    On   | 00000000:07:00.0 Off |                  N/A |
| 24%   24C    P8     4W / 245W |    764MiB /  7973MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1620      G   /usr/lib/xorg/Xorg                            38MiB |
|    0      1880      G   /usr/bin/gnome-shell                          17MiB |
|    0     28943      C   python3                                      695MiB |
+-----------------------------------------------------------------------------+

It doesnā€™t mean it is always at 0%. Try watch nvidia-smi, as the usage varies.

ok wow, I just went and looked I had my fps rurned down to 6 for 2 cameras (front and drive). Garage is a wyze. I just bumped both front and drive to 25fps each and it turned the inference speed to 10.0ā€¦

It seems like your card supports Half precision. You can boost performance by enabling this mode by specifying -e TRT_FLOAT_PRECISION=16 in the command line. Make sure your previous Docker container (but not the image youā€™ve built) is deleted (if you provide --rm flag all the time, it is deleted automatically after stopping). Once a new container starts, it builds an optimised engine, which with the Half precision enabled, can provide less inference time.

BWT, because GPU container builds optimised engine at startup, the container doesnā€™t need to be created every time you run it. Docker container can be stopped and then started again with attached stdout. It will start faster next time as the engine will be loaded from saved file inside the container.

The inference time is summed from all the cameras, so 10.0 is probably the sum of 3.2+3.2+3.2 from 3 cameras youā€™ve got. But the inference is accounted only when motion is detected. So first time you saw 3.2 only because motion was detected on one camera (swaying leaves maybe). As soon as you got out, all 3 cameras detected motion and passed the images for analysis in object detector (where inference time summed).

Ignore this post, I mixed up with coral.fps

Ah that makes a lot of sense. Thanks for explaining, will have to see here in a couple hours theyā€™ll be more active.

I do keep the containers around, I usually name them so I can just do docker start/stop/restart frigate. Iā€™ll blow that away and add the command. One sec