Local realtime person detection for RTSP cameras

zhide.p · July 22, 2020, 2:31am

just finally got it working in esxi vm with pass-through by passing the usb controller into the guest machine. seems to be working well just extra 5 ms delay in a normally machine.

below link shows you how enable pass-through the onboard usb controller

https://www.vediotalk.com/archives/13671

zhide.p · July 22, 2020, 2:38am

hello,

is there any workaround to below issue? i try to set the bouncing box size properly in order to detect a real person the furthest area, but can’t prevent a false alarm close to the camera. It’s keep turning on the light at night.

  frontdoor_01:
    ffmpeg:
      input: rtsp://admin:[email protected]/cam/realmonitor?channel=1&subt                                          ype=0
    take_frame: 4
    mask: frontdoor_01.bmp
    snapshots:
      show_timestamp: False
    objects:
      track:
        - person
        - car
      filters:
        person:
          min_area: 8000
          max_area: 300000
          threshold: 0.62
        car:
          min_area: 30000
          max_area: 10000000
          threshold: 0.60

scstraus · July 22, 2020, 4:00pm

Yes, this is the tradeoff until we have a better model or more advanced false positive rejection techniques. I have set my min size so that it will not detect the most far away humans so as to avoid close cats, but if the humans are far away they can’t do much harm . The only other option is to mask off that corner of the box if it keeps detecting that.

ebendl · July 26, 2020, 8:59pm

Is anybody processing full HD images? What type of CPU usage are you getting?

I’m trying 3 Hikvision 2mpx cameras, running on Ubuntu on an old i3 laptop, with QuickSync supposedly enabled. I used to run the cameras with the 640x480 substream, but I realized the one out back is not picking up persons too far from the camera, while the other two pick up people quite well but also picks up other animals (dogs and cats) as humans.

I then tested out the camera out back with the main, Full HD stream. This actually made the Frigate person detection much more “sensitive”:

This is the 640x480 stream:

But the CPU usage spiked dramatically – up from 1.6% up to 33%!

Other things that are different:

The main stream is variable bitrate, full HD, vs fixed in 2nd stream
The main stream is 25 fps, vs 4 fps in 2nd stream
Main stream’s max bitrate is 8192 vs 256 in 2nd stream
I’ve set frigate’s config to skip frames so that both become an effective 2fps, but I’m guessing this doesn’t really help FFMPEG, right?

The laptop is an i3-2348M – so perhaps it doesn’t actually support hardware acceleration of H264 streams?

I don’t really want to drop the main stream’s specs, as I’m also recording from it. Unfortunately I can’t boost the substream’s resolution (no option to do so) – anything else I can do?

scstraus · July 26, 2020, 9:48pm

Yes I use 4 1080p 4fps streams on a synology ds918+ and am very happy with it.

ebendl · July 26, 2020, 10:07pm

And I assume your CPU usage is fine?

Is that a primary / secondary stream? 4fps sounds perfect for Frigate, but quite low for general recording.

walt · July 26, 2020, 10:08pm

5 1080P 15FPS streams on a i7. Takes about 2 cores usually.

scstraus · July 26, 2020, 10:34pm

Yes, frigate uses less than 50% of total capacity. Yes, it’s the primary stream. I don’t personally see the need for more than 4fps. I’m not trying to make cinema quality recordings, I just want to see who is in my yard and what they are doing (and frigate is the most important part of that system). 4fps is plenty for seeing who they are and what they are doing and saves me loads of disk space which allows me to keep recordings longer. These are all much more important to me than high frame rate security footage.

martinschm · July 27, 2020, 10:16am

Hi,
i Came across this thread looking for a object detecion mechanism for RSTP cams. I am using Proxmox on an Intel Nuc and I am using iobroker as a smart home system. Reading that Frigate also supports MQTT for notifications I got interested in your project.

Is there a way of installing it without using Docker?
Already using proxmox, I would like to avoid using Docker on top of it.

Best
Martin

walt · July 27, 2020, 3:35pm

Probably best to just run with Docker, as you’ll save yourself a lot of time and headache. But if you want you can try to look through the Dockerfile and recreate the system on bare metal:

github.com

blakeblackshear/frigate/blob/53ccc903daac3993b9431e39c1b985381011ad1c/Dockerfile

FROM ubuntu:18.04
LABEL maintainer "[email protected]"

ENV DEBIAN_FRONTEND=noninteractive
# Install packages for apt repo
RUN apt -qq update && apt -qq install --no-install-recommends -y \
    software-properties-common \
    # apt-transport-https ca-certificates \
    build-essential \
    gnupg wget unzip tzdata \
    # libcap-dev \
    && add-apt-repository ppa:deadsnakes/ppa -y \
    && apt -qq install --no-install-recommends -y \
        python3.7 \
        python3.7-dev \
        python3-pip \
        ffmpeg \
        # VAAPI drivers for Intel hardware accel
        libva-drm2 libva2 i965-va-driver vainfo \
    && python3.7 -m pip install -U wheel setuptools \

This file has been truncated. show original

scstraus · July 27, 2020, 3:37pm

It would be easier and more productive just to get rid of Proxmox ;-). Also worth mentioning that VM’s add overhead but Docker does not, so if you want to optimize, get rid of the VM’s.

rpress · July 27, 2020, 4:05pm

I run without Docker on Debian, it is easy enough. I just looked in the Dockerfile as mentioned.

One warning, you need Python 3.7 and Debian sid does not include this.

martinschm · July 27, 2020, 8:06pm

Hi @all,
thanks for your quick feedback. I am quite happy with Proxmox and I am running several applications in seperate containers. Not too keen on setting this stuff up from scratch. Will have a look at the manual installation.

Thanks
Martin

brettyj · July 27, 2020, 8:21pm

Is there anyway to run some video through the frigate detection to see what it saw? The reason I ask is I’m getting some false negatives. Our milkman delivers milk in the dark, and moves fast, and it seems as though frigate doesn’t have much time to register him - despite the threshold being quite low at 80% - it would be great if I could run the video through after the fact to see what percentage it got, if any, if the camera angle wasn’t working great for it, etc.

ebendl · July 27, 2020, 10:58pm

Thanks makes sense all you’ve said.

I’m leaning towards 7 cameras though, so might need to run some of them at lower resolutions.

ItsMee · July 28, 2020, 12:43am

Wow, really cool stuff - frigate was up and running in no time and the initial debug results exceeded what I had expected from the hardware.

Three quick questions to the experts:

Do i need to enable the Coral Edge TPU?
A tagged object rectangle live stream was available once (multiple fps) - but before and after the empty stream only? Why could that be?
http://serverip:5000/camera?fps=2&h=720
CPU utilizatin on my (weak) j4105 seems high. Is that expected, or are there ways to reduce load, e.g. I’m not sure about ffmpeg HW acceleration

Debug stats

{
  "coral": {
    "detection_start": 1595896050.200327,
    "fps": 3.7,
    "inference_speed": 11.79,
    "pid": 23
  },
  "camera1": {
    "camera_fps": 8.3,
    "detection_fps": 2.6,
    "ffmpeg_pid": 34,
    "frame_info": {
      "detect": 1595896050.142969,
      "process": 1595896050.142969,
      "read": 1595896050.142969
    },
    "pid": 40,
    "process_fps": 8.4,
    "read_start": 1595896050.15747,
    "skipped_fps": 0
  },
  "plasma_store_rc": null,
  "camera2": {
    "camera_fps": 12,
    "detection_fps": 1.1,
    "ffmpeg_pid": 38,
    "frame_info": {
      "detect": 1595896050.180223,
      "process": 1595896050.127551,
      "read": 1595896050.180223
    },
    "pid": 41,
    "process_fps": 12,
    "read_start": 0,
    "skipped_fps": 0
  }
}

cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 122
model name      : Intel(R) Celeron(R) J4105 CPU @ 1.50GHz
stepping        : 1
microcode       : 0x32
cpu MHz         : 2372.699
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 24
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault cat_l2 pti cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust smep erms mpx rdt_a rdseed smap clflushopt intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts umip rdpid md_clear arch_capabilities
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 2995.20
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

[..]

htop

  1  [||||||||||||||||||||||||||||||||||||||||||||||||||||                                       53.9%]   Tasks: 181, 571 thr; 1 running
  2  [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||                     72.3%]   Load average: 2.92 3.33 4.62
  3  [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||                                  58.6%]   Uptime: 35 days, 16:10:06
  4  [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||                                60.4%]
  Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||            3.33G/15.3G]
  Swp[||||                                                                                  572M/15.7G]

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
32139 root       20   0 3199M  383M  218M S 98.5  2.4 11:32.33 python3.7 -u detect_objects.py
32413 root       20   0  632M  110M 31504 S 56.4  0.7  6:40.51 ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrup
32416 root       20   0 1740M  163M 96796 S 26.6  1.0  3:03.97 python3.7 -u detect_objects.py
12911 root       20   0  316M  203M 12876 S 22.7  1.3 16:48.43 python3 -m homeassistant --config /config
 3988 root       20   0 3199M  383M  218M S 18.8  2.4  1:23.27 python3.7 -u detect_objects.py
 3987 root       20   0 3199M  383M  218M S 16.2  2.4  1:14.85 python3.7 -u detect_objects.py
32414 root       20   0 3199M  383M  218M S 15.6  2.4  1:55.48 python3.7 -u detect_objects.py
32247 root       20   0 1071M  101M 27984 S 10.4  0.6  0:27.59 python3.7 -u detect_objects.py
32321 root       20   0  579M 59012 31124 S  9.1  0.4  0:56.45 ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrup
 6076 root       20   0 3199M  383M  218M S  8.4  2.4  0:11.24 python3.7 -u detect_objects.py
32427 root       20   0 3199M  383M  218M S  5.8  2.4  0:40.56 python3.7 -u detect_objects.py
32420 root       20   0 1740M  163M 96796 S  5.8  1.0  0:34.79 python3.7 -u detect_objects.py
32444 root       20   0 3199M  383M  218M S  5.8  2.4  0:35.61 python3.7 -u detect_objects.py
32415 root       20   0 1732M  127M 67940 S  5.8  0.8  0:22.12 python3.7 -u detect_objects.py
32419 root       20   0 1740M  163M 96796 S  5.2  1.0  0:38.49 python3.7 -u detect_objects.py
32428 root       20   0 3199M  383M  218M S  5.2  2.4  0:35.31 python3.7 -u detect_objects.py
32442 root       20   0 3199M  383M  218M S  5.2  2.4  0:36.04 python3.7 -u detect_objects.py
32429 root       20   0 3199M  383M  218M S  4.5  2.4  0:30.19 python3.7 -u detect_objects.py
32443 root       20   0 3199M  383M  218M S  4.5  2.4  0:35.64 python3.7 -u detect_objects.py
32445 root       20   0 3199M  383M  218M S  3.9  2.4  0:36.06 python3.7 -u detect_objects.py
32421 root       20   0 1740M  163M 96796 S  3.9  1.0  0:30.54 python3.7 -u detect_objects.py
32322 root       20   0 3199M  383M  218M S  3.9  2.4  0:23.27 python3.7 -u detect_objects.py

Config

web_port: 5000
mqtt:
  [..]
objects:
  track:
    - person
    - car
    - truck
    - cat
  filters:
    person:
      min_area: 5000
      max_area: 100000
      threshold: 0.5
cameras:
  camera1:
    ffmpeg:
      input: rtsp://frigate:password@camera1/videoMain
    take_frame: 1
    snapshots:
      show_timestamp: True
    objects:
      track:
        - person
        - cat
      filters:
        person:
          min_area: 5000
          max_area: 100000
          threshold: 0.5
  camera2:
    ffmpeg:
      input: rtsp://frigate:password@camera2:554/h264Preview_01_main
    take_frame: 1
    snapshots:
      show_timestamp: True
    objects:
      track:
        - person
        - car
        - cat
      filters:
        person:
          min_area: 5000
          max_area: 100000
          threshold: 0.5

blakeblackshear · July 28, 2020, 11:59am

I do this for testing all the time. Frigate can run detection on any source ffmpeg is capable of reading. You just need to update the ffmpeg input params and use the mp4 as the input:

ffmpeg:
  input: /lab/debug/input/garage-20200726153044.mp4
  input_args:
    - -stream_loop
    - "-1"
    - -re

That will loop over the same mp4 again and again.

blakeblackshear · July 28, 2020, 12:06pm

Your inference times indicate that you are using the Coral already. You should be able to enable hwaccel with this config. If you want to reduce CPU utilization further. Use a lower FPS stream and/or a lower resolution. I would suggest 5fps and the lowest resolution that still detects objects where it matters.

calypso · July 29, 2020, 12:43pm

Is there anyway to bump up the motion sensitivity? Since moving from 0.4 to 0.5 my the number of detections has dropped off considerably, mostly due to detection not being detected (tried high and low res streams)

germplan · July 31, 2020, 1:52am

Getting some quite odd behavior.

RTSP stream from Unifi camera at Medium (1024x576). Any ideas?

Config.yml

web_port: 5000

mqtt:
  host: [ip]
  topic_prefix: frigate
  
ffmpeg:
  global_args:
    - -hide_banner
    - -loglevel
    - panic
  hwaccel_args: []
  input_args:
    - -avoid_negative_ts
    - make_zero 
    - -fflags
    - nobuffer
    - -flags
    - low_delay
    - -strict
    - experimental
    - -fflags
    - +genpts+discardcorrupt
    - -vsync
    - drop
    - -rtsp_transport
    - tcp
    - -stimeout
    - '5000000' 
    - -use_wallclock_as_timestamps
    - '1'
  output_args:
    - -vf
    - mpdecimate
    - -f
    - rawvideo
    - -pix_fmt
    - rgb24

objects:
  track:
    - person

  filters:
    person:
      threshold: 0.5

cameras:
  porch:
    ffmpeg:
      input: rtsp://IP/SUFFIX
      height: 1024
      width: 576
      take_frame: 1
      snapshots:
        show_timestamp: True
      objects:
        track:
          - person
        filters:
          person:
            min_area: 5000
            max_area: 100000
            threshold: 0.5

/Debug/Stats

{"coral":{"detection_start":0.0,"fps":0.0,"inference_speed":10.0,"pid":35},"plasma_store_rc":null,"porch":{"camera_fps":3.0,"detection_fps":0.0,"ffmpeg_pid":36,"frame_info":{"detect":1596159937.045059,"process":1596159937.045059,"read":1596159937.177376},"pid":39,"process_fps":1.2,"read_start":1596159937.04957,"skipped_fps":2.0}}

Log

On connect called
Creating ffmpeg process...
ffmpeg -hide_banner -loglevel panic -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -vsync drop -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://IP/suffix -vf mpdecimate -f rawvideo -pix_fmt rgb24 pipe:
Starting detection process: 35
Camera_process started for porch: 39
Starting process for porch: 39
No EdgeTPU detected. Falling back to CPU.
* Serving Flask app "detect_objects" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.

Use a production WSGI server instead.
* Debug mode: off