Local realtime person detection for RTSP cameras

You’re welcome to post pictures of you car, garage (with the bounding box), etc on a public forum and I’ll suggest where I’d mask - but I’m not going to be posting mine :wink:

Just a random google image:

this is where I would not like to detect cars but do want to have people, dogs, etc if there are no cars:

Not useful - to mask, you need to know where it is being detected. And that is the bounding box. Mask the bottom centre of wherever it is being detected, to prevent it being detected, without masking the entire area.

If red is the bounding box, green is where you could mask to prevent it being detected - because it covers the bottom centre/middle of the box.

Pretty sure @tmjpugh already answered that in post 4692:

Where ‘mask: 0,0,1000,0,1000,200,0,20’ is your black area - that would prevent cars only being detected in that area.

objects:
  track:
    - person
    - car
    - dog
    - cat
    - bicycle
  filters:
    car:
      min_area: 5000
      max_area: 100000
      min_score: 0.5
      threshold: 0.7
      mask: 0,0,1000,0,1000,200,0,200
1 Like

Anyone using frigate with nginx proxy?
how you do this? (config example)

So I did this and Frigate restarted ok, will let you know if that works (not listing cars anymore on the parking.

  schuurachter:
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]:554/Streaming/Channels/101/
          roles:
            - detect
            - clips
#            - rtmp
    width: 2048
    height: 1536
    fps: 4
    snapshots:
      enabled: true
      crop: false
      bounding_box: True
    objects:
      track:
        - person
        - car
        - dog
        - bicycle
        - cat
      filters:
        car:
          min_area: 5000
          max_area: 100000
          min_score: 0.5
          threshold: 0.7
          mask:
            - '443,106,775,108,679,365,294,371'
            - '2048,371,2048,585,1136,401,1090,130'
    clips:
      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 2
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 2
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - person
        - car
        - dog
        - bicycle
        - cat
      # Optional: Restrict clips to objects that entered any of the listed zones (default: no required zones)
      required_zones: []
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 10
        # Optional: Per object retention days
        objects:
          person: 100
          car: 50
          dog: 50
          cat: 10
          bicycle: 50

what are these for:

I cannot find an explanation in the documentation. only this https://blakeblackshear.github.io/frigate/configuration/false_positives/ but that does not explain met the meaning and values of min_area and max_area.

1 Like

I highly recommend spending some time reading the documentation start to finish, multiple times, including the commented/hashed lines in the example configuration - just reading, without trying to configure anything, or without skipping back and forth.

min_area and max_area are the area sizes, they’re shown on the bounding box on the snapshots if enabled.

Objects, bottom of the above:

  # Optional: minimum width*height of the bounding box for the detected object (default: 0)
  # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)

… over and out from me, I need a drink :wink:

P.S. You might actually need to at least remove the min_area and/or max_area, depending on the size of the cars being detecting - the numbers on the bounding box snapshots, if enabled, will tell you what the sizes are.

You need to look at a few of the false positives, to work out what the range of sizes are, for example, my car as it enters through the gates is around 10000 (so this is min_area, roughly, if i want it to be detected at that distance)… as it enters the garage and comes closer to the camera it is around 40000, but when parked and as close to the camera as it is going to be it is around 100000.

Naturally the above varies depending on the size of the car, how close the car gets to the camera and/or how far away it can reliably be detected.

Can’t I then better just leave these out?

To start with, probably, yes.

But tuning the min_area and max_areas, for each object, each camera, even separate zones, are a key part of enhancing reliable detection and/or reducing false positives (i.e a person can’t physically be 250000 on some cameras, or zones, so don’t let it think it can be).

But to start with, yes you can probably leave them out… because your need to know the sizes, at the various distances, to configure them properly.

After HA restart it’s working now for telegram & hass app
Seems like reloading the automation wasn’t enough
I thought it was odd when it was still not working even after using nobu casa link
Anyway, it’s just with hass ios notification I’m only getting photos (on hass android I’m getting both video & photos on android)
Does anyone have example on getting video notification with ios hass app?
Here’s the automation config btw

  - alias: Notify of events
    trigger:
      platform: mqtt
      topic: frigate/events
    action:
      - service: notify.telegram
        data_template:
          message: 'A {{trigger.payload_json["after"]["label"]}} was detected.'
          data:
            photo:
              # this url should work for addon users
              - url: 'http://ccab4aaf-frigate:5000/api/events/{{trigger.payload_json["after"]["id"]}}/thumbnail.jpg'
                caption: 'A {{trigger.payload_json["after"]["label"]}} was detected on {{ trigger.payload_json["after"]["camera"] }} camera'
      - delay:
          hours: 0
          minutes: 0
          seconds: 53
          milliseconds: 0
      - service: notify.telegram
        data_template:
          message: Video of alert
          data:
            video:
              - url: 'https://rpixxxxx.duckdns.org:8143/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}/{{trigger.payload_json["after"]["camera"]}}/clip.mp4'
                timeout: 1000
                caption: 'A {{trigger.payload_json["after"]["label"]}} was detected on {{ trigger.payload_json["after"]["camera"] }} camera'
    mode: single 
#notify hass app
  - alias: Notify of events hass
    trigger:
      platform: mqtt
      topic: frigate/events
    action:
      - service: notify.mobile_app_iphone_keel
        data_template:
          message: 'A {{trigger.payload_json["after"]["label"]}} was detected.'
          data:
            image: 'https://rpixxxxx.duckdns.org:8143/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}/thumbnail.jpg?format=android'
            tag: '{{trigger.payload_json["after"]["id"]}}'
            clickAction: https://rpixxxx.duckdns.org:8143/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}/snapshot.jpg
            group: Test
            channel: frigate
            color: "#2DF56D"
            actions:
            - action: URI
              title: View Video
              uri: https://rpixxxxx.duckdns.org:8143/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}/{{trigger.payload_json["after"]["camera"]}}/clip.mp4
            - action: URI
              title: View Snapshot
              uri: https://rpixxxx.duckdns.org:8143/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}/snapshot.jpg

I have a Reolink RLC-420-5MP camera and have been trying to get Frigateto work with it for a couple of days now. Initially it seemed to be working fine on the 640x480 sub stream but it quickly became apparent that this was not detailed enough for decent detection so I’ve been attempting to move to the main stream instead but keep running into errors like this:

frigate.video                  INFO    : driveway: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : driveway: ffmpeg process is not running. exiting capture thread...
ffmpeg.driveway.detect         ERROR   : av_interleaved_write_frame(): Connection reset by peer
ffmpeg.driveway.detect         ERROR   : [flv @ 0xaaaae3041cb0] Failed to update header with correct duration.
ffmpeg.driveway.detect         ERROR   : [flv @ 0xaaaae3041cb0] Failed to update header with correct filesize.
ffmpeg.driveway.detect         ERROR   : Error writing trailer of rtmp://127.0.0.1/live/driveway: Connection reset by peer
frigate.video                  INFO    : driveway: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : driveway: ffmpeg process is not running. exiting capture thread...
ffmpeg.driveway.detect         ERROR   : av_interleaved_write_frame(): Connection reset by peer
ffmpeg.driveway.detect         ERROR   : [flv @ 0xaaab180047d0] Failed to update header with correct duration.
ffmpeg.driveway.detect         ERROR   : [flv @ 0xaaab180047d0] Failed to update header with correct filesize.
ffmpeg.driveway.detect         ERROR   : Error writing trailer of rtmp://127.0.0.1/live/driveway: Connection reset by peer

My configuration is:

mqtt:
  host: core-mosquitto
cameras:
  driveway:
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer
        - -flags
        - low_delay
        - -strict
        - experimental
        - -fflags
        - +genpts+discardcorrupt
        - -rw_timeout
        - '5000000'
        - -use_wallclock_as_timestamps
        - '1'
      inputs:
        - path: rtmp://user:[email protected]/bcs/channel0_main.bcs?token=sdasdasd&channel=0&stream=0&user= user&password=pass
          roles:
            - detect
            - rtmp
            - clips
    width: 2560
    height: 1920
    fps: 10
    objects:
      track:
        - person
        - car
        - cat
        - dog
    rtmp:
      enabled: True
    clips:
      enabled: True
      pre_capture: 5
      post_capture: 5
      retain:
        default: 10

I’m running this on a Home Assistant Blue with a Coral TPU.

Is anyone else successfully using a Reolink camera with Frigate? I’ve been so frustrated with all the little issues that this camera has that I ordered a Dahua IPC-T5442T-ZE last night to replace it as my main camera.

How’s the log output when you’re running the main camera stream with ffprobe?

I can’t figure out how to use ffprobe since my password has an exclamation mark in it and that doesn’t play well with the shell prompt.

Working to try and figure out how to overcome that as I haven’t been able to find a way to escape it.

EDIT: OK… I think I worked it out… Single quotes and double exclamation marks? Still issues though.

fffprobe -show_streams 'rtmp://user:[email protected]/bcs/channel0_sub.bcs?channel=0&stream=0&@192.168.1.143/bcs/channel0_sub.bcs?channel=0&stream=0&user=user&password=pass!!'
ffprobe version 4.4 Copyright (c) 2007-2021 the FFmpeg developers
  built with Apple clang version 12.0.0 (clang-1200.0.32.29)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.4_1 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
rtmp://user:[email protected]/bcs/channel0_sub.bcs?channel=0&stream=0&@192.168.1.143/bcs/channel0_sub.bcs?channel=0&stream=0&user=user&password=pass!!: Input/output error

I’m intentionally trying the sub stream first.

With all the errors it manages to get update every 10 seconds or so and display a few frames on the Frigate camera preview page. Certainly not 5fps as it dips whenever the rush of frames gets displayed.

EDIT: Just did a test and at 640x480 it does not detect peopler cars… There’s not enough detail at any distance.

I have been using the sub stream on a RLC-520 for detection. Using the same flags as you.

Might be worth checking the FPS of the substream on the camera, I think I had to adjust that initially.

Working fine on my iphone IOS with your code

I replace the android by ios on your code only, video can be play on my iphone

Do you mind sharing your modification?
I still can’t quite get it to work on my iphone

- alias: Notify of events detection camera
  trigger:
    platform: mqtt
    topic: frigate/events
  action:
    - service: notify.mobile_app_tonio
      data_template:
        message: '{{trigger.payload_json["after"]["label"]}} a été détecté.'
        data:
          image: 'https://xxx.duckdns.org/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}/thumbnail.jpg?format=ios'
          tag: '{{trigger.payload_json["after"]["id"]}}'
          clickAction: https://xxx.duckdns.org/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}/snapshot.jpg
          group: Test
          channel: frigate
          color: "#2DF56D"
          actions:
          - action: URI
            title: Voir la Vidéo
            uri: https://xxx.duckdns.org/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}/{{trigger.payload_json["after"]["camera"]}}/clip.mp4
          - action: URI
            title: Voir le Snapshot
            uri: https://xxx.duckdns.org/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}/snapshot.jpg
1 Like

Thanks, so the change is just on the image line, will test it out soon