Local realtime person detection for RTSP cameras

In what kind of environment did you set up authelia? On top of hassos?

I am running everything on Unraid. HA is running as HAOS in a VM. Frigate, Authelia and Nginx in docker containers.

I’m going to have to a look into this for my stack, I like the fact it works with k8s and has ldap support :slight_smile:

Ibracorp has a few great videos on youtube about setting up Authelia and nginx.

1 Like

Reverse proxy? I’m using the Full Access Frigate add-on. Am I missing a piece that is required?

Hi all,
after many months that frigate was working excellent with 3 ezviz ip cam on a U55 Beelink with coral TPU it’s since 2 weeks that during night vision it doesn’t record anymore everytime person and dog clips on movements
 During the day everything seems very good, but with night vision it misses many clips.

I cleaned the cams too


Any ideas? Suggestions?

Try debug view, go outside and look if it detect movement

I have too many false positives with trees, trash cans detected as Person. Any way to improve detection accuracy (especially at night time with IR lighting)?
I read about custom models, but couldn’t find any manual on how to train them & apply in Frigate.

Hi when i use your config i getting following error:

2 validation errors for FrigateConfig
cameras -> Eingang2 -> ffmpeg -> output_args -> clips
  extra fields not permitted (type=value_error.extra)
cameras -> Eingang2 -> clips
  extra fields not permitted (type=value_error.extra)

The config not accept the clips Argument
My config:

cameras:

  Eingang2:

    ffmpeg:

      inputs:

        - path: rtsp://admin:[email protected]:554/

          roles:

            - clips

            - detect

      global_args: -hide_banner -loglevel info

      output_args:

        record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy #removed -an parameter to enable audio

        clips: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy #removed -an parameter to enable audio

    clips:

      enabled: True

      pre_capture: 5

      post_capture: 5

    rtmp:

      enabled: True

Hi Chris,
that config was for old version, you need to adjust it for the new version, there were many breaking changes.

This is my latest config:

mqtt:
  host: REDACTED
  user: REDACTED
  password: REDACTED
  topic_prefix: frigate

#database:
  #path: /frigate.db

#https://blakeblackshear.github.io/frigate/configuration/advanced#logger
#logger:
  #default: debug #info

detectors:
  coral:
    type: edgetpu
    device: usb

detect:
  width: 640
  height: 360
  fps: 7

objects:
  track:
    - person
    - car
  filters:
    person:
      # Optional: minimum width*height of the bounding box for the detected object (default: 0)
      min_area: 8000
      # Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
      #max_area: 100000
      # Optional: minimum score for the object to initiate tracking (default: 0.5)
      #min_score: 0.5
      # Optional: minimum decimal percentage for tracked object's computed score to be considered a true positive (default: 0.7)
      #threshold: 0.7
    car:
      min_area: 5000

record:
  enabled: True
  retain_days: 0
  events:
    max_seconds: 300
    pre_capture: 10
    post_capture: 10
    retain:
      default: 14

snapshots:
  enabled: True
  timestamp: True
  bounding_box: True
  crop: True
  retain:
    default: 30

rtmp:
  enabled: True

birdseye:
  enabled: True
  width: 1024 #1280
  height: 576 #720
  quality: 8
  mode: objects #continuous #objects #motion
  
timestamp_style:
  format: "%m.%d.%Y %H:%M:%S"

ffmpeg:
  global_args: -hide_banner -loglevel warning #panic #info
  hwaccel_args:
    -hwaccel qsv -qsv_device /dev/dri/renderD128
  #input_args: #https://ffmpeg.org/ffmpeg-formats.html
    #-avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1
  output_args:
    #detect: -f rawvideo -pix_fmt yuv420p
    record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy #-an removed to hear sound
    #rtmp: -c copy -f flv

cameras:
  pred_domom: #REOLINK RLC-810A firmware IPC_523128M8MP.236_21040831.RLC-810A.IMX415.8MP.REOLINK
    ffmpeg:
      inputs:
        - path: rtsp://<UserName>:<Password>@<IpAddress>:554//h264Preview_01_sub
          roles:
            - detect
            - rtmp
        - path: rtsp://<UserName>:<Password>@<IpAddress>:554/
          roles:
            - record
    motion:
      mask:
        - 434,34,640,34,640,0,436,0
        - 640,271,640,0,224,0
    objects:
      mask:
        - 640,271,640,0,224,0
      filters:
        car:
          mask:
            - 0,360,332,360,202,132,0,128
        person:
          mask:
            - 158,360,640,360,640,0,64,0
    zones:
      PredDomom_PriPrednychOknach:
        coordinates: 0,360,80,360,57,115,0,121
3 Likes

Ok config now valid thx. But I getting the error ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures. I upgraded the software as sensed from support and changed the settings as u described.

well, there is no motion detected unless i don’t switch on my phone flashlight


this is the nightly vision:

image

my question is why until some weeks ago it worked? maybe summer time (?)

this is part of my config:


detectors:
  coral:
    type: edgetpu
    device: usb
[...]
objects:
  track:
    - person
    - dog
    - car
record:
  enabled: true
  retain_days: 0
  events:
    max_seconds: 300
    pre_capture: 5
    post_capture: 5
    retain:
      default: 7
      objects:
        person: 14

snapshots:
  enabled: true
  timestamp: False
  bounding_box: true
  crop: true
  height: 175
  required_zones: []
  retain:
    default: 7
    objects:
      person: 14

rtmp:
  enabled: True

live:
  height: 720
  quality: 8

cameras:
  Fontanella:
    ffmpeg:
      inputs:
        - path: rtsp://xxxxx@<myip>/h264_stream
          roles:
            - detect
            - rtmp
            - record
    motion:
      mask:
        - 1085,339,1280,542,1280,0,982,0,659,0,887,176
        - 235,710,355,710,355,686,235,686
    zones:
      balcony:
        coordinates: 0,1080,546,1080,433,828,358,628,329,610,249,334,207,338,0,380
      garder:
        coordinates: 260,321,258,0,1029,0,1322,211,1604,442,1920,748,1920,1080,591,1080

the camera is a EZVIZ C3WN 1080p.

Thanks!

Hi Chris,
I got them from ffmepg too but only after start. Then they doesn’t appear anymore and all works properly. I didn’t get a chance to paste them here if the community knows what to reconfigure or open an issue on github to get help from developers. But you can try :slight_smile:

After changing some motion parameters (by chance) I’m having motion detection, but not person detected
 :frowning:

  Fontanella:
    ffmpeg:
      inputs:
        - path: <PATH>
          roles:
            - detect
            - rtmp
            - record
    motion:
      threshold: 13
      contour_area: 50
      delta_alpha: 0.5
      frame_alpha: 0.2
      frame_height: 180
      mask:
        - 1085,339,1280,542,1280,0,982,0,659,0,887,176
        - 235,710,355,710,355,686,235,686

I still don’t know what you mean by reverse proxy, as I use no proxy locally.

In case anyone else is having the same issue. Turned out the cameras needed to be changed from Variable Bit Rate to Constant Bit Rate.

Just to reiterate, the issue: lovelace video feeds were not visible on Fire HD 10 v7 tablets using the Frigate integration and Frigate Lovelace Card. Feeds were visible on Apple devices. Must have been a change with HA Core 2021.11.

Hi Everyone,

Not sure if it’s the right place to open a new “thread” but I wanted to ask about “entered_zones”.
I tried using Zones to avoid alerts/events on stationary objects (cars) but nothing seems to solve this issue. I was thinking that “entered_sonzes” might be the solution (if it is in fact another parameter that could be defined). If I can send an alert only if the car enters my driveway from one zone and then pass through the second one, only then it would alert that there is a car there. is it possible? “required_zones” is an “OR” not “AND” as far as I know, so I get events no matter what :confused:

Can you help me, how to configure the Tapo c200 into frigate, and HA? Which components need to be configured? (ffmpeg, stream, frigate.yml, which lovlece card
) Thank you

hello, you install the frigate addon and make a frigate.yml configuration file.

mine looks like:

  host: 192.168.1.41
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: hass
  password: hass
  stats_interval: 60

detectors:
  coral:
    type: edgetpu
    device: pci

logger:
  default: info
  logs:
    frigate.event: debug

birdseye:
  enabled: True
  width: 1280
  height: 720
  quality: 8
  mode: continuous

ffmpeg:
  global_args: 
    - -hide_banner 
    - -loglevel 
    - warning
  # input_args:
  #   - -avoid_negative_ts
  #   - make_zero
  #   - -fflags
  #   - +genpts+discardcorrupt
  #   - -rtsp_transport
  #   - tcp
  #   - -stimeout
  #   - '5000000'
  #   - -use_wallclock_as_timestamps
  #   - '1'
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p

detect:
  width: 1280
  height: 720
  fps: 5
  enabled: False
  max_disappeared: 5

objects:
  track:
    - person

motion:
  threshold: 25
  contour_area: 100
  delta_alpha: 0.2
  frame_alpha: 0.2
  frame_height: 180

record:
  enabled: True
  retain_days: 3
  events:
    max_seconds: 300
    pre_capture: 5
    post_capture: 5
    objects:
      - person
    required_zones: []
    retain:
      default: 10
      objects:
        person: 15

snapshots:
  enabled: True
  timestamp: True
  bounding_box: True
  crop: True
  height: 175
  required_zones: []
  retain:
    default: 10
    objects:
      person: 15

rtmp:
  enabled: True
live:
  height: 720

  quality: 8


timestamp_style:
  position: "tl"
  format: "%d.%m.%Y %H:%M:%S"
  color:
    red: 255
    green: 255
    blue: 255
  thickness: 2

  effect: shadow

cameras:
  front_door:
    ffmpeg:
      inputs:
        - path: rtsp://hass:[email protected]:88/videoMain
          roles:
            - detect
            - rtmp
            - record
    best_image_timeout: 60

    mqtt:
      enabled: True
      timestamp: True
      bounding_box: True
      crop: True
      height: 270
      quality: 70
      required_zones: []

## ------------------------------------------------------------------------------------------------------------------------- ##

  garden_upstairs:
    ffmpeg:
      inputs:
        - path: rtsp://hassio:[email protected]:554/stream1
          roles:
            - detect
            - rtmp
            - record

    best_image_timeout: 60

    mqtt:
      enabled: True
      timestamp: True
      bounding_box: True
      crop: True
      height: 270
      quality: 70
      required_zones: []

## ------------------------------------------------------------------------------------------------------------------------- ##

  garage:
    ffmpeg:
      inputs:
        - path: rtsp://40.40.40.220:554/media/stream1
          roles:
            - detect
            - rtmp
            - record
    best_image_timeout: 60

    mqtt:
      enabled: True
      timestamp: True
      bounding_box: True
      crop: True
      height: 270
      quality: 70
      required_zones: []

## ------------------------------------------------------------------------------------------------------------------------- ##

  garden:
    ffmpeg:
      inputs:
        - path: rtsp://40.40.40.14:554/media/stream1
          roles:
            - detect
            - rtmp
            - record
    best_image_timeout: 60

    mqtt:
      enabled: True
      timestamp: True
      bounding_box: True
      crop: True
      height: 270
      quality: 70
      required_zones: []

My tapo cam is the garden cam configured. No wirries privacy interested peoples. My cams have NO access to internet trying to hack them
 good luck :stuck_out_tongue:

well then i have Dwain*s dashboard installed

Dwains Lovelace Dashboard | An auto generating Home Assistant Lovelace UI dashboard for desktop, tablet and mobile by Dwains for desktop, tablet, mobile (dwainscheeren.github.io)

and I created my more_page addon which lookslike:

## Created by Thorsten Fröhlich
## More-page add-on: Camera
## Version: 1.0.0
- type: custom:swipe-card
  card_width: "1160px"
  parameters:
    effect: "coverflow"
    grabCursor: true
    centeredSlides: true
    slidesPerView: "auto"
    coverflowEffect:
      rotate: 50
      stretch: 0
      depth: 100
      modifier: 1
      slideShadows: true
    pagination:
      type: "bullets"
  cards:
    # - type: custom:frigate-card
    #   menu:
    #     buttons:
    #       image: false
    #       frigate: false
    #       live: false
    #       clips: false
    #       snapshots: false
    #       download: false
    #       fullscreen: false
    #       frigate_ui: false
    #   live:
    #     provider: frigate-jsmpeg
    #     preload: true
    #   frigate:
    #     camera_name: birdseye
    #   view:
    #     default: live
    - type: custom:frigate-card
      camera_entity: camera.front_door
      live:
        provider: frigate
        preload: true
      elements:

        - type: state-badge
          entity: sensor.dark_sky_humidity
          style:
            right: '40px'
            top: 86px
            color: rgba(0,0,0,0)
            opacity: 0.7
        - type: state-badge
          entity: sensor.solar_powered_outdoor_siren_air_temperature
          style:
            right: '-20px'
            top: 110px
            color: rgba(0,0,0,0)
            opacity: 0.7
        - type: custom:frigate-card-menu-state-icon
          entity: lock.main_door
          tap_action:
            action: toggle
        - type: custom:frigate-card-menu-state-icon
          entity: script.restart_foscam
          tap_action:
            action: toggle
        - type: custom:frigate-card-menu-state-icon
          entity: script.preset_garage
          tap_action:
            action: toggle
        - type: custom:frigate-card-menu-state-icon
          entity: script.preset_street
          tap_action:
            action: toggle
        - type: custom:frigate-card-menu-state-icon
          entity: script.preset_maindoor
          tap_action:
            action: toggle
        - type: custom:frigate-card-conditional
          conditions:
            view:
              - live
          elements:
            # - type: custom:frigate-card-menu-icon
            #   icon: mdi:butterfly
            #   tap_action:
            #     action: custom:frigate-card-action
            #     frigate_card_action: image
            - type: icon
              icon: mdi:arrow-up
              style:
                background: rgba(10, 10, 10, 0.75)
                border-radius: 5px
                right: 150px
                bottom: 250px
                transform: scale(3, 3)
              tap_action:
                action: call-service
                service: script.turn_on
                service_data:
                  entity_id: script.turn_up
            - type: icon
              icon: mdi:arrow-down
              style:
                background: rgba(10, 10, 10, 0.75)
                border-radius: 5px
                right: 150px
                bottom: 125px
                transform: scale(3, 3)
              tap_action:
                action: call-service
                service: script.turn_on
                service_data:
                  entity_id: script.turn_down
            - type: icon
              icon: mdi:arrow-left
              style:
                background: rgba(10, 10, 10, 0.75)
                border-radius: 5px
                right: 225px
                bottom: 185px
                transform: scale(3, 3)
              tap_action:
                action: call-service
                service: script.turn_on
                service_data:
                  entity_id: script.turn_left
            - type: icon
              icon: mdi:arrow-right
              style:
                background: rgba(10, 10, 10, 0.75)
                border-radius: 5px
                right: 75px
                bottom: 185px
                transform: scale(3, 3)
              tap_action:
                action: call-service
                service: script.turn_on
                service_data:
                  entity_id: script.turn_right
            - type: icon
              icon: mdi:magnify-plus
              style:
                background: rgba(10, 10, 10, 0.75)
                border-radius: 5px
                right: 75px
                bottom: 450px
                transform: scale(3, 3)
              tap_action:
                action: call-service
                service: script.turn_on
                service_data:
                  entity_id: script.zoom_in
            - type: icon
              icon: mdi:magnify-minus
              style:
                background: rgba(10, 10, 10, 0.75)
                border-radius: 5px
                right: 75px
                bottom: 350px
                transform: scale(3, 3)
              tap_action:
                action: call-service
                service: script.turn_on
                service_data:
                  entity_id: script.zoom_out
      image:
        src: /local/incognito.jpg
      view:
        default: live
        timeout: '60'
      menu:
        buttons:
          image: true
        mode: overlay-bottom
        button_size: 80px
      event_viewer:
        autoplay_clip: true
        controls:
          next_previous:
            style: chevrons
            size: 80px
      dimensions:
        aspect_ratio_mode: dynamic


## END ##
    - type: custom:frigate-card
      camera_entity: camera.garden
      dimensions:
        aspect_ratio_mode: unconstrained
        aspect_ratio: "16:9"
      elements:
        - type: custom:frigate-card-menu-state-icon
          entity: light.porch_garden
          icon: mdi:track-light
          tap_action:
            action: toggle
      menu:
        buttons:
          download: true
          image: false
          frigate_ui: false
        mode: overlay-bottom
        button_size: 80px
      view:
        default: live
        timeout: "15"
      live:
        preload: true
        provider: frigate
        webrtc:
          entity: camera.front_door
      event_viewer:
        autoplay_clip: true
        controls:
          next_previous:
            style: thumbnails
            size: 80px
      frigate:
        url: 192.168.1.41
        camera_name: garden

    - type: custom:frigate-card
      camera_entity: camera.garden_upstairs_2
      dimensions:
        aspect_ratio_mode: unconstrained
        aspect_ratio: "16:9"
      view:
        default: live
        timeout: "15"
      live:
        preload: true
        provider: frigate
        webrtc:
          entity: camera.garden_upstairs_2
      event_viewer:
        autoplay_clip: true
        controls:
          next_previous:
            style: thumbnails
            size: 80px
      menu:
        mode: overlay-bottom
        button_size: 80px

    #       webrtc: {}

    - type: custom:frigate-card
      camera_entity: camera.garage
      dimensions:
        aspect_ratio_mode: unconstrained
        aspect_ratio: "16:9"
      elements:
        - type: custom:frigate-card-menu-state-icon
          entity: light.garage
          tap_action:
            action: toggle
      view:
        default: live
        timeout: "15"
      live:
        preload: true
        provider: frigate
        webrtc:
          entity: camera.garage
      event_viewer:
        autoplay_clip: true
        controls:
          next_previous:
            style: thumbnails
            size: 80px
      menu:
        mode: overlay-bottom
        button_size: 80px

    # - type: custom:frigate-card
    #   camera_entity: camera.front_door_24h
    # - type: custom:frigate-card
    #   camera_entity: camera.garden_24h
    # - type: custom:frigate-card
    #   camera_entity: camera.garage_24h
    # - type: custom:frigate-card
    #   camera_entity: camera.porch_24h

Well but therefor u need a lot of stuff. but it can look something like :slight_smile:

it is a swipe card with frigate lovelace card inside

well but in this ard is more than this :slight_smile: I would recommend to start installing and configuring frigate via addon.

Huhh quite complex configuration. Try to understand. Thank you

my Tapo use different address
rtsp://x.y.z.y:554/stream1

no media folder