Local realtime person detection for RTSP cameras

Ok, got it now in an LXC container with docker and TPU (usb coral) passthrough… will monitor the behaviour…

*** Fingers Crossed ***

Edit: OMG, memory & CPU is almost nothing :slight_smile:

1 Like

** Fingers Crossed ** :grinning_face_with_smiling_eyes:

Oh my, I am just increasing all my 4 cams back to full res… This is amazing… (resource wise)!

1 Like

say what ?! person 80%? help? :-X

3 Likes

One option, create zones, specify the min_area and max_area for objects - i.e a person couldn’t be 9252 in that zone/that close to the camera.

1 Like

Got it, thank you will go through the documentation to set it up, Much appreciated !

Must object be fully in zone to be considered as “in the zone”?

Makes complete sense. I do watch a grid view but I am also someone that is not using frigate for NVR duty. It’s purely notifications. The solution for those that do what a grid view may be doing what I am doing. I have frigate pulling video feeds from my NVR and integrating the notifications to home assistant. If I want the grid view I just load up my NVR and have access to them.

Ok, major problems again… on the proxmox side now. 32GB was full in a day… tried to grow to 64GB then docker in LXC didn’t start… All kinds of errors… made me decide to start over… but now I cant get it running anymore… not with a new LXC id as well…

Can’t sart it:

106.conf:

arch: amd64
cores: 4
features: nesting=1
hostname: frigatelxc
memory: 8192
mp0: /shared,mp=/shared,size=0
nameserver: 192.168.1.90 192.168.1.91
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=B2:72:05:93:9E:23,ip=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-106-disk-0,size=128G
swap: 512
lxc.cgroup.devices.allow: c 226:0 rwm
lxc.cgroup.devices.allow: c 226:128 rwm
lxc.cgroup.devices.allow: c 29:0 rwm
lxc.cgroup.devices.allow: c 189:* rwm
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0, 0
lxc.mount.entry: /dev/bus/usb/002/ dev/bus/usb/002/ none bind,optional,create=dir 0, 0
lxc.cap.drop:

image

help :-)?

when I grey out
mp0: /shared,mp=/shared,size=0

it starts… whats that for?

EDIT: I am back up and running… next/new quiestions will come :slight_smile:

Running Frigate on Home Assistant as addon, NUC with Ubuntu 20.04.

Could someone please share a working automation where I send image and video to Telegram? The example in section Home Assistant do not work and generate
Error sending photo: Can't parse entities: can't find end of the entity starting at byte offset 28. Args: (), kwargs: {'chat_id': xxxxxxxxx, 'photo': <_io.BytesIO object at 0x7f0ae8a29b30>, 'caption': 'A person was detected on hik_kontor camera', 'disable_notification': False, 'reply_markup': None, 'timeout': None, 'parse_mode': 'Markdown'}

Current automation:

automation:
  - alias: Notify of events
    trigger:
      platform: mqtt
      topic: frigate/events
    action:
      - service: notify.xxxxxxxxx
        data:
          message: 'A {{trigger.payload_json["after"]["label"]}} was detected.'
          data:
            photo:
              # this url should work for addon users
              - url: 'http://ccab4aaf-frigate:5000/api/events/{{trigger.payload_json["after"]["id"]}}/thumbnail.jpg'
                caption: 'A {{trigger.payload_json["after"]["label"]}} was detected on {{ trigger.payload_json["after"]["camera"] }} camera'

The issue are the caption which gives a markdown error. If I replace the templates and use static text, it works for sending an image via Telegram.

It’s actually based on the bottom center of the bounding box. Not any part.

is it possible to show bounding boxes when playback clips?

2 Likes

If I run frigate separately and if I connect the (hacs installed) integration to that frigate.
I see all “entities”, but I have loads of questions:

  1. I do see cam images but no video is played when pressing. it is, but takes a long time… Is this due to the fact I have not enable RTMP for the cams? I did enable see config below for 1 cam. also tcp 1935 is forwarded from container to host.
  2. I see loads of entities for a camera but only 2 give output:

    How can it be the others are unavailable?
  3. I have a lot of “empty” camera.xxx entities showing up as unavailable.

What can be the issue?

  terras:
    ffmpeg:
      inputs:
        - path: rtsp://admin:[email protected]:554/Streaming/Channels/101/
          roles:
            - detect
            - clips
            - rtmp
    width: 3840
    height: 2160
    fps: 4
    snapshots:
      enabled: true
      crop: false
    objects:
      track:
        - bird
        - dog
        - cat
    motion:
      mask:
        - '3840,0,3840,1295,3138,1412,3034,769,2958,382,2622,0'
        - '0,652,178,384,444,137,598,303,416,496,104,819,25,777'
        - '0,161,64,242,180,248,248,264,354,178,455,92,532,0,0,0'
      threshold: 25
    clips:
      # Required: enables clips for the camera (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: Number of seconds before the event to include in the clips (default: shown below)
      pre_capture: 2
      # Optional: Number of seconds after the event to include in the clips (default: shown below)
      post_capture: 2
      # Optional: Objects to save clips for. (default: all tracked objects)
      objects:
        - bird
        - dog
        - cat
      # Optional: Restrict clips to objects that entered any of the listed zones (default: no required zones)
      required_zones: []
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 10
        # Optional: Per object retention days
        objects:
          bird: 200
          dog: 3
          cat: 200

Edit:
the entities do work on another instance of home assistant with the addon running…
I can acces the rtmp stream via vlc in the frigate instance:
rtmp://192.168.1.112/live/terras

Edit: the issue was wrongly configured mqtt server. It needs to be the one on the same homeassistant.

I get multiple events/notifications when a car is parked in frame. Is there some way to have it not be triggered multiple times until it leaves frame? or another object comes in to frame?

Or sometimes a person stays in frame but is moving around and it triggers multiple events. Sometimes the times of the events overlap ie. event 1 ~ 1:30:29-1:30:50 ; event 2 ~ 1:30:40-1:30:55

1 Like

Hi @mate

EDIT: Seems both photo and video now works fine on Telegram :slight_smile:
Though, still can’t use templates in caption.

Here’s my automation (HA on Ubuntu, running Frigate as addon):

- alias: 32 - Frigate
  description: Person/car detection in cameras from Frigate
  trigger:
    - platform: mqtt
      topic: frigate/events
  condition:
    - '{{ trigger.payload_json["type"] == "end" }}'
    - '{{ trigger.payload_json["after"]["label"] == "person" }}'
  action:
    - service: notify.my_telegram
      data:
        message: ''
        data:
          photo:
            - url: 'http://192.168.0.10:5000/api/events/{{trigger.payload_json["after"]["id"]}}/snapshot.jpg'
              caption: 'A snapshot'
    - delay:
        hours: 0
        minutes: 0
        seconds: 45
        milliseconds: 0
    - service: notify.my_telegram
      data:
        message: ''
        data:
          video:
            - url: 'https://XYZ.ui.nabu.casa/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}/{{trigger.payload_json["after"]["camera"]}}/clip.mp4'
              timeout: 1000
              caption: 'A video clip'
  mode: single
2 Likes

Here are my kinda hackey work arounds to multiple notifications. Basically, the automation needs 5min without an object detected before it will send a notification. In theory, for a car sitting in the driveway you would get one notification on first detection and then no more until there was no car for 5min.

It wouldn’t tell you if another car showed up. Not sure if frigate reports the number of detected objects or not. If it did I guess it’d be possible to add that logic. There might be more sophisticated ways to do this by event type. Instead of timer maybe just ignore any events that are “type : update”. Its possible that’d detect a new car in the scene? not sure…

alias: Frigate Car Notification
description: ''
trigger:
  - platform: mqtt
    topic: frigate/events
condition:
  - condition: state
    entity_id: binary_sensor.front_car_motion
    state: 'off'
    for: '00:05:00'
  - condition: and
    conditions:
      - condition: template
        value_template: '{{trigger.payload_json["after"]["label"] == "car"}}'
action:
  - service: notify.mobile_app_XXXXX_iphone
    data:
      message: 'A {{trigger.payload_json["after"]["label"]}} was detected.'
      data:
        attachment:
          url: >-
            https://XXXXXXXXXXX.ui.nabu.casa/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}/thumbnail.jpg?format=android
          hide-thumbnail: false
        apns_headers:
          apns-collapse-id: '{{trigger.payload_json["after"]["id"]}}'
mode: single
alias: Frigate Person Notification
description: ''
trigger:
  - platform: mqtt
    topic: frigate/events
condition:
  - condition: state
    entity_id: binary_sensor.front_person_motion
    state: 'off'
    for: '00:05:00'
  - condition: and
    conditions:
      - condition: template
        value_template: '{{trigger.payload_json["after"]["label"] == "person"}}'
action:
  - service: notify.mobile_app_XXXXXX_iphone
    data:
      message: 'A {{trigger.payload_json["after"]["label"]}} was detected.'
      data:
        attachment:
          url: >-
            https://XXXXXXXXXXXXXXX.ui.nabu.casa/api/frigate/notifications/{{trigger.payload_json["after"]["id"]}}/thumbnail.jpg?format=android
          hide-thumbnail: false
        apns_headers:
          apns-collapse-id: '{{trigger.payload_json["after"]["id"]}}'
mode: single

1 Like

@blakeblackshear is there a way to have the 24/7 recording be a toggle so that I can have it active at specified times? Basically I want the cameras to only save clips on person detection but when my babysitter is in the house I want there to be 24/7 recording.

Looks like maybe you can set that with a MQTT message?

frigate/<camera_name>/clips/set #

Topic to turn clips for a camera on and off. Expected values are ON and OFF.

Adding another. If I add - rtmp to the cams I get:

frigate.video                  INFO    : terras: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : terras: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : terras: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : voortuin: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : voortuin: ffmpeg process is not running. exiting capture thread...
frigate.events                 INFO    : Exiting event processor...
frigate.video                  INFO    : parkeerplaats: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : parkeerplaats: ffmpeg process is not running. exiting capture thread...
frigate.video                  INFO    : schuurachter: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : schuurachter: ffmpeg process is not running. exiting capture thread...
frigate.watchdog               INFO    : Exiting watchdog...

I’m not 100% that it will fit as I bought a 180 degree usb-c cable from AliExpress but haven’t tried it as I realised the Pi4 doesn’t supply enough power for the coral and the Argon M.2 case. Looking at the pic I took at the time it might be slightly too big.


I removed the HDMI board, de-soldered the IR sensor and jumpers and removed the Coral plastic housing to fit it in. The cable came after and looks a bit too big:

I will give it a go out of curiosity at some point.

1 Like