Local realtime person detection for RTSP cameras

@SpikeyGG, although it was a bit of a journey (I had to watch a few videos), I ended up using the authentication with Authelia. There are quite a few steps with different components, however, I can see the Frigate UI and view clips, screenshots, events, etc.

Here is the guide I used from Ibracorp - https://youtu.be/kw_pohbKE3Y

1 Like

Good post @palito . I ended up having to use RTMP stream for my Reolink RLC-410-5MP. I do like how you have worked out another method to keep your model using RTSP and stable.

That’s really awesome, thanks for sharing it. It would be cool to wrap my whole system around some consistent authentication platform like that. I’ve saved the video so I can look into doing it down the road.

No worries. It took me a day or so watching the videos and then switching it over to live. Works well so far and gives an extra layer of protection… With NGINX you can not have to expose ports to the internet and also make things go through cloudflare.

My next step is to see if I can make HA go through it without any issues :wink:

Hi!

Just installed one Reolink to my summer cottage. My car has been parked to its location and Frigate gives me constantly events from my car.

image

I’d like to get events when car is arriving or leaving but not from the car which is parked. How to do this? If I put mask to that location then Frigate doesn’t nitice if somebody is walking from that area?

1 Like

Mask bottom half of where car normally park (tires to bottom windows)may reduce confidence to prevent capture

This should still allow person detect

Hi, can u share how did u make coral mount to docker in synology

Thx, @tmjpugh . However it would be really awesome that we could define zones where we could combine motion + object detection together.

Hi guys,

quick question, I have set some masks on the main feed (1080p) and then realized that I can use the sub-stream for the “detect”, but when I switched it, the mask I created earlier apply to this one and because of the lower resolution it covers way more that I want it to be (it lost all the possible motion area).
Is there a way to make this mask for the higher resolution? or maybe I should create the mask based on the lower resolution?

Thanks!
Didi

Hey Y’all,
finaly i setup today a perimeter security with Frigate.
If i turn on the perimeter and some person object apears around the house, it turn on the 12V siren with ESP01 tasmota relay :slight_smile:. I have 10s alarm with 1minutes pause delay. Works like a charm !!! @blakeblackshear LiveLongAndProsper ! <3

1 Like

Why the score differ in one event?

And clip was generated, but i can play only from nas(because of h265,changed it to h264)
So when i examen the clip, close to the dryer was my wife and trolley, she was riding the troley. And was infront of the dryer. That is strange why it saved the dryer around 10 seconds after she passed by with a babytroley…
I have issued a ticket on github

Ok I was able to catch it this time with a rolling upgrade watching thru MQTT Explorer. Frigate had available=online before the upgrade. The EMQX (Mosquitto broker) upgrade started, and I saw Frigate go available=offline, I see two messages about 2 mins apart (guessing it rolled to a new node, the that node rolled so it went to another node as it was then upgraded): frigate.mqtt INFO : MQTT connected

It stayed at available=offline until I redeployed the frigate pod. I’m going to see if it’s possible to reproduce with Mosquitto and if the broker bounces twice within a couple minutes if that’s the key or something with EQMX that Frigate doesn’t like on a reconnect.

Anyone know why below automation magically stop working after server restart?

Trace shows “fail” for template portion matching camera but automation should match camera in MQTT message.
I provide a sample message below that fails for this.

AUTOMATION

- id: '1621092684971'
  alias: NOTIFY_FrigateEntryGateMotionInOut
  description: ''
  trigger:
  - platform: mqtt
    topic: frigate/events
  condition:
  - condition: state
    entity_id: automation.gate_autoopenvisitormode_2
    state: 'off'
  - condition: or
    conditions:
    - condition: template
      value_template: '{{ trigger.payload_json["after"]["current_zones"] == ["outside"]
        }}'
    - condition: template
      value_template: '{{ trigger.payload_json["after"]["entered_zones"] == ["outside"]
        }}'
    - condition: template
      value_template: '{{ trigger.payload_json["after"]["current_zones"] == ["inside"]
        }}'
    - condition: template
      value_template: '{{ trigger.payload_json["after"]["entered_zones"] == ["inside"]
        }}'
  - condition: template
    value_template: '{{ trigger.payload_json["after"]["camera"] == ["Frigate_Gate02"]
      }}'
  action:
  - service: notify.mobile_app_iphone
    data:
      message: A {{trigger.payload_json["after"]["label"]}} is {{ trigger.payload_json["after"]["current_zones"]
        }} the Gate.{{trigger.payload_json["after"]["top_score"]}}
      data:
        tag: NOTIFY_EntryGateMotionInOut
        image: https://server.com/api/frigate/notifications/{{ trigger.payload_json["after"]["id"]
          }}/thumbnail.jpg?for=android
        actions:
        - action: OPEN_ENTRY_GATE
          title: Open Gate
          destructive: true
        - action: URI
          title: APP
          uri: /lovelace/3

MQTT MESSAGE

{
    "before": {
        "id": "1624898615.82943-xmyv4l",
        "camera": "Frigate_Gate02",
        "frame_time": 1624898615.82943,
        "snapshot_time": 0,
        "label": "car",
        "top_score": 0,
        "false_positive": true,
        "start_time": 1624898615.82943,
        "end_time": null,
        "score": 0.82421875,
        "box": [
            464,
            52,
            597,
            149
        ],
        "area": 12901,
        "region": [
            361,
            0,
            661,
            300
        ],
        "current_zones": [],
        "entered_zones": []
    },
    "after": {
        "id": "1624898615.82943-xmyv4l",
        "camera": "Frigate_Gate02",
        "frame_time": 1624898615.936224,
        "snapshot_time": 1624898615.936224,
        "label": "car",
        "top_score": 0.82421875,
        "false_positive": false,
        "start_time": 1624898615.82943,
        "end_time": null,
        "score": 0.83203125,
        "box": [
            467,
            49,
            599,
            149
        ],
        "area": 13200,
        "region": [
            380,
            0,
            680,
            300
        ],
        "current_zones": [
            "outside"
        ],
        "entered_zones": [
            "outside"
        ]
    },
    "type": "new"
}

EDIT
Figured it out after I posted it.
I forget NO bracket was needed around camera name

“camera”
NOT [“camera”]

1 Like

Has anyone successfully got a google coral to pass-through to a VMWare ESXI Host WITHOUT passing through a whole USB hub? I can pass through the USB Device, But it never initializes correctly (As others have noted).

1 Like

I am afraid this is the only solution for Esxi. Other people are using a separate usb3.0 hub with coral plugged in it and pass that one to VM instead.

I’am trying to figure out, how to properly use rtmp-streams restreamed by frigate.

I got the rtmp-stream working in vlc. So far so good. But where the hell should it appear in homeassistant? I got a camera-entity which only shows the low resolution stream used for detection.
Where do I get the high resolution stream?

mqtt:
  host: 192.168.xxx.xxx
  user: xxx
  password: xxx
cameras:
  hof:
    ffmpeg:
      hwaccel_args:
        - -c:v
        - h264_v4l2m2m
      inputs:
        - path: rtsp://user:[email protected]:554/h264Preview_01_main
          roles:
            - clips
            - rtmp
        - path: rtsp://user:[email protected]:554/h264Preview_01_sub
          roles:
            - detect
            - record
    width: 640
    height: 352
    fps: 5
    detect:
      max_disappeared: 100
    objects:
        track:
          - person
          - car
          - dog
          - bicycle
          - cat
          - bird
    motion:
      mask:
        - 304,0,253,187,201,207,163,352,0,352,0,0
        - 640,352,525,352,586,0,640,0
      threshold: 50
      contour_area: 150
    zones:  
      strasse:
        coordinates: 312,82,576,88,585,0,332,0  
      einfahrt:
        coordinates: 171,352,524,352,595,90,270,90
    snapshots:
      enabled: true
      retain:
        default: 10
        objects:
          person: 15
    clips:
      enabled: true
      pre_capture: 5
      post_capture: 5
      retain: 
        default: 14
    record:
      enabled: true
      retain_days: 7

detectors:
  coral:
    type: edgetpu
    device: usb
{
  "cameras": {
    "hof": {
      "best_image_timeout": 60,
      "clips": {
        "enabled": true,
        "objects": null,
        "post_capture": 5,
        "pre_capture": 5,
        "required_zones": [],
        "retain": {
          "default": 14,
          "objects": {}
        }
      },
      "detect": {
        "enabled": true,
        "max_disappeared": 100
      },
      "ffmpeg_cmds": [
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -c:v h264_v4l2m2m -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://user:[email protected]:554/h264Preview_01_main -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/hof-%Y%m%d%H%M%S.mp4 -c copy -f flv rtmp://127.0.0.1/live/hof",
          "roles": [
            "clips",
            "rtmp"
          ]
        },
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -c:v h264_v4l2m2m -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://user:[email protected]:554/h264Preview_01_sub -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /media/frigate/recordings/hof-%Y%m%d%H%M%S.mp4 -r 25 -f rawvideo -pix_fmt yuv420p pipe:",
          "roles": [
            "detect",
            "record"
          ]
        }
      ],
      "fps": 25,
      "frame_shape": [
        352,
        640
      ],
      "height": 352,
      "motion": {
        "contour_area": 150,
        "delta_alpha": 0.2,
        "frame_alpha": 0.2,
        "frame_height": 58,
        "mask": [
          "304,0,253,187,201,207,163,352,0,352,0,0",
          "640,352,525,352,586,0,640,0"
        ],
        "threshold": 50
      },
      "mqtt": {
        "bounding_box": true,
        "crop": true,
        "enabled": true,
        "height": 270,
        "required_zones": [],
        "timestamp": true
      },
      "name": "hof",
      "objects": {
        "filters": {
          "bicycle": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "bird": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "car": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "cat": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "dog": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "person": {
            "mask": [],
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          }
        },
        "mask": null,
        "track": [
          "person",
          "car",
          "dog",
          "bicycle",
          "cat",
          "bird"
        ]
      },
      "record": {
        "enabled": true,
        "retain_days": 7
      },
      "rtmp": {
        "enabled": true
      },
      "snapshots": {
        "bounding_box": false,
        "crop": false,
        "enabled": true,
        "height": null,
        "required_zones": [],
        "retain": {
          "default": 10,
          "objects": {
            "person": 15
          }
        },
        "timestamp": false
      },
      "width": 640,
      "zones": {
        "einfahrt": {
          "coordinates": "171,352,524,352,595,90,270,90",
          "filters": {}
        },
        "strasse": {
          "coordinates": "312,82,576,88,585,0,332,0",
          "filters": {}
        }
      }
    }
  },
  "clips": {
    "max_seconds": 300,
    "retain": {
      "default": 10,
      "objects": {}
    },
    "tmpfs_cache_size": ""
  },
  "database": {
    "path": "/media/frigate/clips/frigate.db"
  },
  "detectors": {
    "coral": {
      "device": "usb",
      "num_threads": 3,
      "type": "edgetpu"
    }
  },
  "environment_vars": {},
  "logger": {
    "default": "INFO",
    "logs": {}
  },
  "model": {
    "height": 320,
    "width": 320
  },
  "mqtt": {
    "client_id": "frigate",
    "host": "192.168.178.7",
    "port": 1883,
    "stats_interval": 60,
    "topic_prefix": "frigate",
    "user": "mqtt"
  },
  "snapshots": {
    "retain": {
      "default": 10,
      "objects": {}
    }
  }
}

And also the fps dont show any effect. Wether i type 5 or 25. It is still 5.

Sounds like you dont have the stream component enabled: Stream - Home Assistant

1 Like

That’s it!

Now it is running fluid. But I got a delay of ~5-6sec.
I expected that it would be less. Without the stream component and 5fps the delay is ~1sec.

Is this delay caused by the same reasons as any other cameradelay on homeassistant?

I’ve got an issue with “No clip available”. It seems about or more than 50% of the time, when I go to watch a clip, it shows that the clip is not available. I look at the log of the docker container and nothing stands out to me:

2021-06-27T23:42:17.347177005Z  * Starting nginx nginx,
2021-06-27T23:42:17.361243571Z    ...done.,
2021-06-27T23:42:18.620084934Z Starting migrations,
2021-06-27T23:42:18.620110603Z peewee_migrate                 INFO    : Starting migrations,
2021-06-27T23:42:18.640004725Z There is nothing to migrate,
2021-06-27T23:42:18.640041304Z peewee_migrate                 INFO    : There is nothing to migrate,
2021-06-27T23:42:18.814084993Z frigate.mqtt                   INFO    : MQTT connected,
2021-06-27T23:42:18.814900238Z detector.coral                 INFO    : Starting detection process: 49,
2021-06-27T23:42:21.930222727Z frigate.app                    INFO    : Camera processor started for front_door: 52,
2021-06-27T23:42:21.930462867Z frigate.edgetpu                INFO    : Attempting to load TPU as usb,
2021-06-27T23:42:21.938782930Z frigate.app                    INFO    : Camera processor started for west_yard: 54,
2021-06-27T23:42:21.938983693Z frigate.edgetpu                INFO    : TPU found,
2021-06-27T23:42:21.939300153Z frigate.app                    INFO    : Camera processor started for east_yard: 56,
2021-06-27T23:42:21.939531851Z frigate.app                    INFO    : Camera processor started for back_yard: 58,
2021-06-27T23:42:21.939803050Z frigate.app                    INFO    : Camera processor started for out_the_back: 59,
2021-06-27T23:42:21.940036212Z frigate.app                    INFO    : Camera processor started for garage: 60,
2021-06-27T23:42:21.940313001Z frigate.app                    INFO    : Camera processor started for storage_room: 61,
2021-06-27T23:42:21.940530758Z frigate.app                    INFO    : Capture process started for front_door: 62,
2021-06-27T23:42:21.940778549Z frigate.app                    INFO    : Capture process started for west_yard: 65,
2021-06-27T23:42:21.940939384Z frigate.app                    INFO    : Capture process started for east_yard: 67,
2021-06-27T23:42:21.941176464Z frigate.app                    INFO    : Capture process started for back_yard: 74,
2021-06-27T23:42:21.941425016Z frigate.mqtt                   INFO    : Turning on detection for west_yard via mqtt,
2021-06-27T23:42:21.941611644Z frigate.app                    INFO    : Capture process started for out_the_back: 81,
2021-06-27T23:42:21.941820009Z frigate.mqtt                   INFO    : Turning on detection for east_yard via mqtt,
2021-06-27T23:42:21.942047264Z frigate.app                    INFO    : Capture process started for garage: 89,
2021-06-27T23:42:21.942251909Z frigate.app                    INFO    : Capture process started for storage_room: 97,
2021-06-27T23:42:21.942503200Z frigate.mqtt                   INFO    : Turning on detection for back_yard via mqtt,
2021-06-27T23:42:21.942702945Z frigate.mqtt                   INFO    : Turning on detection for out_the_back via mqtt,
2021-06-27T23:42:21.942937434Z frigate.mqtt                   INFO    : Turning on detection for storage_room via mqtt,
2021-06-29T13:20:39.324133182Z frigate.watchdog               INFO    : Detection appears to be stuck. Restarting detection process...,
2021-06-29T13:20:39.324368715Z root                           INFO    : Waiting for detection process to exit gracefully...,
2021-06-29T13:21:09.354216716Z root                           INFO    : Detection process didnt exit. Force killing...,
2021-06-29T13:21:09.387673191Z detector.coral                 INFO    : Starting detection process: 190149,
2021-06-29T13:21:12.506571651Z frigate.edgetpu                INFO    : Attempting to load TPU as usb,
2021-06-29T13:21:12.512086973Z frigate.edgetpu                INFO    : TPU found

Is there somewhere to look at more verbose logs inside the docker container? I also checked for the existence of clips mp4 files and they aren’t there either. It’s weird, when I restart the container, I get the clips but sometime after that they stop being created but detections still happen and home assistant alerts me with links using the notification blueprint. When I click the “see clip” link in the notification it’s a dead link. :frowning:

How can I troubleshoot this?