Local realtime person detection for RTSP cameras

The way I’m doing it is to take the mqtt camera for whatever object and send the camera snapshot after the sensor fires.
Using beta 3 and the home assistant integration to do this.
If anyone needs more info let me know and when I’m on a pc I can post more.

thanks jon
just solved this the way i was planning. could have finished this a lot more efficient if i would have realized that the file was only available after the “post_detection time + some writing to disk time”. Anyway, solved now. See below for the flow. maybe usefull to someone else too.

next problem to solve is that the video file isnt opening on my iphone. format issue? because in my telegram webapp it plays normal.
the mp4 it is the YUV 4.2.0 format. Can i change this while recording the event in something playable on my iphone?

[{"id":"ed123003.62d77","type":"tab","label":"Flow 2","disabled":false,"info":""},{"id":"6eaaeada.8cda3c","type":"mqtt in","z":"ed123003.62d77","name":"frigate/events","topic":"frigate/events","qos":"2","datatype":"auto","broker":"77a96594.e33e9c","x":270,"y":620,"wires":[["7358db7e.c2dfdc","abd868da.34c728"]]},{"id":"7358db7e.c2dfdc","type":"json","z":"ed123003.62d77","name":"test","property":"payload","action":"","pretty":false,"x":490,"y":620,"wires":[["9b33a525.55e91","74c0fabc.591064"]]},{"id":"9b33a525.55e91","type":"switch","z":"ed123003.62d77","name":"start switch","property":"payload.after.end_time","propertyType":"msg","rules":[{"t":"nnull"}],"checkall":"true","repair":false,"outputs":1,"x":650,"y":620,"wires":[["a81edade.31f6b8","b681eb88.5fb2f8","dbbe21dd.50b34"]]},{"id":"53875984.567dc8","type":"template","z":"ed123003.62d77","name":"","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"192.168.1.77:5000/clips/{{payload.after.camera}}-{{payload.after.id}}.mp4","output":"str","x":940,"y":620,"wires":[["20e67aa5.0a8a46","5b16fdb.020f904"]]},{"id":"20e67aa5.0a8a46","type":"debug","z":"ed123003.62d77","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1070,"y":700,"wires":[]},{"id":"c06d9cfc.564628","type":"debug","z":"ed123003.62d77","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":1130,"y":440,"wires":[]},{"id":"a81edade.31f6b8","type":"debug","z":"ed123003.62d77","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":830,"y":440,"wires":[]},{"id":"5b16fdb.020f904","type":"http request","z":"ed123003.62d77","name":"","method":"GET","ret":"bin","paytoqs":"ignore","url":"{{{payload}}}","tls":"","persist":false,"proxy":"","authType":"","x":1090,"y":620,"wires":[["b17648d6.250b98","c06d9cfc.564628"]]},{"id":"b17648d6.250b98","type":"function","z":"ed123003.62d77","name":"","func":"msg.payload = {\n    chatId: 11111111,\n    type: 'video',\n    caption : 'You must have a look at this!',\n    content: msg.payload,\n}\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":1300,"y":620,"wires":[["e76305e0.290538"]]},{"id":"e76305e0.290538","type":"telegram sender","z":"ed123003.62d77","name":"","bot":"8e57f19d.161d7","haserroroutput":false,"outputs":1,"x":1480,"y":620,"wires":[[]]},{"id":"b681eb88.5fb2f8","type":"debug","z":"ed123003.62d77","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":830,"y":760,"wires":[]},{"id":"abd868da.34c728","type":"debug","z":"ed123003.62d77","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":430,"y":720,"wires":[]},{"id":"48286e1e.ea221","type":"inject","z":"ed123003.62d77","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"before\": {\"id\": \"1609055550.60505-q70b6d\", \"camera\": \"bdc_frigate\", \"frame_time\": 1609055551.1853, \"label\": \"person\", \"top_score\": 0.66796875, \"false_positive\": false, \"start_time\": 1609055550.60505, \"end_time\": 1609061581.059547, \"score\": 0.76171875, \"box\": [1251, 687, 1414, 879], \"area\": 31296, \"region\": [1176, 633, 1476, 933], \"current_zones\": [], \"entered_zones\": [], \"thumbnail\": null}, \"after\": {\"id\": \"1609055550.60505-q70b6d\", \"camera\": \"bdc_frigate\", \"frame_time\": 1609055551.411159, \"label\": \"person\", \"top_score\": 0.66796875, \"false_positive\": false, \"start_time\": 1609055550.60505, \"end_time\": 1609061581.059547, \"score\": 0.640625, \"box\": [1245, 693, 1415, 900], \"area\": 35190, \"region\": [1180, 629, 1480, 929], \"current_zones\": [], \"entered_zones\": [], \"thumbnail\": null}}","payloadType":"str","x":280,"y":540,"wires":[["7358db7e.c2dfdc"]]},{"id":"74c0fabc.591064","type":"debug","z":"ed123003.62d77","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":630,"y":560,"wires":[]},{"id":"dbbe21dd.50b34","type":"delay","z":"ed123003.62d77","name":"","pauseType":"delay","timeout":"30","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":800,"y":620,"wires":[["53875984.567dc8"]]},{"id":"77a96594.e33e9c","type":"mqtt-broker","name":"Eclips Mosquito","broker":"192.168.1.77","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""},{"id":"8e57f19d.161d7","type":"telegram bot","botname":"Steenbank_Telegram","usernames":"","chatids”:”11111111”,”baseapiurl":"","updatemode":"polling","pollinterval":"300","usesocks":false,"sockshost":"","socksport":"6667","socksusername":"anonymous","sockspassword":"","bothost":"","botpath":"","localbotport":"8443","publicbotport":"8443","privatekey":"","certificate":"","useselfsignedcertificate":false,"sslterminated":false,"verboselogging":false}]

I forgot about the adjustments needed for RTMP camera’s.
Working now.

If you don’t mind pasteing this part of your automation I’d appreciate it!

Same problem here, I get the previous detection image for the MQTT cameras created by the custom component.

This is the automation I use. Should be pretty easy to edit to work for you with pushover.
Just keep in mind that this is using the .80 beta3 and the home assistant integration. Otherwise these cameras don’t exist to pull snapshots from.
If one knows where these snapshots are stored by mqtt it would be even simpler.
This is just my workaround.

alias: Frigate Person Frontyard
description: ''
trigger:
  - platform: state
    entity_id: sensor.frigate_frontyard_person
    to: '1'
condition: []
action:
  - service: camera.snapshot
    data:
      filename: /config/www/tmp/frigate_frontyard_person.jpg
    entity_id: camera.frigate_frontyard_person
  - data:
      data:
        photo:
          - caption: Person In Front
            file: /config/www/tmp/frigate_frontyard_person.jpg
      service: notify.telegram
mode: single

Hope that helps!

1 Like

If you are using the beta with the integration, you should look at the new notification examples.

I may be wrong but maybe best.jpg is the highest ranked image from that camera which may not be the most recent event. My theory: event 1’s highest % is 97, event 2 is 95%. Best is offering the event 1’s 97% image.

Blake can tell us for sure.

That’s correct. I would suggest using the integration and looking at the new notification examples. It was designed to solve exactly this outdated image problem and eliminate the need to use the camera snapshot service.

I don’t do the https://your.public.hass.address.com part. I use VPN to get in from outside my network.
So I wanted to find a way to get the images from a local source (hence the camera snapshot service) and then send them out.
I missed the part in the Readme with the link to the different examples. I’ll definitely check it out.
Thanks

You can fetch the same images directly from frigate at /events/<id>/snapshot.jpg.

First, thank you so much for building this - it’s amazing.

I have the binary sensor working and its sending an image into hass for display via MQTT, but I cant get the notifications to my phone to display the URL from the frigate server.

I have HassIO running on NUC1 and Frigate running in docker on NUC2 (separate machine). Should I be using the integration? I was under the impression that the addon would install the frigate docker container on my hass server.

@blakeblackshear

Hi,

Great add-on - very good docs and easy to install. Beta Integration worked without any issues.

Any chance of the integration natively incorporating a binary sensor that turns on when a person is detected? That way we can use frigate really easily with Alarmo and other alarm based integrations. Moreover, lots of blueprints have revelry appeared that would be able to be used with s binary sensor too.

In the mean time is anyone able to help create a template binary sensor for me? :rofl:

Thanks!

EDIT: Any idea why the stream entity and camera.camera_person entity looks, well, not great? The stream entity doesn’t work play properly and looks like the thumbnail and also shows green smears. I chose the substream (640x480) for detection, rtmp and the 1080p main stream for clips and record. The thumbnails of the clips don’t look great either but the actual video files are fine

Have a look at the example notifications, they should give you enough to get started on writing a specific automation for notifications.

@Eeeeeediot Check out the docs for creating binary sensors. They aren’t created automatically, because there’s a lot that you can do custom for each one.

I have a few and they’re simple to create in the configuration.yaml file in HA:

binary_sensor:
  - platform: mqtt
    state_topic: "frigate/front_patio/person"
    device_class: motion
    availability_topic: "frigate/available"
    name: "Front patio person motion"

  - platform: mqtt
    state_topic: "frigate/driveway/person"
    device_class: motion
    availability_topic: "frigate/available"
    name: "Driveway person motion"

  - platform: mqtt
    state_topic: "frigate/front_yard/person"
    device_class: motion
    availability_topic: "frigate/available"
    name: "Front yard person motion"

@paulstronaut posted docs for 0.7.x, I posted docs from 0.8.x. So check the version you’re using and use the docs accordingly :slight_smile:

Ah yes. The docs links are different, but the MQTT binary sensor configuration is the same :slight_smile:

1 Like

Please forgive my ignorance here, but what is/where can I find the <id> in /events/<id>/snapshot.jpg?

thanks

@paulstronaut @hawkeye217

Thanks, I overlooked that part because I used the custom component!

Any idea why the video stream isn’t working correctly? It works fine in HA directly from the camera

I’m getting the ID from events like below in my telegram automation - using beta3. I didn’t see much about eventid in the beta4 documentation.

  - alias: 'frigate_telegram'
    trigger:
      platform: mqtt
      topic: frigate/events
    condition:
      - "{{ trigger.payload_json['after']['label'] == 'person' }}"
#      - "{{ states('input_boolean.isalarmon') == 'on' }}"
#      - "{{ (now() - state_attr('automation.frigate_telegram','last_triggered')).seconds > 30 }}"
    action:
      - service: notify.telegram_full
        data_template:
          message: 'A {{trigger.payload_json["after"]["label"]}} with ID {{trigger.payload_json["after"]["id"]}} was detected at {{ trigger.payload_json["after"]["camera"] }} {{ "\n" -}} found at: http://<frigateip>:5000/{{ trigger.payload_json["after"]["camera"] }}-{{trigger.payload_json["after"]["id"]}}/snapshot.jpg?h=720'
          data:
            photo:
              - url: 'http://<frigateip>:5000/events/{{trigger.payload_json["after"]["id"]}}/snapshot.jpg?format=android'
                caption : 'A {{trigger.payload_json["after"]["label"]}} was detected at {{ trigger.payload_json["after"]["camera"] }} '