Local realtime person detection for RTSP cameras

You’ve got it
Example: “http://192.168.X.XX:5000/api/pool/person/best.jpg

So when I do http://IP:PORT/api/CAMERANAME/person/best.jpg on my frigate instance, it shows up a black image. I then realized why, but not sure the reason yet, when I restart the container on my Kubernetes instance, the thumbnail is black until the next event comes in:

However if I click on the person or car events, all my events do indeed show up persisted on my storage.

That is intended. The best images are in memory only. It a “best since frigate started”.

That won’t work for notifications on the native app because it shouldn’t be exposed externally.

Got it, thank you!

Also thank you for the integration/nvr, they work amazingly and definitely appreciate the work you’ve put into this.

That’d be fine, I’m protecting cars, which wouldn’t be there if we’re not home. :smiley:
I can’t get it working for some reason anyway. There’s got to be something else going on. Sometimes I get a bad url attachment, other times no attachment at all, with the same automation no changes.

I also tried using the camera.name_person entity and pushed as the camera category. No attachment.
MRobi’s suggestion to use the person entity, however, is much better than mqtt.

Using the dev tools, and an externally hosted image that I KNOW is accessible, it still doesn’t work. I’ve seen threads where this happens periodically.

As I typed that I realized the new integration has an actual motion binary sensor that’s created, so I’m in the process of re-creating with motion = on which should be a bit cleaner and easier to create a group of motion sensors. I use it to turn on lights when I’m home with no notification, and to send a notification when nobody is home.

For anyone interested, I use images from HA camera entities created by Frigate to send as notifications:

http://hass.local:8123{{ state_attr("camera.driveway_person","entity_picture") }}

I maybe obtuse, but I don’t understand how that template works. You use that as the attachment URL? It’s a malformed URL.

1 Like

The entity_picture attribute is a string containing the url path to an image for that camera. I append that to the domain for my HA instance to form a full url. I use Telegram for notifications so the data format is a bit different.

message: Person detected near driveway
data:
  photo:
    caption: '{{ now() | as_timestamp | timestamp_custom("%y-%m-%d %H:%M:%S") }}'
    url: >-
      http://hass.local:8123{{ state_attr("camera.driveway_person","entity_picture") }}

Edit: I just noticed I can simply my code a bit:
now().strftime("%y-%m-%d %H:%M:%S")
instead of
now() | as_timestamp | timestamp_custom("%y-%m-%d %H:%M:%S")

In the web UI, we don’t currently show actual video, but a live-updating JPEG image. There are a bunch of technical reasons for this – but mostly because that’s a debug view and not meant for “live viewing”.

There is work starting around performant output streaming that we can use in the web ui, but it will be some time before that becomes usable.

2 Likes

If I’m understanding correctly, you capture an image from the camera stream at the time of the notification correct? Would it not still be better to use the best.jpg from the frigate api which takes the best snapshot of the event?

HA provides still images of camera streams so that’s what I use.

The image from HA is cropped to just show the detected object. Maybe that image is available through the Frigate API as well but I’d have to go looking for it and I know where to find it in HA already.

I’m using Pushover to send notifications, but you have to use the built-in HA downloader service to first download the image and then attach it to Pushover. Here is my automation file:

downloader:
  download_dir: www
  
notify:
  - name: joe_phone
    platform: pushover
    api_key: <redacted>
    user_key: <redacted>

automation:
  - alias: "driveway notfication"
    trigger:
      platform: mqtt
      topic: frigate/events
    condition: 
      - "{{ trigger.payload_json['after']['label'] == 'person' }}"
      - "{{ trigger.payload_json['after']['camera'] == 'driveway' }}"
      - "{{ ( as_timestamp(now()) - as_timestamp(state_attr('automation.driveway_notfication', 'last_triggered')) |int(0) ) > 45 }}"
    action:
      - service: downloader.download_file
        data_template:
          url: 'http://10.200.200.18:5000/api/events/{{trigger.payload_json["after"]["id"]}}/thumbnail.jpg?format=android'
          filename: "thumbnail.jpg"
          overwrite: true
      - delay: '00:00:01'
      - service: notify.joe_phone
        data:
          message: "Person in Driveway"
          data:
            attachment: /config/www/thumbnail.jpg
            priority: 0

You have to do it this convoluted way because you can’t attach files using Pushover any longer that aren’t whitelisted as documented here: https://github.com/home-assistant/core/issues/32236. So you have to first use the downloader to download the thumbnail.jpg into the whitelisted folder of www and then you can attach the .jpg located in that folder to a Pushover notification.

I have to have to use a 1 second delay between using the Downloader service and actually sending the .jpg to give the downloader service enough time to finish or else Pushover would work too fast and it would attach the .jpg from a prior event.

4 Likes

I’m running my hassio using on a virtual box on a win 10 machine. Also running my Emby media server on it, will adding frigate slow my cpu drastically? Will the coral USB dongle help?

It will indeed use a lot of processing power. I’m waiting on Coral to be shipped, it seems to be out of stock everywhere.

Well I installed it via add-on on my hassio which is located in a virtualbox. It keeps crashing though and stop working.

 * Starting nginx nginx
   ...done.
frigate.app                    INFO    : Creating directory: /tmp/cache
Starting migrations
peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
peewee_migrate                 INFO    : There is nothing to migrate
frigate.mqtt                   INFO    : MQTT connected
detector.coral                 INFO    : Starting detection process: 35
frigate.edgetpu                INFO    : Attempting to load TPU as usb
frigate.edgetpu                INFO    : No EdgeTPU detected.
Process detector:coral:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 152, in load_delegate
    delegate = Delegate(library, options)
  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 111, in __init__
    raise ValueError(capture.message)
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/frigate/frigate/edgetpu.py", line 124, in run_detector
    object_detector = LocalObjectDetector(tf_device=tf_device, num_threads=num_threads)
  File "/opt/frigate/frigate/edgetpu.py", line 63, in __init__
    edge_tpu_delegate = load_delegate('libedgetpu.so.1.0', device_config)
  File "/usr/local/lib/python3.8/dist-packages/tflite_runtime/interpreter.py", line 154, in load_delegate
    raise ValueError('Failed to load delegate from {}\n{}'.format(
ValueError: Failed to load delegate from libedgetpu.so.1.0
frigate.watchdog               INFO    : Detection appears to have stopped. Exiting frigate...
frigate.app                    INFO    : Stopping...
frigate.record                 INFO    : Exiting recording maintenance...
frigate.events                 INFO    : Exiting event processor...
frigate.object_processing      INFO    : Exiting object processor...
frigate.watchdog               INFO    : Exiting watchdog...
frigate.events                 INFO    : Exiting event cleanup...

Please read the docs carefully u must define a detector maybe it’ll be better making /providing standard minimal config with CPU detector

Unable to connect to Frigate. the add-on works but HACS custom component fails.

I’ve made tremendous progress using the add-on and can reliably detect person(s). Thanks for those who’ve helped so far.

I’m trying again to get the HACS custom integration version instead of the add-on. If I’m understanding the docs right then the custom component will add more integration of sensors, etc.

I removed the add-on and installed the integration from HACS. This all seems OK, but when I try to add the integration I get the configuration dialog and enter http://ccab4aaf-frigate:5000 as the host.
“Unable to connect to Frigate” is the response.

My /config/frigate.yml file is unchanged from when I used the add-on. The add-on worked for 24+ hours, survived reboots, and had no log errors.

Am I missing something? I really want to get the custom integration working before I spend too much time creating the UI elements and automations. I eventually plan to have 12+ cameras.

Please treat me as a noob to this environment. I have no knowledge of docker but a good general technology knowledge and reasonable Linux experience (from 10 years ago)

I’m running Debian 10, RPi 4 4G, HA Supervised and booting off an SSD. Everything is latest release and no Beta stuff installed.

Try your Hass IP iso http://ccab4aaf-frigate:5000/