Frigate Review with LLM Vision Notification version 0.4

Blueprints Exchange
, , , ,
1

Use frigate and LLM Vision to send mobile notifications.

Open your Home Assistant instance and show the blueprint import dialog with a specific blueprint pre-filled.

I decided to make this BP after having too many false positives with other blueprints with various triggers and quite often failed attempts at downloading the frigate files from the api for image and video analysis.

I have included a file check with the downloader integration before asking LLM vision to analyse the image or clip, if a failure occurs the automation will loop the actions and check again after a delay should the file download fail or not download in a specified time.

This is a work in progress and a automation i work on as and when i find errors and or improvements etc. so if you have any recommendations feel free to let me know and help if you can.

Configurable Inputs:

  • Frigate Cameras
  • Frigate Zones
  • Custom Zones
  • All Zones Required
  • Frigate Objects
  • Frigate Server address
  • LLM Vision Provider
  • LLM Vision Models (Common)
  • Custom Notification Message (image)
  • Custom Notification Title (image/video)
  • Enable LLM For Inital Notification
  • LLM Vision Prompt (image/video)
  • LLM Vision Target Pixels (image/video)
  • LLM Vision Remember (image/video)
  • LLM Vision Memory (image/video)
  • LLM Vision Tokens (image/video)
  • LLM Vision Creativity (image/video)
  • LLM Vision Frames (video)
  • LLM Vision Generate Title (image/video)
  • Downloader Directory
  • Downloader Sub Directory
  • Delay before snapshot analysis.
  • Delay before review clip analysis.
  • Wait for snapshot download.
  • Wait for review clip download
  • Companion App mobile devices
  • Click action of notification
  • Cool down: This allows a new and end event every 2 minutes for example.

Fixed parameters
Parallel automation with max set at 2.

Includes a logbook log event with quite a few details for debugging if required.

Requires:
- Home Assistant with an MQTT broker setup.
- Frigate integration with MQTT setup.
- LLM Vision integration with provider.
- Downloader integration.

I have only set this up for one Frigate camera, so you must use multiple instances of this automation for multiple cameras.

Give it a try let me know how you get on. Thanks

Links to the brillant integrations used.

Frigate:

LLM Vision:

Downloader:

2 Likes
2

Hello Whag,

Thanks for contributing to the community with a new Blueprint.
I have a suggestion for you. Many people who are not familiar with directory structures will have problems installing this without the Home Assistant MY tools.
Adding a MY link for this Blueprint to your top post would help them a lot.
Here is the link to make that.
Create a link – My Home Assistant
Note: if the original is in the forums here, only 1 code block can be in the top post ot the MY link tool will not work.

3

Delay time reduced before LLM video analysis

4
  • Added sub directory option for downloader.
  • Inputs added for delays before image and clip processing.
  • Variable description changes and spelling corrections.
  • Added Links for frigate and LLM Vision.
5

I can’t seem to get this to work and the traces seem to indicate that it fails on the download.
Does this require using the frigate add-on within haos, or can it use a separate frigate instance? The frigate server config input shows a default port of 8123, which is the ha default port. My frigate server is on a separate server and uses the default frigate port of 8971. Would it not also need a frigate username and password to login to the frigate server? I am not familiar with how the api works. Or, am I supposed to put in the ha server IP here?
Thanks for your work.

EDIT: I got this to work by putting my HomeAssistant server address into the frigate server configuration box. I used https with my internal FQDN and port 8123.

6

Hi Bobby in my instance i am using the HA frigate addon so its all on the same machine so the default value i have entered in the BP was based off my setup, although you have it sorted now anyway. thanks

7

Hi, I am not using the frigate addon in ha, i am using frigate in a container on proxmox but the blueprint wants a frigate camera. How do I go about this?

8

Hi, Are you using the frigate integration this is how the cameras are exposed to HA.

9
10

I already have frigate in a docker, do I still need to use the home assistant integration?

11

Yes you need to install the frigate integration which i linked previously and ensure you enable mqtt in your frigate config.

12

Thanks. That seemed to work, but now i’m getting a couple of warnings and no notification

image
image1996Ă—724 58.2 KB

13

Can you try the url in a browser and if possible the machine running HA to see if the url is accessible.

Regarding the zones have you added a zone in the input field.

14

Yes I have added a zone in the input field. also that url gives me a 404 not found. I am attaching my frigate config.

image
image2242Ă—556 34 KB

mqtt:
  host: 192.168.1.91
  port: 1883
  user: mqtt-user
  password: Looking123
  client_id: frigate
  topic_prefix: frigate

ffmpeg:
  hwaccel_args: preset-vaapi

detectors:
  ov:
    type: openvino
    device: AUTO

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  path: /openvino-model/ssdlite_mobilenet_v2.xml
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

record:
  enabled: true
  retain:
    days: 7
    mode: motion
  alerts:
    retain:
      days: 30
  detections:
    retain:
      days: 30
detect:
  enabled: true

snapshots:
  enabled: true
  retain:
    default: 30

cameras:
  kitchen:
    snapshots:
      enabled: true
      retain:
        default: 10
    detect:
      width: 1280
      height: 720
      fps: 5
    ffmpeg:
      inputs:
        - path: rtsp://rafsanjani:[email protected]:554/stream1
          roles:
            - record
        - path: rtsp://rafsanjani:[email protected]:554/stream2
          roles:
            - detect
    zones:
      zone_1:
        coordinates: 0.014,0.019,0.012,0.98,0.992,0.975,0.984,0.02
        loitering_time: 0
    review:
      alerts:
        required_zones: zone_1
version: 0.15-1
15

Hi there I have not setup frigate on a separate machine before so I do not have a direct means of testing your issues. I can only assume you need the frigate proxy add on or port 5000 needs opened on frigate server side.

Regarding the zone_names error… Did you select a zone from the predefined list or did you manually enter it.

16

I selected the zone from the predefined list. That warning seems to have gone now but it seems the media is still not getting downloaded.

17

Okay well that’s one thing.

I would advise opening port 5000 on frigate and maybe look at installing the proxy add-on or do some research on the external access

18

Is it giving the same error on the trace.

19

Updates: Version 0.2

  • Added sections.

  • Input entry for custom zones.

  • Altered template for custom zone entry as well as the default device selector.

  • Extra models added to models default list.

  • Option to disable LLM Vision Image Analysis of initial notification, along with a custom message for the notification. should this be set to false.

  • Prompt and related options split for image and clip analysis.

  • Altered downloader directory and sub directory variables to add “/” if missing.

  • Variable spelling correction in logbook.log action.

  • Some variable names changed along with a few unused removed.

  • Minor description changes.

To do: add input option for all zones required rather then just any zone matched

20

Hi, i am trying to setup the automation and it triggeres. But never got the notification to fire. it starts a loop and i think when it checks if the clip exist, it does not. is ther something i am missing? I’m not sure about my downloader configuration… is there a way to access the files directly on my frigate box?