Frigate AI Notification - Intelligent Security Camera Automation version 0.43

[Whag]

Frigate LLM AI Notification - Intelligent Security Camera Automation

An advanced Home Assistant automation blueprint that combines Frigate NVR, LLM Vision AI, and mobile notifications to create intelligent, context-aware security alerts. This automation analyzes both snapshots and video clips using AI to provide detailed, actionable security notifications.

Features

AI-Powered Analysis

• Dual Analysis: Processes both initial snapshots and review clips
• Multiple LLM Providers: Supports OpenAI GPT, Claude, Gemini, and Llama models
• Contextual Understanding: AI describes what’s happening, why, and what might happen next
• Memory Integration: Uses LLM Vision’s memory and timeline features for enhanced context

Smart Notifications

• Rich Mobile Notifications: Includes images, videos, and actionable analyse
• Customizable Click Actions: Direct links to camera feeds, downloaded files, or dashboards
• Multiple Device Support: Send notifications to multiple mobile devices
• Notification Grouping: Organizes alerts by camera for better management

Advanced Configuration

• Zone-Based Filtering: Only alert for specific areas of interest
• Object Detection: Filter alerts by specific objects (person, car, etc.)
• Sub-label Support: Advanced filtering with Frigate sub-labels
• Cooldown Periods: Prevents notification spam
• Custom Actions: Run additional automations on events

Local File Management

• Download Integration: Stores Frigate files locally for improved reliability
• Organized Storage: Configurable directory structure for media files
• Parallel Processing: Handles multiple camera events simultaneously

Prerequisites

Before using this automation, ensure you have the following components installed and configured:

Required Integrations

1. Frigate Integration

• Frigate NVR with MQTT broker setup
• Camera configuration with zones and object detection
• Integration installed and configured in Home Assistant

2. LLM Vision Integration

• AI provider configured (OpenAI, Anthropic, Google, etc.)
• Memory and timeline features (optional but recommended)
• Integration installed and configured in Home Assistant

3. Downloader Integration

• Built-in Home Assistant integration
• Directory permissions configured for file storage

4. Home Assistant Companion App

• Installed on mobile devices for notifications
• Notification permissions enabled

System Requirements

• Home Assistant: Version 2023.1 or later
• MQTT Broker: Required for Frigate integration
• Storage Space: Adequate space for downloaded media files
• Mobile Device: Android or iOS with HA Companion App

Installation

Method 1: Blueprint Import (Recommended)

• Import the Blueprint:

https://github.com/willhaggan/HA_Frigate_VLLM_Notification/blob/main/Latest.yaml

1. Navigate to Settings → Automations & Scenes → Blueprints
2. Click “Import Blueprint” and paste the URL above
3. Create a new automation from the imported blueprint

Method 2: Manual Installation

• Download the Latest.yaml file from this repository
• Place the file in your Home Assistant blueprints directory:

config/blueprints/automation/frigate_llm_notification/

1. Restart Home Assistant or reload automations
2. Create automation from the blueprint in the UI

Configuration

The blueprint provides extensive configuration options organized into logical groups:

Frigate Options

• Camera Selection: Choose which Frigate camera to monitor
• Zone Configuration: Select specific zones for monitoring
• Object Filtering: Filter alerts by detected objects
• Sub-label Filtering: Advanced filtering with custom labels

LLM Vision Provider

• Provider Selection: Choose your AI provider (OpenAI, Claude, Gemini, etc.)
• Model Selection: Select specific AI model for analysis
• Performance Settings: Configure tokens, temperature, and creativity

Snapshot Analysis

• AI Analysis Toggle: Enable/disable AI analysis for snapshots
• Custom Prompts: Define how the AI should analyze images
• Title Generation: AI-generated notification titles
• Memory Integration: Use stored context for better analysis

Video Clip Analysis

• Clip Processing: Analyze video clips when events end
• Frame Analysis: Configure how many frames to analyze
• Advanced Prompts: Specialized prompts for video analysis
• Performance Tuning: Optimize for speed vs. quality

Downloader Options

• Directory Configuration: Set up local storage paths
• Download Timeouts: Configure wait times for file downloads
• Retry Logic: Automatic retry on download failures

Notification Settings

• Device Selection: Choose target mobile devices
• Click Actions: Configure notification tap behavior
• Cooldown Periods: Prevent notification spam
• Grouping Options: Organize notifications by camera

Custom Actions

• New Event Actions: Run additional automations on new events
• End Event Actions: Run actions when events complete
• Integration Hooks: Connect with other Home Assistant automations

Documentation

• Quick Start Guide - Get up and running in 15 minutes
• Configuration Guide - Detailed configuration options
• Examples and Use Cases - Real-world implementation examples
• Troubleshooting Guide - Common issues and solutions

Usage Examples

Basic Security Monitoring

Configuration for basic person detection Camera: camera.front_door Zones: [front_porch, driveway] Objects: [person] AI Prompt: “Describe who is approaching the front door and what they appear to be doing.”

Advanced Vehicle Monitoring

Configuration for driveway vehicle detection Camera: camera.driveway Zones: [driveway, street] Objects: [car, truck, motorcycle] AI Prompt: “Analyze the vehicle activity. Is this a delivery, visitor, or potential security concern?”

Multi-Zone Package Detection

Configuration for package delivery monitoring Camera: camera.porch Zones: [front_porch, steps] Objects: [person, package] AI Prompt: “Focus on package delivery activity. Describe if packages are being delivered or picked up.”

Best Practices

Performance Optimization

• Model Selection: Use faster models (gpt-4o-mini, gemini-1.5-flash) for real-time alerts
• Token Limits: Keep token counts reasonable (50-150) for faster responses
• Frame Limits: Limit video analysis to 3-5 frames for better performance

Storage Management

• Regular Cleanup: Implement periodic cleanup of downloaded files
• Directory Structure: Use organized subdirectories by date or camera
• Monitoring: Track storage usage to prevent disk full issues

Notification Strategy

• Cooldown Periods: Use 2-5 minute cooldowns to prevent spam
• Zone Optimization: Configure zones to reduce false positives
• Device Grouping: Group family devices for coordinated notifications

AI Prompt Engineering

• Be Specific: Include relevant context about what you want to detect
• Set Priorities: Specify what events are most important
• Include Context: Mention zones, objects, and expected activities

Troubleshooting

Common Issues

Notifications Not Received

• Verify Mobile App permissions for notifications
• Check device selection in automation configuration
• Ensure devices are online and connected

AI Analysis Failures

• Verify LLM Vision integration is properly configured
• Check API keys and provider status
• Review prompt complexity and token limits

File Download Errors

• Confirm Downloader integration is installed
• Verify directory permissions and available storage
• Check network connectivity to Frigate server

Event Detection Issues

• Review Frigate camera configuration and zones
• Check object detection settings and confidence levels
• Verify MQTT broker connectivity

For detailed troubleshooting, see the Troubleshooting Guide.

Contributing

Contributions are welcome! Please feel free to:

• Report bugs and issues
• Suggest new features
• Submit pull requests
• Share configuration examples
• Improve documentation

License

This project is licensed under the MIT License - see the repository for details.

Acknowledgments

• Frigate - Excellent NVR software
• LLM Vision - AI vision integration
• Home Assistant Community - Ongoing support and inspiration

---

Additional Resources

• Frigate Documentation
• LLM Vision Integration Guide
• Home Assistant Automation Documentation
• MQTT Configuration Guide

Updates July 2025

• Frigate Severity Selection:Select Alert or Detections

• Clickaction Options Expanded

• MQTT before_zones Added to Zone Match Logic

• Temperature parameter removed to align with LLM Vision 1.5
  Version: v0.42
  Author: whag
  Last Updated: July 2025

• Allow different providers/models for image and clip analyse

• Description tidy up

• New Groq models
  Version: v0.43
  Author: whag
  Last Updated: 22 July 2025

[Sir_Goodenough ((SG) WhatAreWeFixing.Today)]

Hello Whag,

Thanks for contributing to the community with a new Blueprint.
I have a suggestion for you. Many people who are not familiar with directory structures will have problems installing this without the Home Assistant MY tools.
Adding a MY link for this Blueprint to your top post would help them a lot.
Here is the link to make that.
Create a link – My Home Assistant
Note: if the original is in the forums here, only 1 code block can be in the top post ot the MY link tool will not work.

[Whag]

Delay time reduced before LLM video analysis

[Whag]

• Added sub directory option for downloader.
• Inputs added for delays before image and clip processing.
• Variable description changes and spelling corrections.
• Added Links for frigate and LLM Vision.

[mhanstead (Bobby D)]

I can’t seem to get this to work and the traces seem to indicate that it fails on the download.
Does this require using the frigate add-on within haos, or can it use a separate frigate instance? The frigate server config input shows a default port of 8123, which is the ha default port. My frigate server is on a separate server and uses the default frigate port of 8971. Would it not also need a frigate username and password to login to the frigate server? I am not familiar with how the api works. Or, am I supposed to put in the ha server IP here?
Thanks for your work.

EDIT: I got this to work by putting my HomeAssistant server address into the frigate server configuration box. I used https with my internal FQDN and port 8123.

[Whag]

Hi Bobby in my instance i am using the HA frigate addon so its all on the same machine so the default value i have entered in the BP was based off my setup, although you have it sorted now anyway. thanks

[rafsanjani (Rafsanjani Aziz)]

Hi, I am not using the frigate addon in ha, i am using frigate in a container on proxmox but the blueprint wants a frigate camera. How do I go about this?

[Whag]

Hi, Are you using the frigate integration this is how the cameras are exposed to HA.

[Whag]

Link to HACS: Repository – My Home Assistant

[rafsanjani (Rafsanjani Aziz)]

I already have frigate in a docker, do I still need to use the home assistant integration?

[Whag]

Yes you need to install the frigate integration which i linked previously and ensure you enable mqtt in your frigate config.

[rafsanjani (Rafsanjani Aziz)]

Thanks. That seemed to work, but now i’m getting a couple of warnings and no notification

[Whag]

Can you try the url in a browser and if possible the machine running HA to see if the url is accessible.

Regarding the zones have you added a zone in the input field.

[rafsanjani (Rafsanjani Aziz)]

Yes I have added a zone in the input field. also that url gives me a 404 not found. I am attaching my frigate config.

mqtt:
  host: 192.168.1.91
  port: 1883
  user: mqtt-user
  password: Looking123
  client_id: frigate
  topic_prefix: frigate

ffmpeg:
  hwaccel_args: preset-vaapi

detectors:
  ov:
    type: openvino
    device: AUTO

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  path: /openvino-model/ssdlite_mobilenet_v2.xml
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

record:
  enabled: true
  retain:
    days: 7
    mode: motion
  alerts:
    retain:
      days: 30
  detections:
    retain:
      days: 30
detect:
  enabled: true

snapshots:
  enabled: true
  retain:
    default: 30

cameras:
  kitchen:
    snapshots:
      enabled: true
      retain:
        default: 10
    detect:
      width: 1280
      height: 720
      fps: 5
    ffmpeg:
      inputs:
        - path: rtsp://rafsanjani:[email protected]:554/stream1
          roles:
            - record
        - path: rtsp://rafsanjani:[email protected]:554/stream2
          roles:
            - detect
    zones:
      zone_1:
        coordinates: 0.014,0.019,0.012,0.98,0.992,0.975,0.984,0.02
        loitering_time: 0
    review:
      alerts:
        required_zones: zone_1
version: 0.15-1

[Whag]

Hi there I have not setup frigate on a separate machine before so I do not have a direct means of testing your issues. I can only assume you need the frigate proxy add on or port 5000 needs opened on frigate server side.

Regarding the zone_names error… Did you select a zone from the predefined list or did you manually enter it.

[rafsanjani (Rafsanjani Aziz)]

I selected the zone from the predefined list. That warning seems to have gone now but it seems the media is still not getting downloaded.

[Whag]

Okay well that’s one thing.

I would advise opening port 5000 on frigate and maybe look at installing the proxy add-on or do some research on the external access

[Whag]

Is it giving the same error on the trace.

[Whag]

Updates: Version 0.2

• Added sections.

• Input entry for custom zones.

• Altered template for custom zone entry as well as the default device selector.

• Extra models added to models default list.

• Option to disable LLM Vision Image Analysis of initial notification, along with a custom message for the notification. should this be set to false.

• Prompt and related options split for image and clip analysis.

• Altered downloader directory and sub directory variables to add “/” if missing.

• Variable spelling correction in logbook.log action.

• Some variable names changed along with a few unused removed.

• Minor description changes.

To do: add input option for all zones required rather then just any zone matched

[THE_WHIFF_MASTER]

Hi, i am trying to setup the automation and it triggeres. But never got the notification to fire. it starts a loop and i think when it checks if the clip exist, it does not. is ther something i am missing? I’m not sure about my downloader configuration… is there a way to access the files directly on my frigate box?