Recent versions of Frigate introduced the ability to use Generative AI to describe reviews or sequences of objects (people, cars, motorcycles, dogs, etc.) in images. The community has been very active in suggesting improvements. While Frigate’s documentation is excellent, it sometimes requires reading, re-reading, and some forum deep-dives to realize that everything you needed was there all along—you just had to connect the dots! (lol)
All joking aside, the documentation is top-tier but requires a learning curve. This tutorial aims to shorten that path for those who want to start using AI to describe their camera scenes without getting lost in the technical weeds.
I won’t go into every detail, so I highly recommend consulting the official docs for:
Currently (v0.17.1), Frigate supports Gemini, Ollama, OpenAI, or Azure OpenAI. For cost-efficiency and because I don’t have the hardware to run Ollama locally, I chose Gemini. It offers a generous free tier of tokens which works perfectly if you don’t keep GenAI running 24/7 or have extremely high demand.
1) Activating GenAI and Configuring Scope
To use GenAI, you need to update your configuration file with your provider, credentials, and model version.
genai:
provider: gemini
api_key: YOUR_GEMINI_API_KEY
model: gemini-2.5-flash
Note: Many beginners try to use a standard Google account key. You must generate a specific API key at Google AI Studio.
Activating GenAI for objects is mandatory for this tutorial because the MQTT topic that triggers descriptions is currently published for objects, not for reviews. You can define specific prompts for each object type to get better results:
objects:
genai:
enabled: true
prompt: >-
Analyze the {label} in these security camera images from {camera}.
Focus on actions, behavior, and possible intent rather than just appearance.
object_prompts:
person: >-
Examine the main person. What are they doing and what do their actions suggest
(e.g., approaching a door, leaving, loitering)? Ignore static environment details.
car: >-
Observe the main vehicle. Focus on movement, direction, or purpose
(e.g., parking, approaching, circling). If it's a delivery vehicle, mention the company.
Cite the license plate if identifiable.
motorcycle: >-
Observe the main motorcycle. Focus on movement, direction, or purpose.
Note if it has a delivery box or similar accessory. Cite the license plate if possible.
You can also enable GenAI for reviews to add flexibility to your alerts:
review:
genai:
enabled: true
Pro Tip: Just like other Frigate settings, these can be overridden at the camera level. This is crucial to filter AI usage and avoid burning through your free tokens. Below is an example of limiting GenAI to specific objects and zones:
cameras:
front_garage:
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/garage_view
roles:
- record
- detect
objects:
track:
- person
- car
- motorcycle
- dog
genai:
enabled: true
objects:
- car
- motorcycle
required_zones:
- garage_entrance
- sidewalk
zones:
sidewalk:
coordinates: 0,0.174,0.035,0.164,0.046,0.075,0.204,0.065,0.406,0.406,0.763,1,0.079,1
garage_entrance:
coordinates: 0.03,0.545,0.083,0.546,0.16,1,0,1,0,0.247
Once configured, you can see these descriptions in the Frigate UI under Reviews or History.
2) Controlling GenAI via Home Assistant
To keep costs down, you might want to turn GenAI off when you are home. Frigate allows interaction via MQTT.
Topics:
- Set:
frigate/<camera_name>/object_descriptions/set - State:
frigate/<camera_name>/object_descriptions/state - Set:
frigate/<camera_name>/review_descriptions/set - State:
frigate/<camera_name>/review_descriptions/state
Automations to Toggle GenAI
Turn GenAI ON (when Alarm is Armed):
alias: Frigate GenAI - Enable when Armed
triggers:
- trigger: state
entity_id: alarm_control_panel.home_alarm
to: "armed_away"
actions:
- repeat:
for_each: "{{ cameras }}"
sequence:
- action: mqtt.publish
data:
topic: frigate/{{ repeat.item }}/review_descriptions/set
payload: "ON"
- action: mqtt.publish
data:
topic: frigate/{{ repeat.item }}/object_descriptions/set
payload: "ON"
variables:
cameras:
- camera_1
- camera_2
Optional: Create MQTT Binary Sensors
Add these to your configuration.yaml to see the status in your Dashboard:
mqtt:
- binary_sensor:
- name: "Camera 1 GenAI Review"
unique_id: camera1_genai_review
state_topic: "frigate/camera1/review_descriptions/state"
payload_on: "ON"
payload_off: "OFF"
- name: "Camera 1 GenAI Object"
unique_id: camera1_genai_object
state_topic: "frigate/camera1/object_descriptions/state"
payload_on: "ON"
payload_off: "OFF"
3) Telegram Notifications with AI Descriptions
We will use the frigate/tracked_object_update topic, filtering for the description type. We also need a REST command to fetch event details.
1. Create the REST command (configuration.yaml):
rest_command:
frigate_get_event:
url: "http://YOUR_FRIGATE_IP:5000/api/events/{{ event_id }}"
method: GET
2. Create the notification automation:
alias: Frigate GenAI Telegram Notification
triggers:
- trigger: mqtt
topic: frigate/tracked_object_update
value_template: "{{ value_json.type }}"
payload: description
actions:
- action: rest_command.frigate_get_event
data:
event_id: "{{ event_id }}"
response_variable: frigate_response
- condition: template
value_template: "{{ frigate_response['status'] == 200 }}"
- variables:
camera: "{{ frigate_response['content']['camera'] }}"
label: "{{ frigate_response['content']['label'] }}"
clip_url: "http://YOUR_FRIGATE_IP:5000/api/events/{{ event_id }}/clip.mp4"
caption: |
📷 *{{ camera }}* detected: *{{ label }}* 📝 {{ description }}
- delay: "00:00:10" # Wait for the clip to finish writing to disk
- action: telegram_bot.send_video
data:
url: "{{ clip_url }}"
caption: "{{ caption }}"
target: YOUR_TELEGRAM_CHAT_ID
variables:
event_id: "{{ trigger.payload_json.id }}"
description: "{{ trigger.payload_json.description }}"
That’s it! Now you have a smart, AI-powered surveillance system that tells you exactly what is happening in your home. Have fun!