Camera, Frigate: Intelligent AI-powered notifications

Hi,

I have a problem with AI-notification. In developer tools it works (AI description) but when my cameras recognise objects I have small notify with “person”, “dog”, “car” etc. I have Frigate (and notification from other blueprint works good). I have Google Generative AI intalled and added API (the same for Google Provider). And it is my notification on IOS (iPhone):

Traces: Traces - Pastebin.com
And my YAML for blueprint:
“alias: AI Event Summary (LLM Vision v1.3.5)
description: “”
use_blueprint:
path: valentinfrlch/event_summary.yaml
input:
mode: Frigate
notify_device:
- baee6028aad0ca03df8ad0dcf7a1ca1b
camera_entities:
- camera.garaz
- camera.kanciapa
- camera.kuchnia
- camera.ogrod
preview_mode: Live Preview
provider: 01JH86KC6YATGA5KPC6G2G7CVY
object_type:
cooldown: 1
message: >-
Describe what you see in this image from my security camera. Focus only on
dynamic objects like cars, people and animals. Rules for the response: 1.
Response must be in Polish 2. Keep it short (max 200 characters) to fit in
a mobile notification 3. Ignore all static objects like buildings, gates,
letterboxes etc. 4. If you see a person: mention their main
characteristics 5. If you see a car: mention its type and color 6. If
you see an animal: mention its species and behavior 7. Add a funny
comment or emoji occasionally 8. If you see multiple objects, focus on the
most important one - this will be most probablu surrounded by a bounding
box from the camer 8. Use present tense
max_tokens: 90
temperature: 0.5
detail: high
model: gemini1.5-flash”

Try changing your model to model: gemini-1.5-flash-8b

1 Like

Great :slight_smile: it works and description is in my language. As title I have „Person seen” - can I translate this?

And can I:

  1. set this that after click on the notification I can see shot clip?
  2. notification on my Telegram with snapshot (I set without snapshot and it works but which url out for last snapshot which was analysed?)

I have no idea! There was a mention about telegram earlier in the post I think, have a read through and I’m 60% sure the solutions are 100% there.

Upon further investigation, I noticed that the description also appears to be truncated in the calendar entries, so perhaps the issue is not with the iOS notification…

In case it might be helpful to someone else, I solved my problem by increasing the Maximum number of tokens to generate to 25.

1 Like

Were you able to get this to work with BI?

Kudos to @valentinfrlch
I’ve used a couple of different AI camera blueprints and this is the one I’m staying with. I can tell a lot of work has gone into this.

My respectful feedback is that it would be wonderful if there were an condition so I could abort the automation if my door sensor was open. (in my case, its a doorbell camera - I don’t need a notification if I’m walking out the door.)

Thank you.

1 Like

I’m trying this out and I get so far that I get a notification with only “Person seen” and the camera name with a snapshot. I do not get any AI information

alias: AI Notification Frigate
description: ""
use_blueprint:
  path: valentinfrlch/event_summary.yaml
  input:
    mode: Frigate
    notify_device:
      - XXX
    camera_entities:
      - camera.uppfart
    object_type:
      - person
    provider: XXX
    model: gemini-1.5-flash
    max_tokens: 90
    cooldown: 1
    preview_mode: Snapshot

In traces it shows this

I also have this error message and the automation is stuck never completing

Logger: homeassistant.components.automation.ai_notification_frigate
Source: components/automation/__init__.py:664
integration: Automation (documentation, issues)
First occurred: 18 January 2025 at 09:22:46 (3 occurrences)
Last logged: 18 January 2025 at 22:31:23

Error rendering variables: UndefinedError: 'dict object' has no attribute 'payload_json'

Same here, also getting this exact same error nowadays but at least the automation still works, and I get the AI summary.

I get the exact same error with the blueprint now. It worked previously well. Then I moved my frigate server and reinstalled everything from scratch in home assistant (including MQTT broker). Now it seems the automation does never trigger and leaves the above error in the log.

Seems indeed related to latest version. I rolled back to 1.3.5 and the automation works like a charm.

I’m running 1.3.5 on the blueprint and I’ve always had that issue

In fact, it came back… I ran the 1.3.5 and all notifications started to flow. Then after a while the error appeared again. I then upgraded to 1.3.7. and downgraded again and it worked again. After a while, it stopped again. I will let it run for a while to see if it is just intermittent. It’s an odd one to reproduce.