Camera, Frigate: Intelligent AI-powered notifications

kuba12331 · January 10, 2025, 4:07pm

Hi,

I have a problem with AI-notification. In developer tools it works (AI description) but when my cameras recognise objects I have small notify with “person”, “dog”, “car” etc. I have Frigate (and notification from other blueprint works good). I have Google Generative AI intalled and added API (the same for Google Provider). And it is my notification on IOS (iPhone):

Traces: Traces - Pastebin.com
And my YAML for blueprint:
“alias: AI Event Summary (LLM Vision v1.3.5)
description: “”
use_blueprint:
path: valentinfrlch/event_summary.yaml
input:
mode: Frigate
notify_device:
- baee6028aad0ca03df8ad0dcf7a1ca1b
camera_entities:
- camera.garaz
- camera.kanciapa
- camera.kuchnia
- camera.ogrod
preview_mode: Live Preview
provider: 01JH86KC6YATGA5KPC6G2G7CVY
object_type:
cooldown: 1
message: >-
Describe what you see in this image from my security camera. Focus only on
dynamic objects like cars, people and animals. Rules for the response: 1.
Response must be in Polish 2. Keep it short (max 200 characters) to fit in
a mobile notification 3. Ignore all static objects like buildings, gates,
letterboxes etc. 4. If you see a person: mention their main
characteristics 5. If you see a car: mention its type and color 6. If
you see an animal: mention its species and behavior 7. Add a funny
comment or emoji occasionally 8. If you see multiple objects, focus on the
most important one - this will be most probablu surrounded by a bounding
box from the camer 8. Use present tense
max_tokens: 90
temperature: 0.5
detail: high
model: gemini1.5-flash”

RoadkillUK · January 10, 2025, 4:16pm

Try changing your model to model: gemini-1.5-flash-8b

kuba12331 · January 10, 2025, 7:34pm

Great it works and description is in my language. As title I have „Person seen” - can I translate this?

And can I:

set this that after click on the notification I can see shot clip?
notification on my Telegram with snapshot (I set without snapshot and it works but which url out for last snapshot which was analysed?)

RoadkillUK · January 10, 2025, 8:17pm

I have no idea! There was a mention about telegram earlier in the post I think, have a read through and I’m 60% sure the solutions are 100% there.

asp · January 12, 2025, 1:22pm

Upon further investigation, I noticed that the description also appears to be truncated in the calendar entries, so perhaps the issue is not with the iOS notification…

asp · January 15, 2025, 11:38am

In case it might be helpful to someone else, I solved my problem by increasing the Maximum number of tokens to generate to 25.

Sireone · January 16, 2025, 1:04pm

Were you able to get this to work with BI?

rayellio77 · January 16, 2025, 7:50pm

Kudos to @valentinfrlch
I’ve used a couple of different AI camera blueprints and this is the one I’m staying with. I can tell a lot of work has gone into this.

My respectful feedback is that it would be wonderful if there were an condition so I could abort the automation if my door sensor was open. (in my case, its a doorbell camera - I don’t need a notification if I’m walking out the door.)

Thank you.

johnhenriksson · January 17, 2025, 2:25pm

I’m trying this out and I get so far that I get a notification with only “Person seen” and the camera name with a snapshot. I do not get any AI information

alias: AI Notification Frigate
description: ""
use_blueprint:
  path: valentinfrlch/event_summary.yaml
  input:
    mode: Frigate
    notify_device:
      - XXX
    camera_entities:
      - camera.uppfart
    object_type:
      - person
    provider: XXX
    model: gemini-1.5-flash
    max_tokens: 90
    cooldown: 1
    preview_mode: Snapshot

In traces it shows this

johnhenriksson · January 19, 2025, 6:34am

I also have this error message and the automation is stuck never completing

Logger: homeassistant.components.automation.ai_notification_frigate
Source: components/automation/__init__.py:664
integration: Automation (documentation, issues)
First occurred: 18 January 2025 at 09:22:46 (3 occurrences)
Last logged: 18 January 2025 at 22:31:23

Error rendering variables: UndefinedError: 'dict object' has no attribute 'payload_json'

ovizii · January 19, 2025, 2:20pm

Same here, also getting this exact same error nowadays but at least the automation still works, and I get the AI summary.

patrick.scheidegger · January 30, 2025, 12:18pm

I get the exact same error with the blueprint now. It worked previously well. Then I moved my frigate server and reinstalled everything from scratch in home assistant (including MQTT broker). Now it seems the automation does never trigger and leaves the above error in the log.

patrick.scheidegger · January 30, 2025, 12:49pm

Seems indeed related to latest version. I rolled back to 1.3.5 and the automation works like a charm.

johnhenriksson · January 31, 2025, 9:08am

I’m running 1.3.5 on the blueprint and I’ve always had that issue

patrick.scheidegger · February 1, 2025, 8:08am

In fact, it came back… I ran the 1.3.5 and all notifications started to flow. Then after a while the error appeared again. I then upgraded to 1.3.7. and downgraded again and it worked again. After a while, it stopped again. I will let it run for a while to see if it is just intermittent. It’s an odd one to reproduce.