Camera, Frigate: Intelligent AI-powered notifications

woody4165 · November 26, 2024, 7:32pm

Hi all

I’ve just installed this fantastic blueprint, but I’m having this issue

2024-11-26 20:12:25.527 ERROR (MainThread) [homeassistant.components.automation.test_ai_event_summary_llm_vision_v1_3_1] Error rendering variables: UndefinedError: 'dict object' has no attribute 'payload_json'

And this is my config, GUI and yaml

alias: TEST - AI Event Summary (LLM Vision v1.3.1)
description: ""
use_blueprint:
  path: valentinfrlch/event_summary.yaml
  input:
    mode: Frigate
    important: false
    notify_device:
      - 27f348f056567af0123402edbcf70fe5
    camera_entities:
      - camera.vialetto_3
    provider: 01JDM83JVTT9FX0DSDAPJGTE15
    detail: high
    motion_sensors: []

I’ve set Gemini as AI provider

What can I check?

Thanks

RoadkillUK · November 26, 2024, 8:15pm

I have this working on my setup using Gemini, this is how I have mine set.

kilopter · November 27, 2024, 3:08am

works perfect with frigate and live preview … not luck with snapshots;( any ideas?

woody4165 · November 27, 2024, 9:18am

Now, I’ve changed from Frigate to Camera and I get just a notification title in the app saying “Motion detected”, but I don’t see anything else

These are now the error messages I get in the log

  File "/config/custom_components/llmvision/__init__.py", line 305, in stream_analyzer
  File "/config/custom_components/llmvision/media_handlers.py", line 428, in add_streams
  File "/config/custom_components/llmvision/media_handlers.py", line 220, in record
  File "/config/custom_components/llmvision/media_handlers.py", line 163, in record_camera
2024-11-27 10:14:17.054 ERROR (MainThread) [homeassistant.components.automation.test_ai_event_summary_llm_vision_v1_3_1] TEST - AI Event Summary (LLM Vision v1.3.1): Error executing script. Unexpected error for choose at pos 4: 'NoneType' object has no attribute 'attributes'
  File "/config/custom_components/llmvision/__init__.py", line 305, in stream_analyzer
  File "/config/custom_components/llmvision/media_handlers.py", line 428, in add_streams
  File "/config/custom_components/llmvision/media_handlers.py", line 220, in record
  File "/config/custom_components/llmvision/media_handlers.py", line 163, in record_camera
2024-11-27 10:14:17.062 ERROR (MainThread) [homeassistant.components.automation.test_ai_event_summary_llm_vision_v1_3_1] While executing automation automation.test_ai_event_summary_llm_vision_v1_3_1
  File "/config/custom_components/llmvision/__init__.py", line 305, in stream_analyzer
  File "/config/custom_components/llmvision/media_handlers.py", line 428, in add_streams
  File "/config/custom_components/llmvision/media_handlers.py", line 220, in record
  File "/config/custom_components/llmvision/media_handlers.py", line 163, in record_camera

Any help?

Thanks

jasonsf · November 28, 2024, 3:46am

Was just going to post this. I also had to handle the apostrophe to get it working.

Rlwhittle · November 28, 2024, 2:49pm

Does anyone know if it’s possible to only trigger this for a certain zone in frigate?

jprazzle · November 29, 2024, 1:28pm

can you share the regex pattern you’re using?

farsonic · November 29, 2024, 10:49pm

Does anyone have a really simple config or example on getting Frigate working with MQTT. Trying to get LLM Vision working but while I can see Frigate connecting to mosquito I’m not getting any MQTT entries.

2024-11-29 10:57:14: New connection from 192.168.0.202:44979 on port 1883.

2024-11-29 10:57:14: New client connected from 192.168.0.202:44979 as frigate (p2, c1, k60, u’frigate’).

Juiceman8686 · December 1, 2024, 4:14pm

did you ever figure out what the issue was? I’m having the same problem

vulcanjedi · December 4, 2024, 4:45pm

Struggling to get this blueprint working? Any insights?

Message malformed: Unable to determine action @ data[‘action’][0][‘choose’][0][‘sequence’][0][‘choose’][0][‘sequence’][0]

description: ""
use_blueprint:
  path: valentinfrlch/event_summary.yaml
  input:
    mode: Frigate
    remember: true
    notify_device:
      - 015706a990344xxxxxx
    camera_entities:
      - camera.frontdoor
    provider: dcaadcde4bxxxxxxxx
    model: gemini-1.5-flash-latest
alias: AItest

spr33 · December 5, 2024, 7:47am

This is a brilliant blueprint! I managed to get it working last night after a few hurdles (probably nothing noteworthy for most people here).

Just wondering if there is any chance of someone explaining how I would go about getting the notification to be read out on my Google Home speakers? Ideally I’d not have to look at my phone/watch to see what it is going on outside my property but to hear what it is whilst I’m working - then I can action if it’s something out of the ordinary or remain seated.

Juiceman8686 · December 5, 2024, 7:22pm

Figured out my own issue. I was getting the error due to running the automation as a test to verify if it was working. Because there was not information to pull from frigate or MQTT this error would occur. If I actually walked in front of my camera the notification would work flawlessly as designed. Thank you for this great blueprint. My wife and I are getting a kick out of the information it spits out.

shunopoli · December 7, 2024, 4:27am

Not sure what im doing wrong but getting this

Logger: homeassistant.components.automation.ai_event_summary_llm_vision_v1_3_1
Source: components/automation/__init__.py:764
integration: Automation (documentation, issues)
First occurred: 10:24:14 PM (1 occurrences)
Last logged: 10:24:14 PM

Error while executing automation automation.ai_event_summary_llm_vision_v1_3_1: Entity does not exist

smolandia · December 7, 2024, 11:56am

I really like the idea of this!! Very cool and fun! However, for me I just get “Person seen” and no other text at all in my notifcations. Maybe the screenshot here is a hint? I have no idea how to troubleshoot…

conma293 · December 9, 2024, 4:24am

I would love this also for Alexa…

Still wish my android would show more than 20 words as I like the sassy reponses

smolandia · December 9, 2024, 9:32am

So I figured out what my problem was. Turns out under “model” I had to put “gemini-1.5-flash” … I assumed “Leave blank to automatically detect the best model.” would choose something that works, but it didnt. Maybe this can help someone else

smolandia · December 9, 2024, 9:35am

So I got this working and obiously it is very fun. The messages are cut off however mid-sentence. It does seem that the limit of number of characters here is smaller than if I send a notification “manually”. Does anyone know what the limit is? I have been trying to find this info with no luck. How are you handling this problem?

trivgar · December 13, 2024, 6:39pm

Hey great blueprint! I am wonder if it possible to have a per camera cool down ? Aside from that it appears to be working very well for me. I have not figured out the character limit for notifications yet either.

Warthog · December 20, 2024, 4:35pm

How did you debug? I was getting same thing, tried openai, tried putting the Gemini model like you did but still getting the generic messages

Edit Logs had some clues and it seems adding the calendar in the integration sorted it for me

Edit2 Or not… Not sure, I seem to now be getting both, something seen and more descriptive later

NJJay · December 22, 2024, 3:21am

Did you ever fix this? I am getting the same error?