my input text is not responding on the vertical stack text example I tried to recreate from the wiki. I only get “response.response_text” for the text field.
my yaml:
alias: LLM doorbell
sequence:
- action: llmvision.image_analyzer
metadata: {}
data:
provider: OpenAI
model: gpt-4o-mini
detail: low
max_tokens: 60
temperature: 0.5
message: Describe what you see. Be short and brief.
image_entity:
- camera.ring_front_doorbell_snapshot
response_variable: response
- action: input_text.set_value
metadata: {}
data:
value: "{{response.response_text}}"
target:
entity_id: input_text.llmvision_response
- action: notify.mobile_app_luffy1987
data:
data:
image:url
entity_id: camera.ring_doorbell_camera
actions:
- action: URI
title: View Doorbell
uri: /dashboard-mobile/security
title: Someone is at the Front Door!
message: "{{response.response_text}}"
description: ""
icon: phu:ring
You can use an input_text helper to store the variable prompt. In your automation, every time a person with a white shirt has been detected, you could change the text_input helper to the new prompt with instructions to look for a blue shirt.
v1.2 Stream Analyzer and new Provider Configurations
Breaking Changes
Unfortunately, due to the changes in how provider configurations are stored, providers may have to be set up again!
provider now requires a config entry. Use the UI to pick your provider configuration!
include_filename is now a required parameter. Make sure you include it in all your scripts and automations!
Changelog
v1.2 adds stream_analyzer: It records for a set duration and analyzes frames in a given interval (much like video_analyzer). This is faster as it avoids writing the files to disk and reading them again.
The setup has been rewritten: You can now have multiple configurations for each provider and a restart is no longer necessary to delete configurations.
This could be useful if you host multiple models on different servers (e.g. a small LLM on a Raspberry Pi and a larger model on a PC)
Not all provider configurations will migrate automatically, so you may have to do the setup again.
Writing scripts and automations can be intimidating.
To make it easier to get started with LLM Vision, I have gathered some useful automations using LLM Vision and converted into blueprints so you can easily import and customize them for your needs.
For example, everyone using Frigate can now easily get event notifications summarized:
You can find this and other blueprints in the examples wiki with more to come. If you have an automation, script or blueprint you’d like to share, please post them here and I’ll add them to the wiki!
i keep getting this error 500.
i can press run again and it works, but still get this error every now and again
Executed: 21 October 2024 at 09:15:45
Error: Fetch failed with status code 500
Result:
params:
domain: llmvision
service: stream_analyzer
service_data:
interval: 2
duration: 1
include_filename: false
target_width: 1280
detail: high
max_tokens: 100
temperature: 0.1
provider: 01JAP03NNG4W9J7Y7J6QP7QSD4
model: gpt-4o-mini
message: describe what you see in a few words
image_entity:
- camera.camera1
target: {}
running_script: false
The error occurs when trying to fetch the latest frame recorded by the camera.
Here’s something you can try: In Home Assistant go to developer tools > States. Search for your camera and check its attributes. It should have entity_picture. Copy this and append to your Home Assistant url (e.g. http://homeassistant.local:8123/api/camera.entity_id?token=<accesstoken>
This should show you the latest camera frame in your browser. Try refreshing a few times. Will this also produce a 500 error?
ok yeah, so its only on the first time it tries to basically load. i can keep refreshing and its fine but if i leave a few seconds i get the error 500 again
This means Home Assistant doesn’t keep the stream running and it takes too long to load it. You can force the camera stream to stay active in the background. This will however take up some resources on the server. Here’s how you can do that:
Find your camera entity in Settings Devices & services > Entities
In the camera preview, click the settings cog
There should be a setting called ‘preload camera stream’
With v1.2.1 the blueprint has gotten some big upgrades:
AI understands what happens in the video, decides whether you should be notified and sends you notifications with a preview and summary of what happened.
Using LLM Vision has never been easier!
I’m trying to configure the llm vision component with Anthropic. I’m getting an invalid key error during setup. The key starts with “sk-ant-api03…”. I’m on the evaluation plan. Do I have to purchase a plan to use this?
Just realized that my remaining balance is 0.00. But the key should still work or not?
Thanks for this project. I’ve been having some fun with it.
I have an automation where at night when the alarm is armed it would wake me up if a person was detected on the driveway. However this was always a pain as rain or even reflections would trigger it even though I have blueiris setup nicely. Could never get it to work properly.
Now I have it setup so that if blueiris thinks there is a person, it will ping the llm to double check before waking me up to let me know. Works 100% more reliably.
Breaking Changes
