Say hello to intelligent security event notifications!
Previously, camera notifications were automated based on object detection and detection zones. Now, with multimodal AI interpreting your camera feed, the possibilities for notification automation are limitless. This enables far more intelligent and context-aware notifications. And it’s easier than ever!
AI understands what happens in the video, decides whether you should be notified and sends you notifications with a preview and summary of what happened.
There are two modes: ‘Frigate’ and ‘Camera’. ‘Frigate’ requires Frigate and mqtt to be installed. ‘Camera’ works with any camera entity and will trigger based on a customizable state (e.g. recording).
Customize your notification preferences by choosing to receive all updates or utilizing AI to filter for important events only. This can be configured using the important parameter. Using AI will add a slight delay to your notifications (<5 seconds when using OpenAI).
When an event is detected, you’ll receive a notification with basic information including the camera name. Event summarization continues in the background and the notification is updated once the summary is available.
The Automation
Trigger
(Frigate) A new mqtt message posted in frigate/events
(Camera) A camera entity changes to trigger_state
Actions
Decide if the event is important (only when important is enabled)
Send a notification with a preview of the event or live preview of the camera that triggered the event
Process the event an generate a summary. The summary is based on a prompt and you can customize it.
Thanks for this, could you add an external trigger, e.g. a motion sensor binary sensor entity as my camera entities (Unifi Protect) don’t change state when motion detected, they are constantly recording.
I am getting the following error when running Ollama with llava model. I upgraded to the most recent LLM vision, deleting my old provider and re-adding
Detected blocking call to open with args (‘/config/custom_components/llmvision/tmp_frames/frame0006.jpg’, ‘rb’) inside the event loop by custom integration ‘llmvision’ at custom_components/llmvision/media_handlers.py, line 239: with Image.open(frame_path) as img: (offender: /usr/local/lib/python3.12/site-packages/PIL/Image.py, line 3277: fp = builtins.open(filename, “rb”)), please create a bug report at Issues · valentinfrlch/ha-llmvision · GitHub For developers, please see Blocking operations with asyncio | Home Assistant Developer Docs Traceback (most recent call last): File “”, line 198, in _run_module_as_main File “”, line 88, in _run_code File “/usr/src/homeassistant/homeassistant/main.py”, line 223, in sys.exit(main()) File “/usr/src/homeassistant/homeassistant/main.py”, line 209, in main exit_code = runner.run(runtime_conf) File “/usr/src/homeassistant/homeassistant/runner.py”, line 190, in run return loop.run_until_complete(setup_and_run_hass(runtime_config)) File “/usr/local/lib/python3.12/asyncio/base_events.py”, line 674, in run_until_complete self.run_forever() File “/usr/local/lib/python3.12/asyncio/base_events.py”, line 641, in run_forever self._run_once() File “/usr/local/lib/python3.12/asyncio/base_events.py”, line 1990, in _run_once handle._run() File “/usr/local/lib/python3.12/asyncio/events.py”, line 88, in _run self._context.run(self._callback, *self._args) File “/usr/src/homeassistant/homeassistant/core.py”, line 2731, in async_call response_data = await coro File “/usr/src/homeassistant/homeassistant/core.py”, line 2774, in _execute_service return await target(service_call) File “/config/custom_components/llmvision/init.py”, line 178, in video_analyzer client = await processor.add_videos(video_paths=call.video_paths, File “/config/custom_components/llmvision/media_handlers.py”, line 239, in add_videos with Image.open(frame_path) as img:
Are you running frigate and home assistant on the same machine?
If you’re running the frigate addon your hostname is different. See this table: Home Assistant Integration | Frigate
Hmm still not working. Nothing in the logs, but just saying “Person seen” and name of camera. However, the name provided is not the camera entity I selected
Are you using frigate or the camera mode? For frigate it will use all cameras you have added in frigate. Camera entities in the blueprint only apply if you use the camera mode.
Camera mode probably wont work in your case if you have the camera entities from frigate. At least for me those always have the state ‘recording’, no matter if they detect something or not. That way the automation never gets triggered. Feel free to create an issue on github. Frigate mode should work!
Ok yeah camera mode didn’t work. I’ll try Frigate again and post the errors
Error 1
This error originated from a custom integration.
Logger: custom_components.llmvision.request_handlers
Source: custom_components/llmvision/request_handlers.py:428
integration: LLM Vision (documentation, issues)
First occurred: 11:31:41 AM (2 occurrences)
Last logged: 11:31:42 AM
Couldn’t fetch frame (status code: 404)
Error 2
This error originated from a custom integration.
Logger: custom_components.llmvision.request_handlers
Source: custom_components/llmvision/request_handlers.py:438
integration: LLM Vision (documentation, issues)
First occurred: 11:31:43 AM (1 occurrences)
Last logged: 11:31:43 AM
Failed to fetch https://xxx.duckdns.org/api/frigate/notifications/1729611091.380678-x6tur7/clip.mp4 after 2 retries
Error 3
Logger: homeassistant.components.automation.ai_event_summary_porch
Source: helpers/script.py:2002
integration: Automation (documentation, issues)
First occurred: 11:31:43 AM (2 occurrences)
Last logged: 11:31:43 AM
AI Event Summary (Porch): Analyze event: choice 1: Error executing script. Error for call_service at pos 1: Failed to fetch frigate clip 1729611091.380678-x6tur7
AI Event Summary (Porch): Error executing script. Error for choose at pos 4: Failed to fetch frigate clip 1729611091.380678-x6tur7
There is a problem with your frigate_url (indicated by the 404 error). Not sure what your setup is, but I think there should be a port. When you open the url in the logs in you browser you should see the clip.
It’s probably docker, https should work correctly. Not a docker expert myself, but if you run frigate and home assistant both on the same docker network or the same stack, the hostname of frigate in home assistant should be whatever the name of the frigate container is. So your frigate_url might be something like http://frigate:5000. Does someone here have a similar setup and can help with this?
Maybe this can help: Networking | Docker Docs
