googleAI.yaml
alias: Camera 1 - Snapshot, AI & Notification
sequence:
- metadata: {}
data:
filename: ./www/snapshots/driveway1_snapshot1.jpg
target:
device_id: d8ccb88c128b0b47abd14a9a651f49a3
enabled: true
action: camera.snapshot
- delay:
hours: 0
minutes: 0
seconds: 0
milliseconds: 500
enabled: true
- metadata: {}
data:
filename: ./www/snapshots/driveway1_snapshot2.jpg
target:
device_id: d8ccb88c128b0b47abd14a9a651f49a3
enabled: true
action: camera.snapshot
- delay:
hours: 0
minutes: 0
seconds: 0
milliseconds: 500
enabled: true
- metadata: {}
data:
filename: ./www/snapshots/driveway1_snapshot3.jpg
target:
device_id: d8ccb88c128b0b47abd14a9a651f49a3
enabled: true
action: camera.snapshot
- metadata: {}
data:
prompt: >-
Motion has been detected, compare and very briefly describe what you see
in the following sequence of images from my driveway camera number 1.
What do you think caused the motion alarm? If a person or car is
present, describe them in detail. Do not describe stationary objects or
buildings. If you see no obvious causes of motion, reply with "No
Obvious Motion Detected." Your message needs to be short enough to fit
in a phone notification.
image_filename:
- ./www/snapshots/driveway1_snapshot1.jpg
- ./www/snapshots/driveway1_snapshot2.jpg
- ./www/snapshots/driveway1_snapshot3.jpg
response_variable: generated_content
action: google_generative_ai_conversation.generate_content
- if:
- condition: template
value_template: "{{ 'No Obvious Motion Detected.' in generated_content.text }}"
then:
- stop: ""
else:
- action: telegram_bot.send_photo
data:
file: /config/www/img/frigate/ai_parkering_2.jpg
caption: >-
{{ generated_content.text }} snap_id:
{{trigger.payload_json["after"]["id"]}}
title: PARKERING AI
data:
message_tag: "{{trigger.payload_json[\"after\"][\"start_time\"]|int}}"
mode: single
Iâm doing this on mobile so indentation may be wrong. Test and if it doesnât work, when Iâm home I can do proper formatting. You should get the gist of the idea though. You need to define a action to call the service and pass it the correct data.
While Ive been sitting here I have it 80% working, Ill post my latest attempt up, its running without error, parsing the 3 photos and then forwarding one photo to Telegram BUT*** I no lo nger have the Google AI text but I cant be far away,
Im using the Visual Code editor which is now highlighting any formatting errors so Im learning as I go âŚ
I cant see if its actually going to Google AI as I cant get debugging to workâŚ
Thanks for looking at this, Terry
alias: Telegram Test
sequence:
- metadata: {}
data:
filename: ./www/snapshots/garden_snapshot1.jpg
target:
entity_id: camera.garden_trackmix_fluent
enabled: true
action: camera.snapshot
- delay:
hours: 0
minutes: 0
seconds: 0
milliseconds: 500
enabled: true
- metadata: {}
data:
filename: ./www/snapshots/garden_snapshot2.jpg
target:
device_id: 7497f7c8e839a8c919a947fea2969f00
enabled: true
action: camera.snapshot
- delay:
hours: 0
minutes: 0
seconds: 0
milliseconds: 500
enabled: true
- metadata: {}
data:
filename: ./www/snapshots/garden_snapshot3.jpg
target:
device_id: 7497f7c8e839a8c919a947fea2969f00
enabled: true
action: camera.snapshot
- metadata: {}
data:
prompt: >-
Motion has been detected, compare and very briefly describe what you see
in the following sequence of images from my driveway camera. What do you
think caused the motion alarm? If a person or car is present, describe
them in detail. Do not describe stationary objects or buildings. If you
see no obvious causes of motion, reply with "No Obvious Motion Detected."
Your message needs to be short enough to fit in a phone notification.
image_filename:
- ./www/snapshots/garden_snapshot1.jpg
- ./www/snapshots/garden_snapshot2.jpg
- ./www/snapshots/garden_snapshot3.jpg
response_variable: generated_content
action: google_generative_ai_conversation.generate_content
- if:
- condition: template
value_template: "{{ 'No Obvious Motion Detected.' in generated_content.text }}"
then:
- stop: ""
else:
- metadata: {}
action: telegram_bot.send_photo
data:
title: Garden Motion Detected
message: "{{generated_content['text'] }}"
data:
authentication: digest
file: /config/www/snapshots/garden_snapshot2.jpg
mode: single
iâm trying to setup the second script as described, but Iâm running into following error:
Message malformed: extra keys not allowed @ data['sequence']
alias: Camera - Driveway 1 - Snapshot, AI & Notification
sequence:
- variables:
trig_ts: "{{ now().strftime('%Y%m%d-%H%M%S') }}"
- service: camera.snapshot
metadata: {}
data:
filename: ./www/snapshots/door_{{trig_ts}}_snapshot1.jpg
target:
device_id: 423c54b976c3a55b3768edf3062b29c3
enabled: true
- delay:
hours: 0
minutes: 0
seconds: 0
milliseconds: 500
enabled: true
- service: camera.snapshot
metadata: {}
data:
filename: ./www/snapshots/door_{{trig_ts}}_snapshot2.jpg
target:
device_id: 423c54b976c3a55b3768edf3062b29c3
enabled: true
- delay:
hours: 0
minutes: 0
seconds: 0
milliseconds: 500
enabled: true
- service: camera.snapshot
metadata: {}
data:
filename: ./www/snapshots/door_{{trig_ts}}_snapshot3.jpg
target:
device_id: 423c54b976c3a55b3768edf3062b29c3
enabled: true
- service: google_generative_ai_conversation.generate_content
metadata: {}
data:
prompt: >-
Motion has been detected, compare and very briefly describe what you see
in the following sequence of images from my driveway camera number 1.
What do you think caused the motion alarm? If a person or car is
present, describe them in detail. Do not describe stationary objects or
buildings. If you see no obvious causes of motion, reply with "No
Obvious Motion Detected." Your message needs to be short enough to fit
in a phone notification.
image_filename:
- ./www/snapshots/door_{{trig_ts}}_snapshot1.jpg
- ./www/snapshots/door_{{trig_ts}}_snapshot2.jpg
- ./www/snapshots/door_{{trig_ts}}_snapshot3.jpg
response_variable: generated_content
- if:
- condition: template
value_template: "{{ 'No Obvious Motion Detected.' in generated_content.text }}"
then:
- stop: ""
else:
- service: notify.mobile_app_iphone_patrick
metadata: {}
data:
title: Driveway 1 Motion Detected
message: "{{generated_content['text'] }}"
data:
image: /local/snapshots/door_{{trig_ts}}_snapshot2.jpg
mode: single
Fist thing I can advise is use the Visual Code Editor in HA as that will highlight all the errors just in spacing / indentation, its a whole new worldâŚ
Firstly, thankyou to fellow Aussie @Aussie_Adam for starting this guide and really inspiring me to do this project for myself. Thankyou to everyone who has contributed to this thread as Iâve used bits and pieces of it to refine my own script as Iâve gained an understanding of what each part does, in particular thanks for alerting me to the cached image issue when using Cloudflared, and the need to timestamp my snapshots, and thanks for @BambamNZ for mentioning the âClearâ stream that available on Reolink cameras.
In the interest of sharing and hopefully helping out, hereâs a couple of my own personal tips (as a HA native automation and script newbie) Iâve picked up along the way:
I would start off with just doing the SCRIPT component first and running it from the menu to make sure that the SCRIPT can run standalone.
For those (like myself) that donât use the HA native AUTOMATION this then means no need to implement the AUTOMATION. I use NodeRED and just call the SCRIPT from there.
If youâre using NodeRED and want to purge the snapshot directory periodically, you can just use the INJECT node and set the interval (say hourly) to trigger an EXEC node with the command:
rm -f /homeassistant/www/snapshots/*.jpg
I found the HA Companion App with its tiny snapshot and instantly disappearing nature very unfriendly for my use case, so I (as a non-Signal/Telegram user) I implemented the notification to go via Whatsapp to a group chat (where I can add or remove members as need be).
b) If you link the API step with your personal phone number you wonât get notified (notification bar, or sounds) of new messages on the chat, so you may want to buy a cheap long life SIM (since it will never make calls)
c) So that you donât fill your phone storage, as the admin of the group you can set the messages to disappear by default after a set period (say 7 days) and you can also mark individual messages to be kept regardless of the default.
d) You may have troubles finding the right path reference for the snapshot - the following YAML may be useful to you in your SCRIPT:
I am using Eufy cameras, they are battery operated, so most of the time they are in âIdleâ mode. I fixed this by adding a âturn on the cameraâ demand in the script. Not sure if this is where i made a mistake.
alias: Camera - Driveway 1 - Snapshot, AI & Notification
sequence:
- device_id: 60c65211d88499ac422cf9028ab14706
domain: button
entity_id: bfaf863c1d8e0fca3fb68a5db3c1df30
type: press
- delay:
hours: 0
minutes: 0
seconds: 0
milliseconds: 500
enabled: true
- metadata: {}
data:
filename: ./www/snapshots/driveway1_snapshot1.jpg
target:
device_id: 60c65211d88499ac422cf9028ab14706
enabled: true
action: camera.snapshot
- delay:
hours: 0
minutes: 0
seconds: 0
milliseconds: 950
enabled: true
- metadata: {}
data:
filename: ./www/snapshots/driveway1_snapshot2.jpg
target:
device_id: 60c65211d88499ac422cf9028ab14706
enabled: true
action: camera.snapshot
- delay:
hours: 0
minutes: 0
seconds: 0
milliseconds: 950
enabled: true
- metadata: {}
data:
filename: ./www/snapshots/driveway1_snapshot3.jpg
target:
device_id: 60c65211d88499ac422cf9028ab14706
enabled: true
action: camera.snapshot
- metadata: {}
data:
prompt: >-
Motion has been detected, compare and very briefly describe what you see
in the following sequence of images from my driveway camera number 1.
What do you think caused the motion alarm? If a person or car is
present, describe them in detail. Do not describe stationary objects or
buildings. If you see no obvious causes of motion, reply with "Camera
has detected motion however no obvious motion observed comparing
snapshots". Your message needs to be short enough to fit in a phone
notification.
image_filename:
- ./www/snapshots/driveway1_snapshot1.jpg
- ./www/snapshots/driveway1_snapshot2.jpg
- ./www/snapshots/driveway1_snapshot3.jpg
response_variable: generated_content
action: google_generative_ai_conversation.generate_content
- if:
- condition: template
value_template: >
{{ generated_content['text'] == ' Camera has detected motion however
no obvious motion observed comparing snapshots.' }}
then:
- stop: ""
else:
- metadata: {}
data:
title: Driveway 1 Motion Detected
message: "{{generated_content['text'] }}"
data:
image: /local/snapshots/driveway1_snapshot2.jpg
action: notify.notify
- device_id: 60c65211d88499ac422cf9028ab14706
domain: button
entity_id: 9c3ef3d90bf4d446c567a4df0d8b04f3
type: press
mode: single
Honestly donât know where to get that, it must be buried in the internal DB somewhere, you can use the the âentity_idâ on the automation, donât need the device_id
Using the entity state - results in a more human readable config
I am just about to give this another go, i couldnât get it to work a couple of months ago, just kept getting useless information about the images.
I wonder if sending a base image with nothing in it would help? and doing something like âwith reference to image1 what has changed and what do you seeâ or even an image from a previous detection?
Thatâs my plan, but in my testing, the images are received by Gemini out of order. You would need to annotate the image to give it â1 of xâ and also annotate the base image.
I tried using timestamps from the cameras to achieve this but Gemini was hit and miss with getting the order right due to not reading the correct timestamp.
It still works very well, but it could be much better with more context like drawing polygons for zones in the base image and such.
I remember why i gave up now⌠its because i use blue iris for recording and the cameras take a long time to load. the image i use for notifications currently is via MQTT from Blue iris, and i couldnât find a way to get blue iris to send a few images easilyâŚ
this is maybe where i got the idea from in the first place, just to compare a base image to the latest notification image and describe the changes/differences between the images.