Guide for CCTV Snapshot on motion, send to Google Generative AI & get notification with description & snapshot

baudneo · November 28, 2024, 9:27pm

googleAI.yaml
alias: Camera 1 - Snapshot, AI & Notification
sequence:
  - metadata: {}
    data:
      filename: ./www/snapshots/driveway1_snapshot1.jpg
    target:
      device_id: d8ccb88c128b0b47abd14a9a651f49a3
    enabled: true
    action: camera.snapshot
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
    enabled: true
  - metadata: {}
    data:
      filename: ./www/snapshots/driveway1_snapshot2.jpg
    target:
      device_id: d8ccb88c128b0b47abd14a9a651f49a3
    enabled: true
    action: camera.snapshot
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
    enabled: true
  - metadata: {}
    data:
      filename: ./www/snapshots/driveway1_snapshot3.jpg
    target:
      device_id: d8ccb88c128b0b47abd14a9a651f49a3
    enabled: true
    action: camera.snapshot
  - metadata: {}
    data:
      prompt: >-
        Motion has been detected, compare and very briefly describe what you see
        in the following sequence of images from my driveway camera number 1.
        What do you think caused the motion alarm? If a person or car is
        present, describe them in detail. Do not describe stationary objects or
        buildings. If you see no obvious causes of motion, reply with "No
        Obvious Motion Detected." Your message needs to be short enough to fit
        in a phone notification. 
      image_filename:
        - ./www/snapshots/driveway1_snapshot1.jpg
        - ./www/snapshots/driveway1_snapshot2.jpg
        - ./www/snapshots/driveway1_snapshot3.jpg
    response_variable: generated_content
    action: google_generative_ai_conversation.generate_content
  - if:
      - condition: template
        value_template: "{{ 'No Obvious Motion Detected.' in generated_content.text }}"
    then:
      - stop: ""
    else:
      - action: telegram_bot.send_photo
        data:
        file: /config/www/img/frigate/ai_parkering_2.jpg
        caption: >-
          {{ generated_content.text }} snap_id:
          {{trigger.payload_json["after"]["id"]}}
        title: PARKERING AI
        data:
          message_tag: "{{trigger.payload_json[\"after\"][\"start_time\"]|int}}"

mode: single

I’m doing this on mobile so indentation may be wrong. Test and if it doesn’t work, when I’m home I can do proper formatting. You should get the gist of the idea though. You need to define a action to call the service and pass it the correct data.

TrOjAn · November 28, 2024, 9:31pm

While Ive been sitting here I have it 80% working, Ill post my latest attempt up, its running without error, parsing the 3 photos and then forwarding one photo to Telegram BUT*** I no lo nger have the Google AI text but I cant be far away,

Im using the Visual Code editor which is now highlighting any formatting errors so Im learning as I go …

I cant see if its actually going to Google AI as I cant get debugging to work…

Thanks for looking at this, Terry

alias: Telegram Test
sequence:
  - metadata: {}
    data:
      filename: ./www/snapshots/garden_snapshot1.jpg
    target:
      entity_id: camera.garden_trackmix_fluent
    enabled: true
    action: camera.snapshot
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
    enabled: true
  - metadata: {}
    data:
      filename: ./www/snapshots/garden_snapshot2.jpg
    target:
      device_id: 7497f7c8e839a8c919a947fea2969f00
    enabled: true
    action: camera.snapshot
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
    enabled: true
  - metadata: {}
    data:
      filename: ./www/snapshots/garden_snapshot3.jpg
    target:
      device_id: 7497f7c8e839a8c919a947fea2969f00
    enabled: true
    action: camera.snapshot
  - metadata: {}
    data:
      prompt: >-
        Motion has been detected, compare and very briefly describe what you see
        in the following sequence of images from my driveway camera. What do you
        think caused the motion alarm? If a person or car is present, describe
        them in detail. Do not describe stationary objects or buildings. If you
        see no obvious causes of motion, reply with "No Obvious Motion Detected."
        Your message needs to be short enough to fit in a phone notification.
      image_filename:
        - ./www/snapshots/garden_snapshot1.jpg
        - ./www/snapshots/garden_snapshot2.jpg
        - ./www/snapshots/garden_snapshot3.jpg
    response_variable: generated_content
    action: google_generative_ai_conversation.generate_content
  - if:
      - condition: template
        value_template: "{{ 'No Obvious Motion Detected.' in generated_content.text }}"
    then:
      - stop: ""
    else:
  - metadata: {}
    action: telegram_bot.send_photo
    data:
      title: Garden Motion Detected
      message: "{{generated_content['text'] }}"
      data:
      authentication: digest
      file: /config/www/snapshots/garden_snapshot2.jpg
mode: single

TrOjAn · November 28, 2024, 9:56pm

@baudneo

Ive fixed it !!!

You actually helped with your code so thank you…

Cheers buddy

theduusK · November 29, 2024, 7:56am

Hey,

i’m trying to setup the second script as described, but I’m running into following error:

Message malformed: extra keys not allowed @ data['sequence']

alias: Camera - Driveway 1 - Snapshot, AI & Notification
sequence:
  - variables:
      trig_ts: "{{ now().strftime('%Y%m%d-%H%M%S') }}"
  - service: camera.snapshot
    metadata: {}
    data:
      filename: ./www/snapshots/door_{{trig_ts}}_snapshot1.jpg
    target:
      device_id: 423c54b976c3a55b3768edf3062b29c3
    enabled: true
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
    enabled: true
  - service: camera.snapshot
    metadata: {}
    data:
      filename: ./www/snapshots/door_{{trig_ts}}_snapshot2.jpg
    target:
      device_id: 423c54b976c3a55b3768edf3062b29c3
    enabled: true
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
    enabled: true
  - service: camera.snapshot
    metadata: {}
    data:
      filename: ./www/snapshots/door_{{trig_ts}}_snapshot3.jpg
    target:
      device_id: 423c54b976c3a55b3768edf3062b29c3
    enabled: true
  - service: google_generative_ai_conversation.generate_content
    metadata: {}
    data:
      prompt: >-
        Motion has been detected, compare and very briefly describe what you see
        in the following sequence of images from my driveway camera number 1.
        What do you think caused the motion alarm? If a person or car is
        present, describe them in detail. Do not describe stationary objects or
        buildings. If you see no obvious causes of motion, reply with "No
        Obvious Motion Detected." Your message needs to be short enough to fit
        in a phone notification. 
      image_filename:
        - ./www/snapshots/door_{{trig_ts}}_snapshot1.jpg
        - ./www/snapshots/door_{{trig_ts}}_snapshot2.jpg
        - ./www/snapshots/door_{{trig_ts}}_snapshot3.jpg
    response_variable: generated_content
  - if:
      - condition: template
        value_template: "{{ 'No Obvious Motion Detected.' in generated_content.text }}"
    then:
      - stop: ""
    else:
      - service: notify.mobile_app_iphone_patrick
        metadata: {}
        data:
          title: Driveway 1 Motion Detected
          message: "{{generated_content['text'] }}"
          data:
            image: /local/snapshots/door_{{trig_ts}}_snapshot2.jpg
mode: single

Can somebody help me out please?

Minglarn · November 29, 2024, 8:02am

Message malformed: extra keys not allowed @ data['sequence']

I dont know if this is allowed…

alias: Camera - Driveway 1 - Snapshot, AI & Notification
sequence:
  - variables:
      trig_ts: "{{ now().strftime('%Y%m%d-%H%M%S') }}"
  - service: camera.snapshot

theduusK · November 29, 2024, 8:11am

Same error happens also without the defined variable.

TrOjAn · November 29, 2024, 2:28pm

Fist thing I can advise is use the Visual Code Editor in HA as that will highlight all the errors just in spacing / indentation, its a whole new world…

Terry

eddomak · December 6, 2024, 7:57am

Firstly, thankyou to fellow Aussie @Aussie_Adam for starting this guide and really inspiring me to do this project for myself. Thankyou to everyone who has contributed to this thread as I’ve used bits and pieces of it to refine my own script as I’ve gained an understanding of what each part does, in particular thanks for alerting me to the cached image issue when using Cloudflared, and the need to timestamp my snapshots, and thanks for @BambamNZ for mentioning the “Clear” stream that available on Reolink cameras.

In the interest of sharing and hopefully helping out, here’s a couple of my own personal tips (as a HA native automation and script newbie) I’ve picked up along the way:

I would start off with just doing the SCRIPT component first and running it from the menu to make sure that the SCRIPT can run standalone.
For those (like myself) that don’t use the HA native AUTOMATION this then means no need to implement the AUTOMATION. I use NodeRED and just call the SCRIPT from there.
If you’re using NodeRED and want to purge the snapshot directory periodically, you can just use the INJECT node and set the interval (say hourly) to trigger an EXEC node with the command:

rm -f /homeassistant/www/snapshots/*.jpg

I found the HA Companion App with its tiny snapshot and instantly disappearing nature very unfriendly for my use case, so I (as a non-Signal/Telegram user) I implemented the notification to go via Whatsapp to a group chat (where I can add or remove members as need be).

a) I did this via GreenAPI and there is a simple guide that took around 10 minutes here: Unlock WhatsApp Messaging in Home Assistant with Green API Custom Notifier | by Tomer Klein | Medium.

b) If you link the API step with your personal phone number you won’t get notified (notification bar, or sounds) of new messages on the chat, so you may want to buy a cheap long life SIM (since it will never make calls)

c) So that you don’t fill your phone storage, as the admin of the group you can set the messages to disappear by default after a set period (say 7 days) and you can also mark individual messages to be kept regardless of the default.

d) You may have troubles finding the right path reference for the snapshot - the following YAML may be useful to you in your SCRIPT:

action: notify.greenapi
metadata: {}
data:
title: Rear Verandah Movement Detected
target: [email protected]
data:
file: ./www/snapshots/camera_snapshot2.jpg
message: “{{ generated_content[‘text’] }}”

I hope this helps someone out there and/or inspires someone.
Cheers!

Licky · December 8, 2024, 9:59am

Possible to post your full code? Still struggling to get this working

rog889 · December 16, 2024, 2:02am

First, thanks for your great script! Is it possible I can notify multiple iOS devices at the same time? If yes, what code do I need to add?

Thank you!

Deneysvd · December 18, 2024, 2:33pm

I am getting the following error, but stangely it works sometimes and sometimes not.

Failed to perform the action script/camera_1_snapshot_ai_notification. Error generating content: 400 Unable to submit request because it has an empty inlineData parameter. Add a value to the parameter and try again. Learn more: Generate content with the Gemini Enterprise API | Generative AI on Vertex AI | Google Cloud

I am using Eufy cameras, they are battery operated, so most of the time they are in “Idle” mode. I fixed this by adding a “turn on the camera” demand in the script. Not sure if this is where i made a mistake.

alias: Camera - Driveway 1 - Snapshot, AI & Notification
sequence:
  - device_id: 60c65211d88499ac422cf9028ab14706
    domain: button
    entity_id: bfaf863c1d8e0fca3fb68a5db3c1df30
    type: press
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
    enabled: true
  - metadata: {}
    data:
      filename: ./www/snapshots/driveway1_snapshot1.jpg
    target:
      device_id: 60c65211d88499ac422cf9028ab14706
    enabled: true
    action: camera.snapshot
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 950
    enabled: true
  - metadata: {}
    data:
      filename: ./www/snapshots/driveway1_snapshot2.jpg
    target:
      device_id: 60c65211d88499ac422cf9028ab14706
    enabled: true
    action: camera.snapshot
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 950
    enabled: true
  - metadata: {}
    data:
      filename: ./www/snapshots/driveway1_snapshot3.jpg
    target:
      device_id: 60c65211d88499ac422cf9028ab14706
    enabled: true
    action: camera.snapshot
  - metadata: {}
    data:
      prompt: >-
        Motion has been detected, compare and very briefly describe what you see
        in the following sequence of images from my driveway camera number 1.
        What do you think caused the motion alarm? If a person or car is
        present, describe them in detail. Do not describe stationary objects or
        buildings. If you see no obvious causes of motion, reply with "Camera
        has detected motion however no obvious motion observed comparing
        snapshots". Your message needs to be short enough to fit in a phone
        notification. 
      image_filename:
        - ./www/snapshots/driveway1_snapshot1.jpg
        - ./www/snapshots/driveway1_snapshot2.jpg
        - ./www/snapshots/driveway1_snapshot3.jpg
    response_variable: generated_content
    action: google_generative_ai_conversation.generate_content
  - if:
      - condition: template
        value_template: >
          {{ generated_content['text'] == ' Camera has detected motion however
          no obvious motion observed comparing snapshots.' }} 
    then:
      - stop: ""
    else:
      - metadata: {}
        data:
          title: Driveway 1 Motion Detected
          message: "{{generated_content['text'] }}"
          data:
            image: /local/snapshots/driveway1_snapshot2.jpg
        action: notify.notify
  - device_id: 60c65211d88499ac422cf9028ab14706
    domain: button
    entity_id: 9c3ef3d90bf4d446c567a4df0d8b04f3
    type: press
mode: single

And this is what i get when running a trace:

JerryAC · December 24, 2024, 12:52am

Where do I locate the Device ID for my Reolink cameras? Clearly somewhere right in front of me but I’m missing it.

BambamNZ · December 25, 2024, 10:33pm

Honestly don’t know where to get that, it must be buried in the internal DB somewhere, you can use the the “entity_id” on the automation, don’t need the device_id

Using the entity state - results in a more human readable config

conditions:
  - condition: state
    entity_id: input_boolean.motion_detection_paused
    state: "off"
  - condition: state
    entity_id: cover.garage_door_door
    state: closed
  - condition: state
    entity_id: lock.front_door_lock
    state: locked

actions:
  - metadata: {}
    data:
      filename: /config/www/snapshot/{{ trigger.from_state.name }}_1.jpg
    target:
      entity_id:
        - camera.driveway_snapshots_clear
    action: camera.snapshot
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500

chris.huitema · December 26, 2024, 1:20am

I am just about to give this another go, i couldn’t get it to work a couple of months ago, just kept getting useless information about the images.

I wonder if sending a base image with nothing in it would help? and doing something like “with reference to image1 what has changed and what do you see” or even an image from a previous detection?

baudneo · December 26, 2024, 1:38am

That’s my plan, but in my testing, the images are received by Gemini out of order. You would need to annotate the image to give it “1 of x” and also annotate the base image.

I tried using timestamps from the cameras to achieve this but Gemini was hit and miss with getting the order right due to not reading the correct timestamp.

It still works very well, but it could be much better with more context like drawing polygons for zones in the base image and such.

chris.huitema · December 26, 2024, 1:42am

I remember why i gave up now… its because i use blue iris for recording and the cameras take a long time to load. the image i use for notifications currently is via MQTT from Blue iris, and i couldn’t find a way to get blue iris to send a few images easily…
this is maybe where i got the idea from in the first place, just to compare a base image to the latest notification image and describe the changes/differences between the images.

baudneo · December 26, 2024, 3:45pm

I use zoneminder and have custom bash scripts to grab images. Maybe BI also has some API or http endpoints setup to pull images from?