Guide for CCTV Snapshot on motion, send to Google Generative AI & get notification with description & snapshot

googleAI.yaml
alias: Camera 1 - Snapshot, AI & Notification
sequence:
  - metadata: {}
    data:
      filename: ./www/snapshots/driveway1_snapshot1.jpg
    target:
      device_id: d8ccb88c128b0b47abd14a9a651f49a3
    enabled: true
    action: camera.snapshot
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
    enabled: true
  - metadata: {}
    data:
      filename: ./www/snapshots/driveway1_snapshot2.jpg
    target:
      device_id: d8ccb88c128b0b47abd14a9a651f49a3
    enabled: true
    action: camera.snapshot
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
    enabled: true
  - metadata: {}
    data:
      filename: ./www/snapshots/driveway1_snapshot3.jpg
    target:
      device_id: d8ccb88c128b0b47abd14a9a651f49a3
    enabled: true
    action: camera.snapshot
  - metadata: {}
    data:
      prompt: >-
        Motion has been detected, compare and very briefly describe what you see
        in the following sequence of images from my driveway camera number 1.
        What do you think caused the motion alarm? If a person or car is
        present, describe them in detail. Do not describe stationary objects or
        buildings. If you see no obvious causes of motion, reply with "No
        Obvious Motion Detected." Your message needs to be short enough to fit
        in a phone notification. 
      image_filename:
        - ./www/snapshots/driveway1_snapshot1.jpg
        - ./www/snapshots/driveway1_snapshot2.jpg
        - ./www/snapshots/driveway1_snapshot3.jpg
    response_variable: generated_content
    action: google_generative_ai_conversation.generate_content
  - if:
      - condition: template
        value_template: "{{ 'No Obvious Motion Detected.' in generated_content.text }}"
    then:
      - stop: ""
    else:
      - action: telegram_bot.send_photo
        data:
        file: /config/www/img/frigate/ai_parkering_2.jpg
        caption: >-
          {{ generated_content.text }} snap_id:
          {{trigger.payload_json["after"]["id"]}}
        title: PARKERING AI
        data:
          message_tag: "{{trigger.payload_json[\"after\"][\"start_time\"]|int}}"

mode: single

Iā€™m doing this on mobile so indentation may be wrong. Test and if it doesnā€™t work, when Iā€™m home I can do proper formatting. You should get the gist of the idea though. You need to define a action to call the service and pass it the correct data.

While Ive been sitting here I have it 80% working, Ill post my latest attempt up, its running without error, parsing the 3 photos and then forwarding one photo to Telegram BUT*** I no lo nger have the Google AI text but I cant be far away,

Im using the Visual Code editor which is now highlighting any formatting errors so Im learning as I go ā€¦

I cant see if its actually going to Google AI as I cant get debugging to workā€¦

Thanks for looking at this, Terry

alias: Telegram Test
sequence:
  - metadata: {}
    data:
      filename: ./www/snapshots/garden_snapshot1.jpg
    target:
      entity_id: camera.garden_trackmix_fluent
    enabled: true
    action: camera.snapshot
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
    enabled: true
  - metadata: {}
    data:
      filename: ./www/snapshots/garden_snapshot2.jpg
    target:
      device_id: 7497f7c8e839a8c919a947fea2969f00
    enabled: true
    action: camera.snapshot
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
    enabled: true
  - metadata: {}
    data:
      filename: ./www/snapshots/garden_snapshot3.jpg
    target:
      device_id: 7497f7c8e839a8c919a947fea2969f00
    enabled: true
    action: camera.snapshot
  - metadata: {}
    data:
      prompt: >-
        Motion has been detected, compare and very briefly describe what you see
        in the following sequence of images from my driveway camera. What do you
        think caused the motion alarm? If a person or car is present, describe
        them in detail. Do not describe stationary objects or buildings. If you
        see no obvious causes of motion, reply with "No Obvious Motion Detected."
        Your message needs to be short enough to fit in a phone notification.
      image_filename:
        - ./www/snapshots/garden_snapshot1.jpg
        - ./www/snapshots/garden_snapshot2.jpg
        - ./www/snapshots/garden_snapshot3.jpg
    response_variable: generated_content
    action: google_generative_ai_conversation.generate_content
  - if:
      - condition: template
        value_template: "{{ 'No Obvious Motion Detected.' in generated_content.text }}"
    then:
      - stop: ""
    else:
  - metadata: {}
    action: telegram_bot.send_photo
    data:
      title: Garden Motion Detected
      message: "{{generated_content['text'] }}"
      data:
      authentication: digest
      file: /config/www/snapshots/garden_snapshot2.jpg
mode: single

@baudneo

Ive fixed it !!!

You actually helped with your code so thank youā€¦

Cheers buddy :+1:

Hey,

iā€™m trying to setup the second script as described, but Iā€™m running into following error:

Message malformed: extra keys not allowed @ data['sequence']
alias: Camera - Driveway 1 - Snapshot, AI & Notification
sequence:
  - variables:
      trig_ts: "{{ now().strftime('%Y%m%d-%H%M%S') }}"
  - service: camera.snapshot
    metadata: {}
    data:
      filename: ./www/snapshots/door_{{trig_ts}}_snapshot1.jpg
    target:
      device_id: 423c54b976c3a55b3768edf3062b29c3
    enabled: true
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
    enabled: true
  - service: camera.snapshot
    metadata: {}
    data:
      filename: ./www/snapshots/door_{{trig_ts}}_snapshot2.jpg
    target:
      device_id: 423c54b976c3a55b3768edf3062b29c3
    enabled: true
  - delay:
      hours: 0
      minutes: 0
      seconds: 0
      milliseconds: 500
    enabled: true
  - service: camera.snapshot
    metadata: {}
    data:
      filename: ./www/snapshots/door_{{trig_ts}}_snapshot3.jpg
    target:
      device_id: 423c54b976c3a55b3768edf3062b29c3
    enabled: true
  - service: google_generative_ai_conversation.generate_content
    metadata: {}
    data:
      prompt: >-
        Motion has been detected, compare and very briefly describe what you see
        in the following sequence of images from my driveway camera number 1.
        What do you think caused the motion alarm? If a person or car is
        present, describe them in detail. Do not describe stationary objects or
        buildings. If you see no obvious causes of motion, reply with "No
        Obvious Motion Detected." Your message needs to be short enough to fit
        in a phone notification. 
      image_filename:
        - ./www/snapshots/door_{{trig_ts}}_snapshot1.jpg
        - ./www/snapshots/door_{{trig_ts}}_snapshot2.jpg
        - ./www/snapshots/door_{{trig_ts}}_snapshot3.jpg
    response_variable: generated_content
  - if:
      - condition: template
        value_template: "{{ 'No Obvious Motion Detected.' in generated_content.text }}"
    then:
      - stop: ""
    else:
      - service: notify.mobile_app_iphone_patrick
        metadata: {}
        data:
          title: Driveway 1 Motion Detected
          message: "{{generated_content['text'] }}"
          data:
            image: /local/snapshots/door_{{trig_ts}}_snapshot2.jpg
mode: single

Can somebody help me out please?

Message malformed: extra keys not allowed @ data['sequence']

I dont know if this is allowedā€¦

alias: Camera - Driveway 1 - Snapshot, AI & Notification
sequence:
  - variables:
      trig_ts: "{{ now().strftime('%Y%m%d-%H%M%S') }}"
  - service: camera.snapshot

Same error happens also without the defined variable.

Fist thing I can advise is use the Visual Code Editor in HA as that will highlight all the errors just in spacing / indentation, its a whole new worldā€¦

Terry

Firstly, thankyou to fellow Aussie @Aussie_Adam for starting this guide and really inspiring me to do this project for myself. Thankyou to everyone who has contributed to this thread as Iā€™ve used bits and pieces of it to refine my own script as Iā€™ve gained an understanding of what each part does, in particular thanks for alerting me to the cached image issue when using Cloudflared, and the need to timestamp my snapshots, and thanks for @BambamNZ for mentioning the ā€œClearā€ stream that available on Reolink cameras.

In the interest of sharing and hopefully helping out, hereā€™s a couple of my own personal tips (as a HA native automation and script newbie) Iā€™ve picked up along the way:

  1. I would start off with just doing the SCRIPT component first and running it from the menu to make sure that the SCRIPT can run standalone.

  2. For those (like myself) that donā€™t use the HA native AUTOMATION this then means no need to implement the AUTOMATION. I use NodeRED and just call the SCRIPT from there.

  3. If youā€™re using NodeRED and want to purge the snapshot directory periodically, you can just use the INJECT node and set the interval (say hourly) to trigger an EXEC node with the command:

rm -f /homeassistant/www/snapshots/*.jpg

  1. I found the HA Companion App with its tiny snapshot and instantly disappearing nature very unfriendly for my use case, so I (as a non-Signal/Telegram user) I implemented the notification to go via Whatsapp to a group chat (where I can add or remove members as need be).

    a) I did this via GreenAPI and there is a simple guide that took around 10 minutes here: Unlock WhatsApp Messaging in Home Assistant with Green API Custom Notifier | by Tomer Klein | Medium.

    b) If you link the API step with your personal phone number you wonā€™t get notified (notification bar, or sounds) of new messages on the chat, so you may want to buy a cheap long life SIM (since it will never make calls)

    c) So that you donā€™t fill your phone storage, as the admin of the group you can set the messages to disappear by default after a set period (say 7 days) and you can also mark individual messages to be kept regardless of the default.

    d) You may have troubles finding the right path reference for the snapshot - the following YAML may be useful to you in your SCRIPT:

action: notify.greenapi
metadata: {}
data:
title: Rear Verandah Movement Detected
target: [email protected]
data:
file: ./www/snapshots/camera_snapshot2.jpg
message: ā€œ{{ generated_content[ā€˜textā€™] }}ā€

I hope this helps someone out there and/or inspires someone.
Cheers!

1 Like