Guide for CCTV Snapshot on motion, send to Google Generative AI & get notification with description & snapshot

[Aussie_Adam]

Hi all,
A few weeks I posted a rough guide on a couple HA facebook pages giving folks the basics of getting a CCTV camera snapshot taken by HA, sending that to Google AI and having the description and image come back as a notification. Not sure if anyone has posted anything similar on here so here I am…

When I originally posted, the script used one snapshot image and you got alot of false notifications… So now I’ve updated it so it takes 3 snapshots 0.5 seconds apart, sends those 3 images google to compare and if google notices no significant motion, the notification WILL NOT get sent.

This hopefully cuts down on false notifications or ones from trees etc moving! It also helps the AI notice actual movement of things like cars…

This works on Iphone, unsure if the image will come through with the notification on Android.

Anyway, I see lots of others asking about this so I thought I’d share my YAML I use as an easy way to do it, as well as creating the actual snapshot WITHOUT using Frigate or your camera to take snapshots.

Your cameras (and phone notifications) will of course need to be already added in HA and they will need to support motion detection and HA must have an entity to be able to see that motion alarm status…

If so, read on.

All the YAML for both the script and automation live here:
AdamGit69/code-snippets (github.com)

To get it all running follow these instructions:

First up install the integration from here: Google Generative AI - Home Assistant

Once you have that installed, create a directory under your www folder called “snapshots”.

Ensure your camera has motion detection turned on and that HA can see the alarm when motion is detected.

Then create a AUTOMATION named: “Camera 1 - Snapshot on motion” using the following code: code-snippets/cctv_ai_notifications_automation at e963791be142fdf3352f676b5d2cf934c01edb1d · AdamGit69/code-snippets · GitHub

Edit to suit, switching to visual editor will help you find your camera and entity id’s. This automation is to run the below script when HA receives the motion alarm from your camera.

After that create a SCRIPT called: “Camera 1 - Snapshot, AI & Notification” with the following code: code-snippets/cctv_ai_notification_script_UPDATE at ff3b6b884645cb7d37add05513d5b31dcb49f7f4 · AdamGit69/code-snippets · GitHub

BE AWARE: If you change the google prompt, especially the "If you see no obvious causes of motion, reply with “Camera has detected motion however no obvious motion observed comparing snapshots”. part, you will also have to change the IF statement part to suit.

ALSO NOTE: this sends 3 images to google, so if you get a “Error generating content: 400 Request payload size exceeds the limit” error, if you manually run the script, your camera snapshots are too big, and you have to adjust it back to 2 images by removing the last snapshot action, the 0.5 second delay before it and then removing the reference to the 3rd image from under image_filename in the google ai service part…

Phew, I hope that all makes sense!

Happy HA’ing!

[rossk]

Love this use of Ai, thanks for sharing.

Just when I think I have finished a particular project someone comes along and shows something way better.

[Roxy]

Thanks, will look this up for a future project.

I was just sitting in bed last night wondering how I could be notified if a white was on my driveway.

[Aussie_Adam]

Isn’t that what AI stands for? Alpaca Identification? or am I doing it wrong? hahaha

[doctorkb (kb)]

Love it! Thanks!

[doctorkb (kb)]

Interestingly, it seems to think the sequence is opposite of what it is – a vehicle pulling into my driveway is seen as “backing out”.

Could fixing this just be a matter of reversing the image sequence as sent to Google?

[Aussie_Adam]

Possibly. I noticed mine does this as well on one camera when I enter the drive. I will have a play and report back.

[tmsblgh (Tamas Balogh)]

Google AI is not available in Europe… is this could work with ChatGPT? Or any other AI solution?

[smithey253 (Matt Smith)]

Works a treat! Initially got a 400 error about billing but once I added a payment account it worked straight away. It seems UK and EEA areas don’t get gemini for ‘free’ and it has to have a linked payment account. Unsure whether it’ll cost or not as the documentation is unclear.

[Anto79-ops]

hey, how do you deal with the occassional 500 errors from Google?

For example, in my case, I just want to Google to tell us what type of bird it is from the photo thumbnail, but if it fails (because of a 500 error), to just say there is motion detected

With a 500 error, it stops the whole automation so I tried this if-else, but it seems to always default to else, even if Google responded correctly:

  - service: tts.speak
    data:
      cache: true
      media_player_entity_id: media_player.fire_tablet
      message: >-        
        {% if 'text' in generated_response %}
            Motion has been detected at your bird feeder. {{ generated_response['text'] }}
        {% else %}
            Motion has been detected at your bird feeder. No AI avialable.
        {% endif %}
    target:
      entity_id: tts.piper

The photo part for this is a little different than text, because with text you can work with the action_done attribute, but I don’t think the photo generation has this?

thanks

[m90att (Matt)]

Hi @Aussie_Adam , I just want to thank you for this. I’ve been trying to achieve similar for days and then came across this and it helped no end. Kudos sir!

[Schottburg (Jordan Schott)]

This is phenominal, thank you for the write-up! My only issue is I’m seeing an old/cached snapshot in the notification, even though the snapshot is up to date in the folder. Has this happened to anyone else?

[Aussie_Adam]

Sorry for the slow reply, but I haven’t come across this issue on my end with Iphone or Ipad. Did you end up getting it sorted or are you still seeing cached images?

[Dragon1473 (Daniel Kozachek)]

Thank you so much for this! I do have one quick question / issue I’m running in to. Even when using the updated script, I’m still receiving notifications indicating “Camera has detected motion however no obvious motion observed comparing snapshots.”

Any thoughts on why it may be triggering? When I check the traces, it’s evaluating as false.

[Aussie_Adam]

Not sure without seeing your YAML however just double check that you don’t have an extra space or something in the text in the prompt or in the if statement, especially at the end after the full stop. It needs to match exactly or it will still trigger and when I wrote, one space did my head in for ages so just check! If still no joy, post up your YAML on a GitHub Gist or even post below and I’ll try spot what is going on

[Dragon1473 (Daniel Kozachek)]

Maybe it is something I’m just missing after looking at it over and over. I tried changing the text as well. Apprecaite any guidance on anything I might have missed! I’m posting from the call for the Generative AI on - the actual camera calls for the snapshots works great!

Once again, thanks for posting this!

  - action: google_generative_ai_conversation.generate_content
    metadata: {}
    data:
      prompt: >-
        Motion has been detected, compare and very briefly describe what you see
        in the following sequence of images from my driveway camera number. What
        do you think caused the motion alarm? If a person or car is present,
        describe them in detail. Do not describe stationary objects or
        buildings. If you see no obvious causes of motion, reply with "No
        Obvious Motion Detected." Your message needs to be short enough to fit
        in a phone notification. 
      image_filename:
        - ./www/snapshots/driveway1_snapshot1.jpg
        - ./www/snapshots/driveway1_snapshot2.jpg
        - ./www/snapshots/driveway1_snapshot3.jpg
    response_variable: generated_content
  - if:
      - condition: template
        value_template: |
          {{ generated_content['text'] == 'No Obvious Motion Detected.'}} 
    then:
      - stop: 
    else:
      - metadata: {}
        data:
          title: Driveway Motion Detected
          message: "{{generated_content['text'] }}"
          data:
            image: /local/snapshots/driveway1_snapshot2.jpg
        action: notify.mobile_app_danielws_iphone_2
mode: single

[Aussie_Adam]

Ok my first reply was wrong, but I fixed it and it runs fine on my end now. Let me know if it works for you now:

  - action: google_generative_ai_conversation.generate_content
    metadata: {}
    data:
      prompt: >-
        Motion has been detected, compare and very briefly describe what you see
        in the following sequence of images from my driveway camera number 1. What
        do you think caused the motion alarm? If a person or car is present,
        describe them in detail. Do not describe stationary objects or
        buildings. If you see no obvious causes of motion, reply with "No
        Obvious Motion Detected." Your message needs to be short enough to fit
        in a phone notification. 
      image_filename:
        - ./www/snapshots/driveway1_snapshot2.jpg
        - ./www/snapshots/driveway1_snapshot1.jpg
    response_variable: generated_content
  - if:
      - condition: template
        value_template: >-
          {{ 'No Obvious Motion Detected.' in generated_content.text }}
    then:
      - stop: " No motion detected"
    else:
      - metadata: {}
        data:
          title: Driveway Motion Detected
          message: "{{generated_content['text'] }}"
          data:
            image: /local/snapshots/driveway1_snapshot2.jpg
        action: notify.mobile_app_danielws_iphone_2

[Dragon1473 (Daniel Kozachek)]

Just tested it - worked great! Thank you so much!

[RaA11]

Thank You for sharing

I’m getting notification with old snapshot and not the correct one, there’s any way to refresh before sending the picture from the snapshot folder?

[RaA11]

Solution:

image: >-
https://HA-URL/local/snapshots/intercom_snapshot2.jpg?{{as_timestamp(now())}}