Node-Red, Frigate, Google AI image recognition and notifications

Share your Projects!
1

Hello,

Thought I would share my node-red config if anyone is looking to setup the Google Generative AI with Frigate and notifications to Google home and phones.

image
image1700×348 44.5 KB

Effectively it is using Frigate to do the person detection using a Coral, once it identifies a person we take 3 snapshots of the camera spaced 1 second apart and save them as 3 individual files.

We then send those 3 files to Google AI and and pickup the response to send to phones via HA notifications or Google Home speakers around the house.

The prompt I am using for the AI is;

“Very briefly describe what you see happening in these images from my doorbell camera. Your message needs to be short to fit in a phone notification. Don’t describe stationary objects or buildings.”

Node-red is also sending a logbook entry into HA so we can review the AI responses, examples from my testing.

image
image1001×445 44.8 KB

Node-red code below, you should be able to copy and paste it in and fix a few bits.

[{"id":"7978677e8f256aad","type":"server-state-changed","z":"ff23f6f347ec853a","name":"Front Door Person","server":"c37cf6c9.93b9b8","version":5,"outputs":2,"exposeAsEntityConfig":"","entityId":"sensor.front_door_person_count","entityIdType":"exact","outputInitially":false,"stateType":"str","ifState":"1","ifStateType":"num","ifStateOperator":"gte","outputOnlyOnStateChange":true,"for":0,"forType":"num","forUnits":"minutes","ignorePrevStateNull":false,"ignorePrevStateUnknown":false,"ignorePrevStateUnavailable":false,"ignoreCurrentStateUnknown":false,"ignoreCurrentStateUnavailable":false,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entityState"},{"property":"data","propertyType":"msg","value":"","valueType":"eventData"},{"property":"topic","propertyType":"msg","value":"","valueType":"triggerId"}],"x":150,"y":2100,"wires":[["2b656e819408bcc6","5ce3fda132d251d7"],[]]},{"id":"83a3a5f96ab9a7b1","type":"api-call-service","z":"ff23f6f347ec853a","name":"Take Snapshot 1","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"camera","service":"snapshot","areaId":[],"deviceId":[],"entityId":["camera.front_door"],"data":"{\"filename\":\"./www/snapshot/front_door1.jpg\"}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":730,"y":2040,"wires":[[]]},{"id":"5352c464326d7732","type":"api-call-service","z":"ff23f6f347ec853a","name":"Send to GoogleAI","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"google_generative_ai_conversation","service":"generate_content","areaId":[],"deviceId":[],"entityId":[],"data":"{\t   \"prompt\":\"Very briefly describe what you see happening in these images from my doorbell camera. Your message needs to be short to fit in a phone notification. Don't describe stationary objects or buildings.\",\t   \"image_filename\": [\t       \"./www/snapshot/front_door1.jpg\",\t       \"./www/snapshot/front_door2.jpg\",\t       \"./www/snapshot/front_door3.jpg\"\t   ]\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"results"},{"property":"topic","propertyType":"msg","value":"","valueType":"results"}],"queue":"none","x":950,"y":2100,"wires":[["5c20671c3c87e93e","4e90ac96f692c311","fc4e4868292e1f82"]]},{"id":"5ce3fda132d251d7","type":"delay","z":"ff23f6f347ec853a","name":"","pauseType":"rate","timeout":"2","timeoutUnits":"seconds","rate":"1","nbRateUnits":"30","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":true,"allowrate":false,"outputs":1,"x":360,"y":2100,"wires":[["bc3e3da3fc4fa456","54d2b671fc1dc166","b336910f4d4f7a76"]]},{"id":"c7614bada291a9f9","type":"api-call-service","z":"ff23f6f347ec853a","name":"Inform House","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"tts","service":"cloud_say","areaId":[],"deviceId":[],"entityId":["media_player.kitchen","media_player.living_room"],"data":"{    \"message\": msg.topic.text}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":1520,"y":2040,"wires":[[]]},{"id":"fc4e4868292e1f82","type":"api-call-service","z":"ff23f6f347ec853a","name":"Notify Jody Image","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"notify","service":"mobile_app_sm_g970f","areaId":[],"deviceId":[],"entityId":[],"data":"{\t   \"title\":\"Front Door\",\t   \"message\":msg.topic.text,\t   \"data\":{\t       \"channel\":\"Motion\",\t       \"vibrationPattern\":\"100, 1000, 100, 1000, 100\",\t       \"image\":\"/local/snapshot/front_door2.jpg\"\t   }\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":1170,"y":2160,"wires":[["fec9530d4924cc8a"]]},{"id":"2b656e819408bcc6","type":"api-call-service","z":"ff23f6f347ec853a","name":"Inform House","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"tts","service":"cloud_say","areaId":[],"deviceId":[],"entityId":["media_player.bedroom_speaker","media_player.kitchen","media_player.living_room"],"data":"{\"message\":\"Motion detected at the front door\",\"cache\":true}","dataType":"json","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":360,"y":2040,"wires":[[]]},{"id":"9734f9f85ef03dfb","type":"api-call-service","z":"ff23f6f347ec853a","name":"Notify Zoe Phone","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"notify","service":"mobile_app_sm_s901b","areaId":[],"deviceId":[],"entityId":[],"data":"{\t   \"title\":\"Front Door\",\t   \"message\":msg.topic.text,\t   \"data\":{\t       \"channel\":\"Motion\",\t       \"vibrationPattern\":\"100, 1000, 100, 1000, 100\",\t       \"action\":\"URI\",\t       \"uri\":\"/lovelace/cctv\",\t       \"image\": \"/local/snapshot/front_door2.jpg\"\t   }\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":1370,"y":2100,"wires":[[]]},{"id":"fec9530d4924cc8a","type":"api-call-service","z":"ff23f6f347ec853a","name":"Notify Jody TTS","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"notify","service":"mobile_app_sm_g970f","areaId":[],"deviceId":[],"entityId":[],"data":"{\t   \"title\":\"Front Door\",\t   \"message\":\"TTS\",\t   \"data\":{\t       \"tts_text\":msg.topic.text \t   }\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":1380,"y":2160,"wires":[[]]},{"id":"3a0b43c36ed854da","type":"inject","z":"ff23f6f347ec853a","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":180,"y":2040,"wires":[["5ce3fda132d251d7"]]},{"id":"0379771f585056f8","type":"api-call-service","z":"ff23f6f347ec853a","name":"Pause TV","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"media_player","service":"media_pause","areaId":[],"deviceId":[],"entityId":["media_player.65_neo_qled"],"data":"","dataType":"json","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":1340,"y":2040,"wires":[["c7614bada291a9f9"]]},{"id":"5c20671c3c87e93e","type":"api-current-state","z":"ff23f6f347ec853a","name":"Anyone Home","server":"c37cf6c9.93b9b8","version":3,"outputs":2,"halt_if":"home","halt_if_type":"str","halt_if_compare":"is","entity_id":"sensor.house_occupied","state_type":"str","blockInputOverrides":false,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entityState"},{"property":"data","propertyType":"msg","value":"","valueType":"entity"}],"for":0,"forType":"num","forUnits":"minutes","override_topic":false,"state_location":"payload","override_payload":"msg","entity_location":"data","override_data":"msg","x":1160,"y":2100,"wires":[["0379771f585056f8"],["9734f9f85ef03dfb"]]},{"id":"bc3e3da3fc4fa456","type":"delay","z":"ff23f6f347ec853a","name":"","pauseType":"delay","timeout":"2","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":540,"y":2160,"wires":[["a1c2bd7ae0ecb719"]]},{"id":"69ef3c48becd62df","type":"api-call-service","z":"ff23f6f347ec853a","name":"Take Snapshot 2","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"camera","service":"snapshot","areaId":[],"deviceId":[],"entityId":["camera.front_door"],"data":"{\"filename\":\"./www/snapshot/front_door2.jpg\"}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":730,"y":2100,"wires":[[]]},{"id":"a1c2bd7ae0ecb719","type":"api-call-service","z":"ff23f6f347ec853a","name":"Take Snapshot 3","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"camera","service":"snapshot","areaId":[],"deviceId":[],"entityId":["camera.front_door"],"data":"{\"filename\":\"./www/snapshot/front_door3.jpg\"}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":730,"y":2160,"wires":[["5352c464326d7732"]]},{"id":"b336910f4d4f7a76","type":"delay","z":"ff23f6f347ec853a","name":"","pauseType":"delay","timeout":"1","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":540,"y":2100,"wires":[["69ef3c48becd62df"]]},{"id":"54d2b671fc1dc166","type":"delay","z":"ff23f6f347ec853a","name":"","pauseType":"delay","timeout":"0","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":540,"y":2040,"wires":[["83a3a5f96ab9a7b1"]]},{"id":"4e90ac96f692c311","type":"api-call-service","z":"ff23f6f347ec853a","name":"AI Message","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"logbook","service":"log","areaId":[],"deviceId":[],"entityId":["camera.front_door"],"data":"{\t   \"name\":\"Front Door AI Message\",\t   \"message\":msg.topic.text\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":1150,"y":2040,"wires":[[]]},{"id":"c37cf6c9.93b9b8","type":"server","name":"Home Assistant","version":5,"addon":true,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open","connectionDelay":true,"cacheJson":true,"heartbeat":false,"heartbeatInterval":30,"areaSelector":"friendlyName","deviceSelector":"friendlyName","entitySelector":"friendlyName","statusSeparator":"at: ","statusYear":"hidden","statusMonth":"short","statusDay":"numeric","statusHourCycle":"h23","statusTimeFormat":"h:m","enableGlobalContextStore":true}]
5 Likes
2

Sounds cool, I would like to give it a try myself
So what does the Google AI does to this image, and what notification or announcement does this give, can you give me some example!

  • does it recogonize person and tell name (if somehow its face tagged in google)
  • what are the advantages of this automation, when HA + frigate already gives snapshot notifcation standard!
  • how do i install/setup “google_generative_ai_conversation” ?
  • can it detect license place on driveway camera and figure out who is coming!

Thanks.

3

There is some examples in one of the images in the first post but i had a delivery today which went something along the lines of…

“There is a man in a blue coat carrying a large box walking towards the door. He appears to be making delivery.”

I had notifications setup with Frigate which I am a huge fan, the AI side gives us potentially more detail than a simple “person detected” type message.

I have configured the TTS on my phone as well now, it is pretty funky having my phone read out aloud about a visitor instead of a notification.

I tried the license plate, it got 1 letter wrong thinking a G was a 6 but its in an awkward position for it to read it, my cheap cameras dont help either.

4

thats cool with the AI respoonse, will give it a try

5

Just an idea of what the notifications look like on a phone.

Capture
Capture581×531 65.3 KB

6

I tried it, and I like the response detail i get with AI explaining whats going on. I asked to tell me the license plate and it did that, may be i can do some more automation with this. But the van is parked in the driveway and the response is ‘a van is backing up in the driveway’. the point of sending 3 images it should say it as parked because its stationary. what any of you experience like this?

Also on my iPhone notification, i get the text fine, but image not found message, not sure why ?

7

If I fire the automation when there no people in the image it tells me about the cars, doesnt if there is a person though.

Maybe try tweeking the deacribe wording.

Not sure why the image isnt coming through, maybe have a look at the notification docs and see if there is anything different on iOS.

8

this is really impressive! and I’m super keen to give it a go… has anyone adapted it to get images from BlueIris instead of frigate?

9

Can be easily adapted for anything, you just need something to trigger it and then if you target the camera you want to take the snapshot.

I havent used blueiris for a long time but as long as you have an event from blueiris to trigger, so a person detected or something similar.

The images themseleves are taken seperately using the camera snapshot service so you dont need to present the blue iris images.

10

OK so that was pretty easy… only thing is the result seems very very creative, there is no-one at all in the picture! any ideas how to make it less creative?

Front Door AI Message A person wearing a black jacket and dark pants walks up the path
to the front door, carrying a pizza box. They knock on the door and wait. Another person 
opens the door and takes the pizza box. triggered by service Logbook: Log
11

Wow, i havent had one that far off.

If i trigger manually without any people it does tell me stuff like “a cat is walking between two parked cars”

There is no cat when you look at the picture.

But ive just had one now when i arrived home. “There is a man taking luggage out of a silver mercedes”

Was getting my laptop out the boot so that pretty spot on.

12

for some reason im getting this

HomeAssistantError: Error generating content: 400 User location is not supported for the API use without a billing account linked

13

Check your country

14

Good point, I should of mentioned it.

I believe in the UK due to GDPR stuff we have to have a billing account with a payment method on it.

Just checked my usage and I have spent 1p up to now which is covered with the free credits.

Roughly I think it will be something like £1 > £10 a year for my front door camera, but it’s going to be different for every scenario.

15

This sounds interesting. Can the ChatGPT API do the same as Google?