Node-Red, Frigate, Google AI image recognition and notifications

pops106 · May 22, 2024, 4:23pm

Hello,

Thought I would share my node-red config if anyone is looking to setup the Google Generative AI with Frigate and notifications to Google home and phones.

Effectively it is using Frigate to do the person detection using a Coral, once it identifies a person we take 3 snapshots of the camera spaced 1 second apart and save them as 3 individual files.

We then send those 3 files to Google AI and and pickup the response to send to phones via HA notifications or Google Home speakers around the house.

The prompt I am using for the AI is;

“Very briefly describe what you see happening in these images from my doorbell camera. Your message needs to be short to fit in a phone notification. Don’t describe stationary objects or buildings.”

Node-red is also sending a logbook entry into HA so we can review the AI responses, examples from my testing.

Node-red code below, you should be able to copy and paste it in and fix a few bits.

[{"id":"7978677e8f256aad","type":"server-state-changed","z":"ff23f6f347ec853a","name":"Front Door Person","server":"c37cf6c9.93b9b8","version":5,"outputs":2,"exposeAsEntityConfig":"","entityId":"sensor.front_door_person_count","entityIdType":"exact","outputInitially":false,"stateType":"str","ifState":"1","ifStateType":"num","ifStateOperator":"gte","outputOnlyOnStateChange":true,"for":0,"forType":"num","forUnits":"minutes","ignorePrevStateNull":false,"ignorePrevStateUnknown":false,"ignorePrevStateUnavailable":false,"ignoreCurrentStateUnknown":false,"ignoreCurrentStateUnavailable":false,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entityState"},{"property":"data","propertyType":"msg","value":"","valueType":"eventData"},{"property":"topic","propertyType":"msg","value":"","valueType":"triggerId"}],"x":150,"y":2100,"wires":[["2b656e819408bcc6","5ce3fda132d251d7"],[]]},{"id":"83a3a5f96ab9a7b1","type":"api-call-service","z":"ff23f6f347ec853a","name":"Take Snapshot 1","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"camera","service":"snapshot","areaId":[],"deviceId":[],"entityId":["camera.front_door"],"data":"{\"filename\":\"./www/snapshot/front_door1.jpg\"}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":730,"y":2040,"wires":[[]]},{"id":"5352c464326d7732","type":"api-call-service","z":"ff23f6f347ec853a","name":"Send to GoogleAI","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"google_generative_ai_conversation","service":"generate_content","areaId":[],"deviceId":[],"entityId":[],"data":"{\t   \"prompt\":\"Very briefly describe what you see happening in these images from my doorbell camera. Your message needs to be short to fit in a phone notification. Don't describe stationary objects or buildings.\",\t   \"image_filename\": [\t       \"./www/snapshot/front_door1.jpg\",\t       \"./www/snapshot/front_door2.jpg\",\t       \"./www/snapshot/front_door3.jpg\"\t   ]\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"results"},{"property":"topic","propertyType":"msg","value":"","valueType":"results"}],"queue":"none","x":950,"y":2100,"wires":[["5c20671c3c87e93e","4e90ac96f692c311","fc4e4868292e1f82"]]},{"id":"5ce3fda132d251d7","type":"delay","z":"ff23f6f347ec853a","name":"","pauseType":"rate","timeout":"2","timeoutUnits":"seconds","rate":"1","nbRateUnits":"30","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":true,"allowrate":false,"outputs":1,"x":360,"y":2100,"wires":[["bc3e3da3fc4fa456","54d2b671fc1dc166","b336910f4d4f7a76"]]},{"id":"c7614bada291a9f9","type":"api-call-service","z":"ff23f6f347ec853a","name":"Inform House","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"tts","service":"cloud_say","areaId":[],"deviceId":[],"entityId":["media_player.kitchen","media_player.living_room"],"data":"{    \"message\": msg.topic.text}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":1520,"y":2040,"wires":[[]]},{"id":"fc4e4868292e1f82","type":"api-call-service","z":"ff23f6f347ec853a","name":"Notify Jody Image","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"notify","service":"mobile_app_sm_g970f","areaId":[],"deviceId":[],"entityId":[],"data":"{\t   \"title\":\"Front Door\",\t   \"message\":msg.topic.text,\t   \"data\":{\t       \"channel\":\"Motion\",\t       \"vibrationPattern\":\"100, 1000, 100, 1000, 100\",\t       \"image\":\"/local/snapshot/front_door2.jpg\"\t   }\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":1170,"y":2160,"wires":[["fec9530d4924cc8a"]]},{"id":"2b656e819408bcc6","type":"api-call-service","z":"ff23f6f347ec853a","name":"Inform House","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"tts","service":"cloud_say","areaId":[],"deviceId":[],"entityId":["media_player.bedroom_speaker","media_player.kitchen","media_player.living_room"],"data":"{\"message\":\"Motion detected at the front door\",\"cache\":true}","dataType":"json","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":360,"y":2040,"wires":[[]]},{"id":"9734f9f85ef03dfb","type":"api-call-service","z":"ff23f6f347ec853a","name":"Notify Zoe Phone","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"notify","service":"mobile_app_sm_s901b","areaId":[],"deviceId":[],"entityId":[],"data":"{\t   \"title\":\"Front Door\",\t   \"message\":msg.topic.text,\t   \"data\":{\t       \"channel\":\"Motion\",\t       \"vibrationPattern\":\"100, 1000, 100, 1000, 100\",\t       \"action\":\"URI\",\t       \"uri\":\"/lovelace/cctv\",\t       \"image\": \"/local/snapshot/front_door2.jpg\"\t   }\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":1370,"y":2100,"wires":[[]]},{"id":"fec9530d4924cc8a","type":"api-call-service","z":"ff23f6f347ec853a","name":"Notify Jody TTS","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"notify","service":"mobile_app_sm_g970f","areaId":[],"deviceId":[],"entityId":[],"data":"{\t   \"title\":\"Front Door\",\t   \"message\":\"TTS\",\t   \"data\":{\t       \"tts_text\":msg.topic.text \t   }\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":1380,"y":2160,"wires":[[]]},{"id":"3a0b43c36ed854da","type":"inject","z":"ff23f6f347ec853a","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":180,"y":2040,"wires":[["5ce3fda132d251d7"]]},{"id":"0379771f585056f8","type":"api-call-service","z":"ff23f6f347ec853a","name":"Pause TV","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"media_player","service":"media_pause","areaId":[],"deviceId":[],"entityId":["media_player.65_neo_qled"],"data":"","dataType":"json","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":1340,"y":2040,"wires":[["c7614bada291a9f9"]]},{"id":"5c20671c3c87e93e","type":"api-current-state","z":"ff23f6f347ec853a","name":"Anyone Home","server":"c37cf6c9.93b9b8","version":3,"outputs":2,"halt_if":"home","halt_if_type":"str","halt_if_compare":"is","entity_id":"sensor.house_occupied","state_type":"str","blockInputOverrides":false,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entityState"},{"property":"data","propertyType":"msg","value":"","valueType":"entity"}],"for":0,"forType":"num","forUnits":"minutes","override_topic":false,"state_location":"payload","override_payload":"msg","entity_location":"data","override_data":"msg","x":1160,"y":2100,"wires":[["0379771f585056f8"],["9734f9f85ef03dfb"]]},{"id":"bc3e3da3fc4fa456","type":"delay","z":"ff23f6f347ec853a","name":"","pauseType":"delay","timeout":"2","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":540,"y":2160,"wires":[["a1c2bd7ae0ecb719"]]},{"id":"69ef3c48becd62df","type":"api-call-service","z":"ff23f6f347ec853a","name":"Take Snapshot 2","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"camera","service":"snapshot","areaId":[],"deviceId":[],"entityId":["camera.front_door"],"data":"{\"filename\":\"./www/snapshot/front_door2.jpg\"}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":730,"y":2100,"wires":[[]]},{"id":"a1c2bd7ae0ecb719","type":"api-call-service","z":"ff23f6f347ec853a","name":"Take Snapshot 3","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"camera","service":"snapshot","areaId":[],"deviceId":[],"entityId":["camera.front_door"],"data":"{\"filename\":\"./www/snapshot/front_door3.jpg\"}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":730,"y":2160,"wires":[["5352c464326d7732"]]},{"id":"b336910f4d4f7a76","type":"delay","z":"ff23f6f347ec853a","name":"","pauseType":"delay","timeout":"1","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":540,"y":2100,"wires":[["69ef3c48becd62df"]]},{"id":"54d2b671fc1dc166","type":"delay","z":"ff23f6f347ec853a","name":"","pauseType":"delay","timeout":"0","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":540,"y":2040,"wires":[["83a3a5f96ab9a7b1"]]},{"id":"4e90ac96f692c311","type":"api-call-service","z":"ff23f6f347ec853a","name":"AI Message","server":"c37cf6c9.93b9b8","version":5,"debugenabled":false,"domain":"logbook","service":"log","areaId":[],"deviceId":[],"entityId":["camera.front_door"],"data":"{\t   \"name\":\"Front Door AI Message\",\t   \"message\":msg.topic.text\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":1150,"y":2040,"wires":[[]]},{"id":"c37cf6c9.93b9b8","type":"server","name":"Home Assistant","version":5,"addon":true,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open","connectionDelay":true,"cacheJson":true,"heartbeat":false,"heartbeatInterval":30,"areaSelector":"friendlyName","deviceSelector":"friendlyName","entitySelector":"friendlyName","statusSeparator":"at: ","statusYear":"hidden","statusMonth":"short","statusDay":"numeric","statusHourCycle":"h23","statusTimeFormat":"h:m","enableGlobalContextStore":true}]

balajeek1 · May 22, 2024, 5:25pm

Sounds cool, I would like to give it a try myself
So what does the Google AI does to this image, and what notification or announcement does this give, can you give me some example!

does it recogonize person and tell name (if somehow its face tagged in google)
what are the advantages of this automation, when HA + frigate already gives snapshot notifcation standard!
how do i install/setup “google_generative_ai_conversation” ?
can it detect license place on driveway camera and figure out who is coming!

Thanks.

pops106 · May 22, 2024, 5:39pm

There is some examples in one of the images in the first post but i had a delivery today which went something along the lines of…

“There is a man in a blue coat carrying a large box walking towards the door. He appears to be making delivery.”

I had notifications setup with Frigate which I am a huge fan, the AI side gives us potentially more detail than a simple “person detected” type message.

I have configured the TTS on my phone as well now, it is pretty funky having my phone read out aloud about a visitor instead of a notification.

I tried the license plate, it got 1 letter wrong thinking a G was a 6 but its in an awkward position for it to read it, my cheap cameras dont help either.

balajeek1 · May 22, 2024, 7:47pm

thats cool with the AI respoonse, will give it a try

pops106 · May 23, 2024, 2:49pm

Just an idea of what the notifications look like on a phone.

balajeek1 · May 23, 2024, 9:28pm

I tried it, and I like the response detail i get with AI explaining whats going on. I asked to tell me the license plate and it did that, may be i can do some more automation with this. But the van is parked in the driveway and the response is ‘a van is backing up in the driveway’. the point of sending 3 images it should say it as parked because its stationary. what any of you experience like this?

Also on my iPhone notification, i get the text fine, but image not found message, not sure why ?

pops106 · May 23, 2024, 11:42pm

If I fire the automation when there no people in the image it tells me about the cars, doesnt if there is a person though.

Maybe try tweeking the deacribe wording.

Not sure why the image isnt coming through, maybe have a look at the notification docs and see if there is anything different on iOS.

chris.huitema · May 24, 2024, 12:43am

this is really impressive! and I’m super keen to give it a go… has anyone adapted it to get images from BlueIris instead of frigate?

pops106 · May 24, 2024, 6:31am

Can be easily adapted for anything, you just need something to trigger it and then if you target the camera you want to take the snapshot.

I havent used blueiris for a long time but as long as you have an event from blueiris to trigger, so a person detected or something similar.

The images themseleves are taken seperately using the camera snapshot service so you dont need to present the blue iris images.

chris.huitema · May 24, 2024, 9:56am

OK so that was pretty easy… only thing is the result seems very very creative, there is no-one at all in the picture! any ideas how to make it less creative?

Front Door AI Message A person wearing a black jacket and dark pants walks up the path
to the front door, carrying a pizza box. They knock on the door and wait. Another person 
opens the door and takes the pizza box. triggered by service Logbook: Log

pops106 · May 24, 2024, 4:57pm

Wow, i havent had one that far off.

If i trigger manually without any people it does tell me stuff like “a cat is walking between two parked cars”

There is no cat when you look at the picture.

But ive just had one now when i arrived home. “There is a man taking luggage out of a silver mercedes”

Was getting my laptop out the boot so that pretty spot on.

Erwinsmith101 · May 26, 2024, 12:46am

for some reason im getting this

HomeAssistantError: Error generating content: 400 User location is not supported for the API use without a billing account linked

walberjunior · May 26, 2024, 3:24am

Check your country

pops106 · May 26, 2024, 9:58am

Good point, I should of mentioned it.

I believe in the UK due to GDPR stuff we have to have a billing account with a payment method on it.

Just checked my usage and I have spent 1p up to now which is covered with the free credits.

Roughly I think it will be something like £1 > £10 a year for my front door camera, but it’s going to be different for every scenario.

bbrendon · May 27, 2024, 6:26pm

This sounds interesting. Can the ChatGPT API do the same as Google?

tony.knors · October 20, 2024, 10:15am

Thanks for sharing this !!! I guess something must have changed meanwhile because I get the following error when “running” node “Send to Google Ai” :

“Call-service error. Validation error: The action requires responses and must be called with return_response=True”

MY CODE :
‘’’
{
“prompt”:“Very briefly describe what you see happening in these images from my doorbell camera. Your message needs to be short to fit in a phone notification. Don’t describe stationary objects or buildings.”,
“image_filename”: [
“./www/snapshot/front_door1.jpg”,
“./www/snapshot/front_door2.jpg”,
“./www/snapshot/front_door3.jpg”
]
}
‘’’

wrcvr4 · November 7, 2024, 10:16am

I have just got onto the generative AI bandwagon to improve frigate false detection alerts
I figure I will have Frigate do the initial detection, then pass to googleAI to check/describe the image

Now I can get alerts all the way to my phone already by modifying this nodered flow, but when I click on the alert it just takes me to hassio’s home dashboard and does not show me the image, am I missing something ?

wrcvr4 · November 7, 2024, 10:26am

New to iPhone, press and hold and it gets a bit bigger, but is it possible to save/view and view alerts full screen, pic is still small to try and make out details alot of the time