here is what you can do
1 Like
Where in the blueprint can i add the code ?
I tried do it like you are ,but cant figure out what i need to do in the notification input (the last part in your screenshot )
Found the problem I had around āold imageā being included in the notifications. I suspected it to be a cache problem, and indeed it is. I am using Cloudflare for my domain and Cloudflared tunnel and it turns out Cloudflare was caching the image and because the names of the images are the same (e.g. _0.jpg) I always got a cached (old) version of the image.
I solved it by introducing a cache rule to not cache ājpgā images. As described here: Home Assistant sends cached images in iOS notification? - #8 by msl2424
2 Likes
I am looking for a way to generate PNG images to display them in my display which is in LVGL esphome and which only takes transparent PNG, do you have any solutions please
I donāt use blueprint but a new automation with LLM vision
in LLM vision is it possible to add Deepseek
FYI before anyone decides to spend $ on the OpenAI api, I would suggest you try Gemini first. Iāve ran into multiple prompts where OpenAI just doesnāt like the prompt Iām feeding it and says āsorry I canāt help with thatā. Gemini seems to be a lot more straight forward with the information it spits out where OpenAI tries to go for a human touch which isnāt what Iām trying to accomplish with the security prompts.
Iāll post my node-red example flow here soon, Iām sure someone can improve on what I built.
Node Red Example Flow here.
Few notes, Iām using the following integrations to help:
- Google Cast, specifically Google Home Hubs (Can be switched with Alexa or whatever you have)
- Frigate
- Unifi Protect
- Companion App.
Frigate might be unnecessary in this case, a binary sensor can be used to replace. Same thing with Unifi Protect, just change to your own camera setup. Something however has to trigger the person detection.
I also removed my provider from these flows because it feels like it could be tied to my API key, you can find your provider by going to Developer Tools ā going to actions ā selecting āLLM Vision. Image Analyzerā, and at the bottom clicking on āgo to yaml modeā.
Paste the provider and include quotes
[{"id":"b2b424ddbd92c0e2","type":"tab","label":"Example Security Flow","disabled":false,"info":"","env":[]},{"id":"2cdded6300218d5e","type":"comment","z":"b2b424ddbd92c0e2","name":"Driveway AI Warning, CLICK ON COMMENT NODES FOR FURTHER EXPLAINATION","info":"","x":340,"y":40,"wires":[]},{"id":"662d722c631b6961","type":"server-state-changed","z":"b2b424ddbd92c0e2","name":"Occupancy detected for 500 MS","server":"2e31528c.f3973e","version":6,"outputs":2,"exposeAsEntityConfig":"","entities":{"entity":["binary_sensor.driveway_person_occupancy"],"substring":[],"regex":[]},"outputInitially":false,"stateType":"str","ifState":"on","ifStateType":"str","ifStateOperator":"is","outputOnlyOnStateChange":true,"for":".5","forType":"num","forUnits":"seconds","ignorePrevStateNull":false,"ignorePrevStateUnknown":false,"ignorePrevStateUnavailable":false,"ignoreCurrentStateUnknown":false,"ignoreCurrentStateUnavailable":false,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entityState"},{"property":"data","propertyType":"msg","value":"","valueType":"eventData"},{"property":"topic","propertyType":"msg","value":"","valueType":"triggerId"}],"x":130,"y":280,"wires":[["79b5817c845a0263"],[]]},{"id":"911940e0ea4500d5","type":"api-call-service","z":"b2b424ddbd92c0e2","name":"Driveway camera snapshot","server":"2e31528c.f3973e","version":7,"debugenabled":false,"action":"camera.snapshot","floorId":[],"areaId":[],"deviceId":[],"entityId":["camera.driveway_high"],"labelId":[],"data":"{\"filename\":\"/media/snapshots/driveway_person.jpg\"}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","blockInputOverrides":false,"domain":"camera","service":"snapshot","x":960,"y":280,"wires":[["27de42ed98e84e6f"]]},{"id":"0bb2f0386aa72212","type":"api-call-service","z":"b2b424ddbd92c0e2","name":"Ask Google about person","server":"2e31528c.f3973e","version":7,"debugenabled":false,"action":"llmvision.image_analyzer","floorId":[],"areaId":[],"deviceId":[],"entityId":[],"labelId":[],"data":"{\t \"provider\":\"xxx\",\t \"model\":\"gemini-1.5-pro\",\t \"message\":\"Scare the person by describing them, they are most likely attempting to break in, keep it short.\",\t \"image_file\":\"/media/snapshots/driveway_person.jpg\",\t \"target_width\":650,\t \"max_tokens\":100,\t \"temperature\":0.3,\t \"include_filename\": \"true\"\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[{"property":"response_text","propertyType":"msg","value":"","valueType":"results"}],"queue":"none","blockInputOverrides":false,"domain":"llmvision","service":"image_analyzer","x":1890,"y":260,"wires":[["767e40a9c729037c"]]},{"id":"27de42ed98e84e6f","type":"api-call-service","z":"b2b424ddbd92c0e2","name":"Send to GoogleAI","server":"2e31528c.f3973e","version":7,"debugenabled":false,"action":"llmvision.image_analyzer","floorId":[],"areaId":[],"deviceId":[],"entityId":[],"labelId":[],"data":"{\t \"provider\":\"xxx\",\t \"model\":\"gemini-2.0-flash-exp\",\t \"message\":\"Reply with a simple yes or no; is person in the photo holding a package or is there a UPS, USPS, FedEx, or Amazon truck in the picture?\",\t \"image_file\":\"/media/snapshots/driveway_person.jpg\",\t \"target_width\":650,\t \"max_tokens\":100,\t \"temperature\":0.3,\t \"include_filename\": \"true\"\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[{"property":"response_text","propertyType":"msg","value":"","valueType":"results"}],"queue":"none","blockInputOverrides":false,"domain":"llmvision","service":"image_analyzer","x":1190,"y":280,"wires":[["3c90dbfb4c71fc25","b38f7bf29cfdce23"]]},{"id":"b38f7bf29cfdce23","type":"switch","z":"b2b424ddbd92c0e2","name":"Check for yes/no","property":"response_text.response_text","propertyType":"msg","rules":[{"t":"cont","v":"Yes","vt":"str"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":1430,"y":280,"wires":[["be94bd0220304c46"],["0bb2f0386aa72212","695bfa6b95cdfefe","f4abfbf69ddc08f2"]]},{"id":"422b9ae59a7b796d","type":"change","z":"b2b424ddbd92c0e2","name":"","rules":[{"t":"set","p":"payload","pt":"flow","to":"response_text.response_text","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":2320,"y":260,"wires":[["edabed6540fa80fa","9edbb302b654e93d"]]},{"id":"767e40a9c729037c","type":"function","z":"b2b424ddbd92c0e2","name":"Trim Newline","func":"// Strip the newline character from msg.response_text\nif (msg.response_text.response_text) {\n msg.response_text.response_text = msg.response_text.response_text.trim();\n}\nreturn msg;","outputs":1,"timeout":"","noerr":0,"initialize":"","finalize":"","libs":[],"x":2130,"y":260,"wires":[["422b9ae59a7b796d"]]},{"id":"695bfa6b95cdfefe","type":"api-call-service","z":"b2b424ddbd92c0e2","name":"Set the Driveway speaker to 100%","server":"2e31528c.f3973e","version":7,"debugenabled":false,"action":"media_player.volume_set","floorId":[],"areaId":[],"deviceId":[],"entityId":["media_player.driveway_speaker"],"labelId":[],"data":"{\"volume_level\":1}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","blockInputOverrides":true,"domain":"media_player","service":"volume_set","x":1720,"y":340,"wires":[[]]},{"id":"20a4e389b36e32b3","type":"api-call-service","z":"b2b424ddbd92c0e2","name":"TTS Driveway","server":"2e31528c.f3973e","version":7,"debugenabled":false,"action":"tts.speak","floorId":[],"areaId":[],"deviceId":[],"entityId":["tts.home_assistant_cloud"],"labelId":[],"data":"{\"message\":\"{{flow.payload}}\",\"media_player_entity_id\":\"media_player.driveway_speaker\",\"cache\":\"true\"}","dataType":"json","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","blockInputOverrides":false,"domain":"tts","service":"speak","x":3400,"y":260,"wires":[[]]},{"id":"edabed6540fa80fa","type":"api-call-service","z":"b2b424ddbd92c0e2","name":"Phone Notification - Dan","server":"2e31528c.f3973e","version":7,"debugenabled":false,"action":"notify.mobile_app_dan_pixel_8_pro","floorId":[],"areaId":[],"deviceId":[],"entityId":[],"labelId":[],"data":"{\"title\":\"Warning sent to intruder\",\"message\":\"Message: {{flow.payload}}\",\"data\":{\"channel\":\"Intruder\",\"tag\":\"intruder-driveway\",\"image\":\"/media/local/snapshots/driveway_person.jpg\",\"clickAction\":\"/lovelace/cameras\",\"priority\":\"high\",\"ttl\":0}}","dataType":"json","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","blockInputOverrides":false,"domain":"notify","service":"mobile_app_dan_pixel_8_pro","x":2570,"y":200,"wires":[[]]},{"id":"79b5817c845a0263","type":"api-current-state","z":"b2b424ddbd92c0e2","name":"Package Expected","server":"2e31528c.f3973e","version":3,"outputs":2,"halt_if":"off","halt_if_type":"str","halt_if_compare":"is","entity_id":"input_boolean.test_driveway_warning","state_type":"str","blockInputOverrides":true,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entityState"},{"property":"data","propertyType":"msg","value":"","valueType":"entity"}],"for":"0","forType":"num","forUnits":"minutes","override_topic":false,"state_location":"payload","override_payload":"msg","entity_location":"data","override_data":"msg","x":470,"y":280,"wires":[["01a241bb67ffbc3f"],[]]},{"id":"057d5a368b0e4a91","type":"inject","z":"b2b424ddbd92c0e2","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":1640,"y":180,"wires":[["0bb2f0386aa72212"]]},{"id":"1b7b618912b6b64e","type":"inject","z":"b2b424ddbd92c0e2","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":1020,"y":200,"wires":[["27de42ed98e84e6f"]]},{"id":"3c90dbfb4c71fc25","type":"debug","z":"b2b424ddbd92c0e2","name":"debug 3","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1360,"y":500,"wires":[]},{"id":"9edbb302b654e93d","type":"api-current-state","z":"b2b424ddbd92c0e2","name":"Driveway warning test is on","server":"2e31528c.f3973e","version":3,"outputs":2,"halt_if":"on","halt_if_type":"str","halt_if_compare":"is","entity_id":"input_boolean.test_driveway_warning","state_type":"str","blockInputOverrides":true,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entityState"},{"property":"data","propertyType":"msg","value":"","valueType":"entity"}],"for":"0","forType":"num","forUnits":"minutes","override_topic":false,"state_location":"payload","override_payload":"msg","entity_location":"data","override_data":"msg","x":2580,"y":260,"wires":[[],["e8e6da043d261007"]]},{"id":"01a241bb67ffbc3f","type":"delay","z":"b2b424ddbd92c0e2","name":"","pauseType":"rate","timeout":"5","timeoutUnits":"seconds","rate":"1","nbRateUnits":"5","rateUnits":"minute","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":true,"allowrate":false,"outputs":1,"x":680,"y":280,"wires":[["911940e0ea4500d5"]]},{"id":"e8e6da043d261007","type":"time-range-switch","z":"b2b424ddbd92c0e2","name":"","lat":"35.69052","lon":"-78.29317","startTime":"23:00","endTime":"07:00","startOffset":0,"endOffset":0,"x":2970,"y":260,"wires":[["2f23a242efe854c2"],[]]},{"id":"d62465aac319339d","type":"api-call-service","z":"b2b424ddbd92c0e2","name":"Set Volume Guest Room Display","server":"2e31528c.f3973e","version":7,"debugenabled":false,"action":"media_player.volume_set","floorId":[],"areaId":[],"deviceId":[],"entityId":["media_player.guest_room_display"],"labelId":[],"data":"{\"volume_level\":0.7}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","blockInputOverrides":false,"domain":"media_player","service":"volume_set","output_location":"","output_location_type":"none","x":2454,"y":442,"wires":[["053bc8be21fc1eee"]]},{"id":"c96cd6bb8b93554a","type":"api-call-service","z":"b2b424ddbd92c0e2","name":"TTS Guest Room","server":"2e31528c.f3973e","version":7,"debugenabled":false,"action":"tts.speak","floorId":[],"areaId":[],"deviceId":[],"entityId":["tts.home_assistant_cloud"],"labelId":[],"data":"{\t \"message\":\"Person Spotted in the driveway, please check\",\t \"media_player_entity_id\": \"media_player.guest_room_display\",\t \"cache\":\"true\"\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","blockInputOverrides":false,"domain":"tts","service":"speak","x":2944,"y":442,"wires":[["9c5e2c2b06905886"]]},{"id":"053bc8be21fc1eee","type":"delay","z":"b2b424ddbd92c0e2","name":"","pauseType":"delay","timeout":"1","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":2700,"y":440,"wires":[["c96cd6bb8b93554a"]]},{"id":"9c5e2c2b06905886","type":"delay","z":"b2b424ddbd92c0e2","name":"","pauseType":"delay","timeout":"4","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":3134,"y":442,"wires":[["afe5b1fa2b6cf289"]]},{"id":"afe5b1fa2b6cf289","type":"api-call-service","z":"b2b424ddbd92c0e2","name":"TTS Guest Room","server":"2e31528c.f3973e","version":7,"debugenabled":false,"action":"tts.speak","floorId":[],"areaId":[],"deviceId":[],"entityId":["tts.home_assistant_cloud"],"labelId":[],"data":"{\t \"message\":\"Person Spotted in the driveway, please check\",\t \"media_player_entity_id\": \"media_player.guest_room_display\",\t \"cache\":\"true\"\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","blockInputOverrides":false,"domain":"tts","service":"speak","x":3344,"y":442,"wires":[["dfd1286d79b8553d"]]},{"id":"5d036f540a7cf592","type":"api-call-service","z":"b2b424ddbd92c0e2","name":"Media Stop","server":"2e31528c.f3973e","version":7,"debugenabled":false,"action":"media_player.turn_off","floorId":[],"areaId":[],"deviceId":[],"entityId":["media_player.guest_room_display","media_player.master_bedroom_display","media_player.downstairs_office_display","media_player.upstairs_display"],"labelId":[],"data":"","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","blockInputOverrides":false,"domain":"media_player","service":"turn_off","output_location":"","output_location_type":"none","x":4104,"y":442,"wires":[[]]},{"id":"dfd1286d79b8553d","type":"delay","z":"b2b424ddbd92c0e2","name":"","pauseType":"delay","timeout":"4","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":3554,"y":442,"wires":[["7b62a977eca43fe1"]]},{"id":"7b62a977eca43fe1","type":"api-call-service","z":"b2b424ddbd92c0e2","name":"Stream Camera","server":"2e31528c.f3973e","version":7,"debugenabled":false,"action":"camera.play_stream","floorId":[],"areaId":[],"deviceId":[],"entityId":["camera.driveway_medium_resolution_channel"],"labelId":[],"data":"{\t \"media_player\": [\t \"media_player.downstairs_office_display\",\t \"media_player.master_bedroom_display\",\t \"media_player.upstairs_display\",\t \"media_player.guest_room_display\"\t ]\t}","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","blockInputOverrides":false,"domain":"camera","service":"play_stream","output_location":"","output_location_type":"none","x":3734,"y":442,"wires":[["34fc28d0d7c302c1"]]},{"id":"34fc28d0d7c302c1","type":"delay","z":"b2b424ddbd92c0e2","name":"","pauseType":"delay","timeout":"6","timeoutUnits":"minutes","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":3914,"y":442,"wires":[["5d036f540a7cf592"]]},{"id":"f4abfbf69ddc08f2","type":"api-current-state","z":"b2b424ddbd92c0e2","name":"Driveway warning test is on","server":"2e31528c.f3973e","version":3,"outputs":2,"halt_if":"on","halt_if_type":"str","halt_if_compare":"is","entity_id":"input_boolean.test_driveway_warning","state_type":"str","blockInputOverrides":true,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entityState"},{"property":"data","propertyType":"msg","value":"","valueType":"entity"}],"for":"0","forType":"num","forUnits":"minutes","override_topic":false,"state_location":"payload","override_payload":"msg","entity_location":"data","override_data":"msg","x":1700,"y":460,"wires":[[],["14d8d3497464e5cb"]]},{"id":"14d8d3497464e5cb","type":"time-range-switch","z":"b2b424ddbd92c0e2","name":"","lat":"35.69052","lon":"-78.29317","startTime":"23:00","endTime":"07:30","startOffset":0,"endOffset":0,"x":1964,"y":462,"wires":[["836795b1e2561ab6"],[]]},{"id":"87d8bc9478388799","type":"api-call-service","z":"b2b424ddbd92c0e2","name":"Turn on Guest Lights","server":"2e31528c.f3973e","version":7,"debugenabled":false,"action":"light.turn_on","floorId":[],"areaId":[],"deviceId":[],"entityId":["light.guest_room_fan_light","light.guest_room_light"],"labelId":[],"data":"","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","blockInputOverrides":false,"domain":"light","service":"turn_on","x":2414,"y":502,"wires":[[]]},{"id":"be94bd0220304c46","type":"api-call-service","z":"b2b424ddbd92c0e2","name":"Phone Notification - Dan","server":"2e31528c.f3973e","version":7,"debugenabled":false,"action":"notify.mobile_app_dan_pixel_8_pro","floorId":[],"areaId":[],"deviceId":[],"entityId":[],"labelId":[],"data":"{\"title\":\"Delivery Person Spotted\",\"message\":\"Delivery person spotted, please check\",\"data\":{\"channel\":\"Delivery Driver\",\"tag\":\"delivery-driveway\",\"image\":\"/media/local/snapshots/driveway_person.jpg\",\"clickAction\":\"/lovelace/cameras\"}}","dataType":"json","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","blockInputOverrides":false,"domain":"notify","service":"mobile_app_dan_pixel_8_pro","x":1690,"y":100,"wires":[[]]},{"id":"836795b1e2561ab6","type":"api-current-state","z":"b2b424ddbd92c0e2","name":"Front Door Locked","server":"2e31528c.f3973e","version":3,"outputs":2,"halt_if":"locked","halt_if_type":"str","halt_if_compare":"is","entity_id":"lock.front_door","state_type":"str","blockInputOverrides":true,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entityState"},{"property":"data","propertyType":"msg","value":"","valueType":"entity"}],"for":"0","forType":"num","forUnits":"minutes","override_topic":false,"state_location":"payload","override_payload":"msg","entity_location":"data","override_data":"msg","x":2164,"y":462,"wires":[[],["d62465aac319339d","87d8bc9478388799"]]},{"id":"94f997700e010570","type":"inject","z":"b2b424ddbd92c0e2","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":3754,"y":382,"wires":[["5d036f540a7cf592"]]},{"id":"67cf298f9251ee69","type":"inject","z":"b2b424ddbd92c0e2","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":3320,"y":360,"wires":[["7b62a977eca43fe1"]]},{"id":"f3c69ea892f4a7a1","type":"comment","z":"b2b424ddbd92c0e2","name":"Delay Occupancy slightly for a better picture","info":"Using Frigate in this instance, I delay\nthe occupancy detection slightly to get \na better picture. Can change this with\nUnifi or another binary sensor such as \na motion sensor.","x":170,"y":220,"wires":[]},{"id":"b1ac5f51d35efd44","type":"comment","z":"b2b424ddbd92c0e2","name":"Check if Package Expected Boolean is True, Rate limit messages","info":"Created a manual boolean for Amazon flex\npackages, since there isn't a way to \npoll Amazon to see if a package is being\ndelievered overnight.\n\nRate limit messages to 1 every 5 minutes","x":610,"y":220,"wires":[]},{"id":"84f474b8634a9775","type":"comment","z":"b2b424ddbd92c0e2","name":"Take a picture, create an initial prompt","info":"Note: I removed my provider from this \nmessage, you can find your provider by\nopening Developer Tools, going to actions,\nselecting \"LLM Vision. Image Analyzer\", and \nat the bottom clicking on \"go to yaml mode\"\nPaste the provider and include quotes\n\nTake a snapshot using a camera integrated\nwith HA. Using that image, send it to\nGemini and ask it to identify delivery\ntrucks or potential packages to avoid\nsending unnecessary messages.\n\nUsing a switch node, check Gemini response\nand route as necessary.\n\nCan use gemini 1.5 flash here","x":990,"y":340,"wires":[]},{"id":"e171df5880556c4a","type":"comment","z":"b2b424ddbd92c0e2","name":"Send a package notification message","info":"Send a simple package notification\n message with image.","x":1710,"y":60,"wires":[]},{"id":"d6a498a79071a0d0","type":"comment","z":"b2b424ddbd92c0e2","name":"Take a picture, create an initial prompt","info":"Output debug message, should be under\ndata -> response text -> response text","x":1430,"y":540,"wires":[]},{"id":"1ad2c0b72b8c2b2d","type":"comment","z":"b2b424ddbd92c0e2","name":"Set Camera Volume","info":"Prep to speak out the camera","x":1670,"y":380,"wires":[]},{"id":"494b23d15d482e4b","type":"comment","z":"b2b424ddbd92c0e2","name":"Test mode Boolean","info":"Send if test mode is off","x":1670,"y":500,"wires":[]},{"id":"80b1f69e9f45a632","type":"comment","z":"b2b424ddbd92c0e2","name":"Ask about the person, trim the response for invalid charactes, set flow variable","info":"Note: I removed my provider from this \nmessage, you can find your provider by\nopening Developer Tools, going to actions,\nselecting \"LLM Vision. Image Analyzer\", and \nat the bottom clicking on \"go to yaml mode\"\nPaste the provider and include quotes\n\nPrompt to ask gemini about the person,\nonce response is received, trim the\nresponse (new line character caused error)\n\nSet a global variable because for some\nreason I can't make it happy\n","x":2050,"y":200,"wires":[]},{"id":"a10b33b4e299326e","type":"comment","z":"b2b424ddbd92c0e2","name":"Test mode Boolean","info":"Send if test mode is off","x":2550,"y":300,"wires":[]},{"id":"5795ab35434ccf44","type":"comment","z":"b2b424ddbd92c0e2","name":"Send a notification to phone with image and message","info":"Send a notification to phone with image\nand Gemini response","x":2660,"y":140,"wires":[]},{"id":"a855f29d0a923f68","type":"comment","z":"b2b424ddbd92c0e2","name":"Check if between time, TTS from Camera","info":"Check if between listed hours and TTS at \nperson on camera using response","x":3080,"y":180,"wires":[]},{"id":"2f23a242efe854c2","type":"api-current-state","z":"b2b424ddbd92c0e2","name":"Front Door Locked","server":"2e31528c.f3973e","version":3,"outputs":2,"halt_if":"locked","halt_if_type":"str","halt_if_compare":"is","entity_id":"lock.front_door","state_type":"str","blockInputOverrides":true,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entityState"},{"property":"data","propertyType":"msg","value":"","valueType":"entity"}],"for":"0","forType":"num","forUnits":"minutes","override_topic":false,"state_location":"payload","override_payload":"msg","entity_location":"data","override_data":"msg","x":3190,"y":260,"wires":[[],["20a4e389b36e32b3"]]},{"id":"aea7763958c6a300","type":"comment","z":"b2b424ddbd92c0e2","name":"Turn on Lights in Room, Set Notification Volume, then stream camera to multiple google nest hubs for 6 minutes","info":"","x":2720,"y":380,"wires":[]},{"id":"5d41038ba4521936","type":"comment","z":"b2b424ddbd92c0e2","name":"Note if using unifi camera","info":"Unifi camera stream was too high for\ngoogle nest hubs and caused buffering.\n\nSuggest using Medium Resolution RSTP stream from\nUnifi instead of high resolution.","x":3750,"y":500,"wires":[]},{"id":"2e31528c.f3973e","type":"server","name":"Home Assistant","version":5,"addon":true,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open","connectionDelay":true,"cacheJson":true,"heartbeat":false,"heartbeatInterval":"30","areaSelector":"friendlyName","deviceSelector":"friendlyName","entitySelector":"friendlyName","statusSeparator":"at: ","statusYear":"hidden","statusMonth":"short","statusDay":"numeric","statusHourCycle":"h23","statusTimeFormat":"h:m","enableGlobalContextStore":true}]
1 Like
Hi all
anyone using LLM Vision with frigate-notify ?
I am using frigate-notify to send a notification via Telegram with a snapshot and a link to local frigate event clip and also to send a webhook to do some other automation.
Is there a way to analyze with LLM vision a jpg or mp4 having only the link and not with a local filename?
Thanks
DEEPSEEK V3 LLM Vision is it possible in the future
Hi all
Iām trying to use llm vision in Nodered using the llmvision.video_analyzer action
The config node is this
and the json data are these
{ "provider": "01JGVZYCCTSVSHKWD5W0701P7B",
"model": "gemini-1.5-flash",
"message": "Descrivi cosa vedi, solo se sono persone.",
"remember": true,
"video_file": "",
"event_id": msg.payload.id,
"max_frames": 5,
"target_width": 1280,
"detail": "high",
"max_tokens": 100,
"temperature": 0.2
}
Iām using msg.payload.id as Frigate event_id
But Iām getting error reporting invalid json
and in the node json visual editor I get this
But I donāt see the error.
Can you help, please?
Thanks
Hi
Trying to install it and getting
Could not connect to the server. Check you API key or IP and port
when adding my Open Ai api key .
any ideas ?
Thanks for the Blueprints. Needing assistance with one part. For the notification on my phone, I am having trouble getting it to open to the image (snapshot). Is {{video if
video != āāāā else image}} the correct entry to use for that to happen?
Hi, i just discover this amazing blueprint and integration. I setup LocalAI and llm integration and the blueprint. Im looking for documentation for the blueprint and LocalAI. Witch model i need to use in LocalAI ? and on the blueprint how i can set the path of my frigate and i how can tell localai to tell me a description ?
Thank you for putting this together! It is extremely helpful (and fun) to use. Iāve set everything up to use Ollama and Llava and things are working great. I got the Events Calendar working as well. Now, iām trying to figure out how to get things set up with Assist so that I can ask it about remembered events, but I canāt figure out how to get this working with Ollama. The documentation only points to how to get things working with Extended OpenAI Conversation, but no other examples. Any help is appreciated!
As a quick follow up question, iād also love to know if there is a way to attach the image to the event in the event calendar so that it isnāt just the LLM response without the contextual image.
Havenāt found a way to attach images directly to calendar events unfortunately. It might be possible to store the images separately and include a reference to the image in the calendar event though.
1 Like
Thought the same thing. Less than ideal, but it is what it is for now.
It should technically be possible to use Ollama with Extended OpenAI Conversation as Ollama has OpenAI compatible endpoints. I canāt get it to use tools properly however. If someone figures that out, please feel free to share!
I know that Qwen2.5 has tools support. Iāll give it a try and see what happens. What iām stuck on, however, is the spec portion. That doesnāt exist for the Ollama integration. Any ideas on how to specify that?