Last Post Here: https://community.home-assistant.io/t/building-a-custom-alexa-skill-part-6-part-2/480368
A lot has transpired in the past 6 months in the world of IT and I have decided to ‘scrap’ what I had been working on (more on that later), and completely re-approach what I was working on. I blame ChatGPT . . . .
Making software “smart” is hard . . . you end up coding a bunch of “if/thens” (assuming you aren’t using ML) . . . testing gets cumbersome along with debugging. So I actually got frustrated and threw in the towel. A couple of weeks ago I had a BRILLIANT (to me) idea:
What if I could use the voice capabilities of Alexa, with the smarts of ChatGPT, and some creative utilization of Node Red? Whelp, that is where we are. If you are looking to go down this route (as I am mid-implementation), this post will kick you down the right road, but you are going to have to do your own prompt engineering to get ChatGPT to “converse how you want”.
So what are my goals for this?
- Voice → code. Alexa can do this for us in spades.
- Voice → action → NodeRed. Grab my voice, and make a webservice call with the appropriate data to “effect” an action in my house.
- Respond with more than just ‘OK’. I want to provide AI with data about my house so that the AI can “come up with a conversation and potential suggested actions”.
And to NOT be as convoluted as this is (even tho I can do it . . . what a mess): Amazon Alexa Smart Home Skill - Home Assistant
WARNING!!! This is gonna look “hacky”, but a non-insignificant amount of thought has gone into this, and I hope that you will see this.
SBOM:
- Tailscale account (there is an official addon) (DuckDNS sucks for me, if you can use DuckDNS with your router over SSL (Https) and it all works for you, more power to you. The steps will be the same). You will need to set it up as a Funnel so that you are publicly exposed. DO NOT CHOSE USERSPACE NETWORKING. Make sure Proxy is on so that you can terminate SSL
- Alex Development Account – This CAN NOT BE the same email address as you amazon account is tied to. Get a new one at gmail (you will see why in a few)
- https:///api/webservice/ access. This HAS TO BE over SSL for your alexa skill to talk to you. This is an Amazon requirement, and I couldn’t get https over port 1880 to work (as I wanted to use the http in node)
- Node Red running, with this ChatGPT node installed in your pallet: node-red-contrib-custom-chatgpt (node) - Node-RED
- PAID ChatGPT account . . . I have mine set up to top off to $10US every time it gets to $5US
- Alexa Media installed and configured via HACS. You need access to the sensor: sensor.last_alexa
In order to prevent having to repeat myself. Follow these:
When creating your skill MAKE SURE TO USE YOUR SECONDARY EMAIL ADDRESS AND NOT THE ONE FOR YOUR ALEXA ACCOUNT. You have been warned . . .
Follow them in order, but ignore the NodeRed port requirements. We are going to use HA’s webservice capabilities for simplicity. Additionally, ignore the “Intent Handler” section. We are changing our approach here too. You will still need “intents”, but you code will be identical for each intent, and only consists of a couple of lines. You will need axios for node http calls . . .
Now assuming you have at least a single Intent, with slots and utterances defined. Here is what you need to “code” within your handler:
var thisURL = baseURL;
custom_utilities.logger(__filename, methodName, "Start of Intent HANDLERINPUT handlerInput", handlerInput);
var noderedcall = await request.fetchURLwithJSON(thisURL, handlerInput.requestEnvelope.request.intent);
Create another file called “http_utils.js”, and include the following code:
//import axios from "axios";
const axios = require('axios');
const fetchURLwithJSON = async (url, jsonobj) => {
try {
let config = {
auth:
{
username: 'xxxxx',
password: 'xxxxx!'
},
params: {
jsonobj
}
};
const { data } = await axios.post(url, config);
return data;
} catch (error) {
console.error('cannot fetch data from NodeRed', error);
}
};
const fetchURL = async (url) => {
try {
const { data } = await axios.get(url, {
auth: {
username: 'ssssss',
password: 'sssssss'
}
});
return data;
} catch (error) {
console.error('cannot fetch quotes', error);
}
};
module.exports= {fetchURL, fetchURLwithJSON};
At the top of your index.js file, with all of your intents, add this:
const request = require('./http_utils');
const baseURL = 'https://<domain>/api/webhook/<yourURLHere>
Ok. Kick back over to Node Red and drop a HomeAssistant WebHook Node. Open the node and set the allowed methods to POST and GET. (post to push data, get to retrieve). Take note of the provided URL. I got online and generated a GUID/UUID with no hyphens and appended it to the URL that was provided. . . . security by obscurity.
I will include the JSON for the flows at the end of this writeup.
Here is my Node Red “input”:
I will dive more into this. But the key components here . . . the switch, and the link out calls.
The switch consists of this:
NOTE!!! My msg path will be different than yours, as it will depend on what you call ‘jsonobj’ (I will show this intent code next). But you do need to validate against the “Intent Call” as it is in your Skill.
I have chosen to use the “mid-flow” links for simplicity. And you will see why on the HouseLightsOnOffIntents.
First TODO Node has this code:
var action = msg.payload.params.jsonobj.slots.action.value;
var entity = msg.payload.params.jsonobj.slots.entity.value;
flow.set("action", action);
flow.set("entity", entity);
var HAaction = "turn_on";
if(action == "off") //Needed due to Slot Value
{
HAaction = "turn_off";
}
flow.set("HALightAction", HAaction);
var lights = flow.get('lightMap');
var error = {};
error.value = false;
var msgg = {};
var name = lights[entity];
if(name === undefined)
{
error.value = true;
return [null, msgg.payload = error.value];
}
else{
flow.set("lightEntity", name);
return [msg,null];
}
My lightmap is used to use “English” to convert to HomeAssistant entity names:
var lights = {
"master bedroom":"",
"masterbedroom":"",
"master bedroom lamps":"light.bedroom_lamps",
"bedroom lamps":"light.bedroom_lamps",
"bedroom lamp":"light.bedroom_lamps",
"master bedrom lamps":"light.bedroom_lamps",
"master":"",
"hallway":"",
"hall":"",
"bedroom":"",
"sunroom":"switch.sunroom_rope_lights_switch",
"sun room":"switch.sunroom_rope_lights_switch",
"livingroom":"light.living_room_all",
"living room":"light.living_room_all",
"kitchen":"light.kitchen",
"bathroom":"light.bathroom",
"bed":"light.bed",
"dining room":"light.dining_room",
"diningroom":"light.dining_room",
"kitchen sink light":"light.kitchen_sink_light",
"kitchen sink":"light.kitchen_sink_light",
"living room lamps":"light.living_room_lamps"
};
flow.set("lightMap", lights);
return msg;
I have it set to be called by a trigger node that only outputs upon start-up, and only does so once. You can see it leveraged in the “TODO” node code.
The TODO node has 2 outputs. This is used in an error situation (as you can see in the code’s ‘returns’) The happy path (all good lets do the thing and get a response), starts with the next function: build payload for light call.
This one is simple:
msg.payload.service = flow.get('HALightAction');
msg.payload.data.entity_id = flow.get('lightEntity');
return msg;
This allows us to use a “pre-configured” service call node. The preconfiguration is nothing more than just stating that the domain is “light”. Everything else is handled via msg properties.
The next function node is where I do the “prompt” engineering for ChatGPT (note, this is subject to change as I work to refine my interactions, but it should get you a good start:
var entity = flow.get('entity');
var action = flow.get('action');
//var entity = 'kitchen';
//var action = 'turn on';
var toPrompt =
"You are the brains for my smart home. You can perform the following tasks: Turn lights on or off. Brighten or dim (on a scale of 0 to 254 or by percent from 0 to 100) or change color temperature (in kelvin from 3500 to 6400). the bedroom lamps are full color philips hue bulbs. All lights, except for the bathroom vanity and the 'sink' leds are philips hue devices. You can also suggest setting a timer to turn lights off after a period of time. My name is Andy, I live at XXXXXXXXXXXXX. Please use this address for any response that may need to be geolocated. Speak as if you are talking to me. I am using the opensource home automation platform called HomeAssistant. Make any suggestions that you deem relevant. Your goals are to be concise, conversational, and to make a relevant suggestion at the same time. Relevant can mean either a suggestion based upon the requested task, or any other suggestion based upon the data provided, or any suggestions due to external influences. Weather information can be found in the first JSON document included. The results of the action that I requested are: The "+ entity+" has been turned "+action+". Using the following data: ";
toPrompt += JSON.stringify(global.get('houseJSON'));
toPrompt += "Your goals are to be concise, conversational, and to make a relevant suggestion at the same time. Relevant can mean either a suggestion based upon the requested task, or any other suggestion based upon the data provided, or any suggestions due to external influences (weather information can be found in the first JSON document). You will use phrasing such as 'I would' or 'I think' or 'You should', you may also use slang as appropriate.";
toPrompt += "The top level node in the above JSON data is the name of the entity in HomeAssistant. As an example, for the 'andy_awake' node. The child, 'entity_id', is the programmatic name of the device. 'state' means what is the status of the device, and 'friendlyName' is the human readable name for the device. Your recommendations are constrained to setting a timer to shut off lights that have been turned on, changing the brightness level of a light, adjusting the thermostat. Your recommendation should be based upon the weather, weather forecast, time of day, sun position, inside and outside temperatures. Please prioritize your recommendations and only make one. If a room has a 'binary_sensor', or a 'motion sensor / light level sensor' the mapping will be found in the following JSON document: ";
toPrompt += JSON.stringify(global.get('SensorMapping'));
toPrompt += " The data in this document is as such: Top level Nodes are the names of rooms (children of 'room'). Alias are other names that you may use for that room. The path ‘Lights.groups’ are the name of groupings of lights (Friendly Name), and the programmatic name of the thing is the value for entity_id. automation means if the entity_id is triggered by an automation script in HomeAssistant or not. members (an array), is an array of the programmatic names of the things in the group. The path ‘room.Bedroom.entities.lights’ as an example, is a list of all the programmatic names of the lights. The path ‘entities.motion’ are the programmatic names of the motion sensors for the room. window represents an array of window open close sensors using the programmatic name of the sensor. climate is the thermostats programmatic name (if available); likewise with the humidity node. Please make your suggestion in the form of a question, using up to 5 sentences. Your response will be in the form of a JSON object with the following structure: {'entity_id':xxx, 'action': yyyy, 'response':zzzz}. replace xxx with the programmatic name of the entity, yyyy with the action you recommend, and zzzz with your response. Ensure that the JSON document is properly formatted. All values are strings. If you identify a suggestion, you will pose it in a yes or no question. Possible values for 'action' in the JSON document to be returned are are: turn_on, turn_off, brighten, dim, set_timer. If you recommend an action of 'set_timer', 'brighten', or 'dim', you will add a child node to this key as to your recommended value for the action and this node will be named 'value'."
msg.payload = toPrompt
return msg;
If you actually read through the entire prompt, you have seen calls to two global variables. As you can’t “store” the history of stuff, nor can ChatGPT make API calls right now, I tell ChatGPT the current state of my house. Some of this is automated, some not so much, and you will see in my “set up” nodes:
Moving Left To Right . . . here is how each node is configured.
The first time you set this up, you will need to set this node to “output on Connect”, as this will set you up for success with the function nodes, and your ChatGTP calls. Disable it afterwards as then each state change will update the Global value, which updates homeJSON when needed
Input To Globals:
var values = {};
values.entity_id = msg.data.entity_id;
values.state = msg.payload;
values.friendlyName = msg.data.old_state.attributes.friendly_name
values.type = msg.data.entity_id.split(".")[0];
var globalname = msg.data.entity_id.split(".")[1];
global.set(globalname, values);
msg.array_key = globalname;
return msg;
This next node will be used as I enhance my prompts to ChatGPT:
var sensorMapping = {
"room":
{
"Bedroom":
{
"alias": ['Master Bedroom', 'Master', 'Bedroom'],
"lights":
{
"groups":
{
"1":
{
"Friendly Name":"Bedroom Lamps",
"entity_id":"light.bedroom_lamps",
"automation": false,
"members": ['light.left_bedroom_lamp','light.right_bedroom_lamp_light']
},
"2":
{
"Friendly Name":"Bed",
"entity_id":"light.bed",
"automation":false,
"members": ['light.right_bedroom_leds_light','light.left_bed_leds']
}
}
},
"entities":
{
"lights":['light.right_bedroom_leds_light',' light.left_bed_leds','light.left_bedroom_lamp','light.right_bedroom_lamp_light'],
"motion":['binary_sensor.lumi_lumi_sensor_motion_aq2_motion_2'],
"window":['binary_sensor.left_master_window_opening_2'],
"climate":"",
"humidity": '',
"lock": "",
"cover": ''
}
},
"Kitchen":
{
"alias": ['Kitchen', 'Kitchen Overhead', 'Kitchen Overhead Lights'],
"lights":
{
"groups":
{
"1":
{
"Friendly Name":"Kitchen",
"entity_id":"light.kitchen",
"automation": false,
"members": ['light.kitchen_right_overhead_light_2','light.left_kitchen_overhead']
},
"2":
{
"Friendly Name":"Counter Lights",
"entity_id":"",
"automation":true,
"members": ['automation.left_of_sink_leds_on','automation.left_of_sink_off','automation.right_of_sink_leds_off','automation.right_of_sink_leds_on']
}
}
},
"entities":
{
"lights":['light.kitchen_right_overhead_light_2','lght.kitchen_left_overhead'],
"motion":['binary_sensor.kitchen_motion_sensor'],
"window":[''],
"climate":"",
"humidity": '',
"lock": "",
"cover": ''
}
},
"Dining Room":
{
"alias": ['Dining', 'Dining Room', 'Dinner Table'],
"lights":
{
"groups":
{
"1":
{
"Friendly Name":"Dining Room",
"entity_id":"light.dining_room",
"members": ['light.dining_room_1_light','light.dining_room_2_light','light.dining_room_4_light_2','light.dining_room_5_light']
}
}
},
"entities":
{
"lights":['light.dining_room_1_light','light.dining_room_2_light','light.dining_room_4_light_2','light.dining_room_5_light'],
"motion":['binary_sensor.kitchen_motion_sensor'],
"window":['binary_sensor.left_dining_room_window_sensor_opening'],
"climate":"climate.thermostat",
"humidity": '',
"lock": "lock.front_door_lock_door_lock",
"cover": ''
}
},
"Living Room":
{
"alias": ['Living Room', 'Family Room'],
"lights":
{
"groups":
{
"1":
{
"Friendly Name":"Living Room All",
"entity_id":"light.living_room_all",
"members": ['light.living_room_overhead_1_light','light.living_room_overhead_2_light_2','light.living_room_overhead_3_light_3']
},
"2":
{
"Friendly Name":"Living Room Lamps",
"entity_id":"light.living_room_lamps",
"members": ['light.left_living_room_light','light.right_living_room_lamp_light']
}
}
},
"entities":
{
"lights":['light.left_living_room_light','light.right_living_room_lamp_light','light.living_room_overhead_1_light','light.living_room_overhead_2_light_2','light.living_room_overhead_3_light_3'],
"motion":['binary_sensor.laundry_room_motion_motion'],
"window":[''],
"climate":'',
"humidity": '',
"lock": "",
"cover": ''
}
},
"Bathroom":
{
"alias": ['Bath Room', 'Throne Room', 'Bathroom', 'Shitter'],
"lights":
{
"groups":
{
"1":
{
"Friendly Name":"Bathroom",
"entity_id":"light.bathroom",
"members": ['light.bathroom_1_light, light.bathroom_2_light_2','light.bathroom_3_light','light.bathroom_4_light_3']
}
}
},
"entities":
{
"lights":['light.bathroom_1_light, light.bathroom_2_light_2','light.bathroom_3_light','light.bathroom_4_light_3','light.bathroom'],
"motion":['binary_sensor.bathroom_motion_sensor_motion'],
"window":[''],
"climate":'',
"humidity": 'sensor.bathroom_humidity_humidity',
"lock": "",
"cover": ''
}
},
"Laundry Room":
{
"alias": ['Laundry Room'],
"lights":
{
"groups":
{
}
},
"entities":
{
"lights":[''],
"motion": ['binary_sensor.laundry_room_motion_motion'],
"window":[''],
"climate":'',
"humidity":'',
"lock":"lock.lock.back_door_lock_door_lock",
"cover":''
}
}
}
};
global.set('SensorMapping', sensorMapping);
return msg;
Globals To House JSON (and for the love of god . . . make sure you have your “exclusion” list done (the if statement below). If you don’t, I’m not gonna be responsible for node crashing on you. Ask me how I know . . . )
Things to note, if you don’t have a lot of entites, you probably won’t have to worry about memory issues (I did run node out of memory), but you really should exclude the global.homeassistant and globlal.houseJSON. Neither of these bring any value to your document. You would just be adding global.houseJSON to its self and homeasssitant contains a bunch of info that isn’t necessary:
var keys = global.keys();
var i = 0;
var toReturn = {};
while( i < keys.length)
{
var key = keys[i];
if(
!(key.includes('houseJSON') || key.includes('speedtest') ||
key.includes('texas') ||
key.includes('usage') ||
key.includes('voltage') ||
key.includes('input_booleans') ||
key.includes('sensors') ||
key.includes('automations') ||
key.includes('homeassistant'))
)
{
//node.warn(keys[i]);
var x = global.get(keys[i]);
var entity = keys[i];
if(i === 0){
toReturn[entity] = x;
}
else
{
var newobj = {};
newobj[entity] = x;
Object.assign(toReturn, newobj);
}
}
else
{
//node.error("invalid key: " + keys[i]) ;
//node.warn(toReturn);
}
i++
}
global.set("houseJSON", toReturn);
return msg; //.payload = global.get(keys[i]);
These 3 nodes work to create global variables that look like this:
The Really nice thing about this approach, is that on each state change my global variables are updated “in real time” so I am always feeding ChatGPT the most recent data.
My House JSON (which is long) looks like this:
If you follow this guide . . . and use some of these “meta” programming techniques . . . your houseJSON will look different than mine (and will probably be called something different too!)
And surprisingly . . . we are essentially “done”. We have the structure and the framework necessary. From here on out it is just following the patterns and adding your “intents” “slots”, and “values” as we expand our integration.
USING OUR SKILL!!!
We have done all this work . . . so now what? You COULD create your own webservice and just have Alexa make those calls for you, but I like this approach. I only have to pass around the data I need to.
So . . . how do you publish . . . I’m not going to dive to far into it, but inside the developer console for an alexa skill (the website). You need to ensure that the Distribution Tab is completely filled out. Use the instructions around icons and privacy URL found at: https://www.home-assistant.io/integrations/alexa.intent/
Now here is the trick . . . remember that this is under your “non-alexa” email? You are going to invite yourself to be a beta tester. Amazon has some documentation on this (and it is accurate). Once you invite yourself to be a beta tester, you can use your skill!!! You can add it just like normal
Here is an example chat GPT response from my prompt (kitchen / on was “asked”):
{
"entity_id": "light.kitchen",
"action": "turn_on",
"response": "I have turned on the lights in the kitchen. Would you like me to adjust the brightness or color temperature?"
}
I still have some engineering to do (from a prompt standpoint), but I’m getting there.
Now, as for the final piece . . . speaking.
All LinkOut nodes are set to return to calling node (the added benefit is in the “yes/no” responses:
Function 18:
var nextAction = global.get('ChatGPTFollowUpData');
//node.warn(nextAction);
msg.payload = {};
msg.payload.nextAction ={};
msg.payload.nextAction = nextAction;
return msg;
Switch:
Lights On link Node
Link Out Node:
The benefit to having all f the link out nodes be return to calling node (when possible) is that I can start to allow complex actions to occur (multiple at once for example: Light on and set to 50% brightness) by just putting in a “switch” and using a call out node. Each Return Node, will send the flow “back up the stack”.
And how do we “speak the response” from ChatGPT?
It sounds like a lot (and it kinda is), but what I haven’t fully thought through is maintaining state . . . “Would you like me to set a timer to turn off the lights in 30 min” “YES”. Once this is solved (as you CAN pass your last request through as a “reminder”), I should be off to the races.