AppDaemon + Alexa - Multi Step Interactions!

I’ve been playing around with Alexa since I added basic support for her in AppDaemon Apps - and now I have an App working with a multi-step skill. The subject matter isn’t related to HASS, but it very easily could be - imagine a multi step discussion to set up a light schedule or pick a scene out of a list - it makes for a more natural and easier way to interact with voice for more elaborate automations.

Once I get the code a little more polished I’ll move it into the AppDaemon base code in a future version.

Here is a sample of the code you would write in your App - still working on it though so it might change :slight_smile:

    def BeerIntent(self, data):

        dialog_state = self.get_alexa_dialog_state(data)
        self.log(dialog_state)
        if dialog_state == "STARTED":
            return self.format_delegate()
        elif dialog_state == "IN_PROGRESS":
            return self.format_delegate()
        elif dialog_state == "COMPLETED":
            type = self.get_alexa_slot_value(data, slot="type")
            time = self.get_alexa_slot_value(data, slot="time")
            resp = "OK, holding your {} for the next {} minutes".format(type, time)
            return self.format_response(speech=resp, card=resp,title="Beer Situation")
8 Likes

Is this still functional? I would like the ability for multi step functions but have not used appdaemon except for ha dashboards.

Yes it should still work fine (pauses to test it - it works!)

I was planning to get the Alexa helper functions finished and into the AppDaemon API but haven’t done that yet, but what I did should be in the Alexa example App.

Do you have a link to the example app.I cant seem to find the one with this example.

Sorry - its not in the version in the repository - here is is:

import appdaemon.appapi as appapi
import random
import globals

class Alexa(appapi.AppDaemon):

    def initialize(self):
        self.register_endpoint(self.api_call)

    def api_call(self, data):
        intent = self.get_alexa_intent(data)

        if intent is None:
            self.log("Alexa error encountered: {}".format(self.get_alexa_error(data)))
            return "", 201

        intents = {
            "AMAZON.CancelIntent": self.AmazonCancelIntent,
            "BeerIntent": self.BeerIntent,
            "StatusIntent": self.StatusIntent,
            "LocateIntent": self.LocateIntent,
            "MorningModeIntent": self.MorningModeIntent,
            "DayModeIntent": self.DayModeIntent,
            "EveningModeIntent": self.EveningModeIntent,
            "NightModeIntent": self.NightModeIntent,
            "NightModeQuietIntent": self.NightModeQuietIntent,
            "SecureHouseIntent": self.SecureHouseIntent,
            "QueryHouseIntent": self.QueryHouseIntent,
            "QueryGarageDoorIntent": self.QueryGarageDoorIntent,
            "QueryHeatIntent": self.QueryHeatIntent,
            "ConditionsIntent": self.ConditionsIntent,
            "TravelIntent": self.TravelIntent,
        }

        if intent in intents:
            resp = intents[intent](data)
            if resp["type"] == "answer":
                response = self.format_alexa_response(speech = resp["speech"], card = resp["card"], title = resp["title"])
                self.log("Recieved Alexa request: {}, answering: {}".format(intent, resp["speech"]))
            elif resp["type"] == "delegate":
                response = self.format_alexa_delegate_directive()
                self.log("Recieved Alexa request, answering with follow-on")
            else:
                response = self.format_alexa_response(speech="Invalid response type".format(intent))
                self.log("Invalid response type")
        else:
            response = self.format_alexa_response(speech = "I'm sorry, the {} does not exist within AppDaemon".format(intent))

        #self.log(response)

        return response, 200

    def format_response(self, speech, card, title):
        return {"type": "answer", "speech": speech, "card": card, "title": title}

    def format_delegate(self, slots = None):
        response = {"type": "delegate"}
        return response


    def format_alexa_delegate_directive(self, slots = None):
        response = \
            {
                "version": "1.0",
                "response":
                    {
                        "outputSpeech": None,
                        "card": None,
                        "directives": [{
                            "type": "Dialog.Delegate",
                            "updatedIntent": None
                        }],
                        "reprompt": None,
                        "shouldEndSession": False
                    }
            }

        #if slots:response["response"]["directives"][0]["updatedIntent"] = \
        # {
        #     "name": "BeerIntent",
        #     "confirmationStatus": "NONE",
        #     "slots": {
        #         "type": {
        #             "name": "type",
        #             "value": "lager",
        #             "confirmationStatus": "NONE"
        #         },
        #         "time": {
        #             "name": "time",
        #             "value": 10,
        #             "confirmationStatus": "NONE"
        #         }
        #     }
        #
        # }

        return response

    def get_alexa_dialog_state(self, data):
        if "request" in data and "dialogState" in data["request"]:
            return (data["request"]["dialogState"])
        else:
            return None

    def BeerIntent(self, data):

        #pretty = appapi.Formatter()
        #self.log(pretty(data))
        dialog_state = self.get_alexa_dialog_state(data)
        self.log(dialog_state)
        if dialog_state == "STARTED":
            return self.format_delegate()
        elif dialog_state == "IN_PROGRESS":
            return self.format_delegate()
        elif dialog_state == "COMPLETED":
            type = self.get_alexa_slot_value(data, slot="type")
            time = self.get_alexa_slot_value(data, slot="time")
            resp = "OK, holding your {} for the next {} minutes".format(type, time)
            return self.format_response(speech=resp, card=resp,title="Beer Situation")

    def LocateIntent(self, data):

        user = self.get_alexa_slot_value(data, slot = "User")

        if user is not None:
            if user.lower() == "jack":
                response = self.Jack()
            elif user.lower() == "andrew":
                response = self.Andrew()
            elif user.lower() == "wendy":
                response = self.Wendy()
            elif user.lower() == "brett":
                response = "I have no idea where Brett is, he never tells me anything"
            else:
                response = "I'm sorry, I don't know who {} is".format(user)
        else:
            response = "I'm sorry, I don't know who that is"

        return self.format_response(response, response, "Where is {}?".format(user))


    def AmazonCancelIntent(self, data):
        response = "OK, cancelled"
        return self.format_response(response, response, "Cancel")

    def QueryHouseIntent(self, data):
        security = self.get_app(self.args["apps"]["secure"])
        secure, response = security.query_house({"type": "query", "caller": "alexa"})
        return self.format_response(response, response, "Query House Security")

    def SecureHouseIntent(self, data):
        security = self.get_app(self.args["apps"]["secure"])
        secure, response = security.query_house({"type": "secure", "caller": "alexa"})
        return self.format_response(response, response, "Secure House")

    def MorningModeIntent(self, data):
        self.fire_event("MODE_CHANGE", mode = "Morning")
        response = "Good Morning"
        return self.format_response(response, response, "Morning Mode")

    def DayModeIntent(self, data):
        self.fire_event("MODE_CHANGE", mode = "Day")
        response = "Good Day"
        return self.format_response(response, response, "Day Mode")

    def EveningModeIntent(self, data):
        self.fire_event("MODE_CHANGE", mode = "Evening")
        response = "Good Evening"
        return self.format_response(response, response, "Evening Mode")

    def NightModeIntent(self, data):
        modes = self.get_app(self.args["apps"]["modes"])
        response = modes.night(False, True)
        return self.format_response(response, response, "Night Mode")

    def NightModeQuietIntent(self, data):
        modes = self.get_app(self.args["apps"]["modes"])
        response = modes.night(True, True)
        return self.format_response(response, response, "Night Mode Quiet")

    def TravelIntent(self, data):

        if self.now_is_between("05:00:00", "12:00:00"):
            commute = self.entities.sensor.wendy_home_to_work.state
            direction = "from home to work"
            response = "Wendy's commute time {} is currently {} minutes".format(direction, commute)
        elif self.now_is_between("12:00:01", "20:00:00"):
            commute = self.entities.sensor.wendy_work_to_home.state
            direction = "from work to home"
            response = "Wendy's commute time {} is currently {} minutes".format(direction, commute)
        else:
            response = "Are you kidding me? Don't go to work now!"

        return self.format_response(response, response, "Travel Time")

    def StatusIntent(self, data):
        response = self.HouseStatus()
        return self.format_response(response, response, "House Status")

    def ConditionsIntent(self, data):
        temp = float(self.entities.sensor.side_temp_corrected.state)
        if  temp <= 70:
            response = "It is {} degrees outside. The conditions have been met.".format(temp)
        else:
            response = "It is {} degrees outside. The conditions have not been met.".format(temp)

        return self.format_response(response, response, "Conditions Query")

    def QueryGarageDoorIntent(self, data):
        response = self.Garage()
        return self.format_response(response, response, "Is the garage open?")

    def QueryHeatIntent(self, data):
        response = self.Heat()
        return self.format_response(response, response, "Is the heat on?")

    def HouseStatus(self):

        status = self.Heat()
        status += "The downstairs temperature is {} degrees farenheit,".format(self.entities.sensor.downstairs_thermostat_temperature.state)
        status += "The upstairs temperature is {} degrees farenheit,".format(self.entities.sensor.upstairs_thermostat_temperature.state)
        status += "The outside temperature is {} degrees farenheit,".format(self.entities.sensor.side_temp_corrected.state)
        status += self.Garage()
        status += self.Wendy()
        status += self.Andrew()
        status += self.Jack()

        return status

    def Garage(self):
        return "The garage door is {},".format(self.entities.cover.garage_door.state)

    def Heat(self):
        return "The heat is switched {},".format(self.entities.input_boolean.heating.state)

    def Wendy(self):
        location = self.get_state(globals.wendy_tracker)
        if location == "home":
            status = "Wendy is home,"
        else:
            status = "Wendy is away,"

        return status

    def Andrew(self):
        location = self.get_state(globals.andrew_tracker)
        if location == "home":
            status = "Andrew is home,"
        else:
            status = "Andrew is away,"

        return status

    def Jack(self):
        responses = [
            "Jack is asleep on his chair",
            "Jack just went out bowling with his kitty friends",
            "Jack is in the hall cupboard",
            "Jack is on the back of the den sofa",
            "Jack is on the bed",
            "Jack just stole a spot on daddy's chair",
            "Jack is in the kitchen looking out of the window",
            "Jack is looking out of the front door",
            "Jack is on the windowsill behind the bed",
            "Jack is out checking on his clown suit",
            "Jack is eating his treats",
            "Jack just went out for a walk in the neigbourhood",
            "Jack is by his bowl waiting for treats"
        ]

        return random.choice(responses)

    def Response(self):
        responses = [
            "OK",
            "Sure",
            "If you insist",
            "Done",
            "No worries",
            "I can do that",
            "Leave it to me",
            "Consider it done",
            "As you wish",
            "By your command",
            "Affirmative",
            "Yes oh revered one",
            "I will",
            "As you decree, so shall it be",
            "No Problem"
        ]

        return random.choice(responses)


1 Like

That looks great!

Can you please elaborate on the yaml configuration part in order to navigate alexa’s requests to this code?

Thank you.

I’m interested in knowing more about all of this. How is AppDeamon configured in the configuration.yaml file? Does anything need to be installed in the base OS? I will have to get my Python skills up, it’s been a long time… Thanks!

AppDaemon is a standalone package that needs to be installed and configured outside of Home Assistant. ,Full docs here:

http://appdaemon.readthedocs.io/en/latest/

Thanks. I have AppDeamon up and running in HASSio. Now, what do I need to do in configuration.yaml, if anything? I already have ALEXA:. Do I need anything in intent_script: section? My intents do not work and Alexa tells me that “This intent is not yet configured within Home Assistant” when I try to test it.

If you are planning on using AppDaemon for Alexa support it is completely separate from home assistant, you don;t need to configure anything. One caveat is that you will probably need to configure nginx to make this work.

There is more information here:

http://appdaemon.readthedocs.io/en/latest/APPGUIDE.html#alexa-support

1 Like

Thanks. I think I’m progressing. I installed NGINX and started it using the following default parameters:

{
  "domain": "mydomain.duckdns.org",
  "certfile": "fullchain.pem",
  "keyfile": "privkey.pem"
}

I read the documentation and I understand most of it. I access HA using an SSL cert provided by Let’s Encrypt (and that works well). But what configuration do I need locally in the NGINX add-on to forward Alexa calls to AppDeamoon?

I have the following in the server section for my SSL:

    location /api/appdaemon/ {
    allow all;
    proxy_pass http://localhost:5000;
    proxy_set_header Host $host;
    proxy_redirect https:// https://;

this maps the specific URL over to AppDaemon while still alowing access to the rest of hass on the same port.

Thank you. I understand but I have no clue on how to apply that configuration to the NGINX add-on options within HASS. Any clue on how to format it in JSON?

If you are asking about hass.io I’m sorry I don’t know, I don’t use it.

@aimc Thank you for this example and the documentation for it, I was looking for a way to make conversations with home assistant for a long time. So again, thank you. :slight_smile:

I was able to create conversation altering the response sent back to alexa with a reprompt object and the shouldEndSession flag set to false.

Everything works extremely well! :sunglasses:

I only have one problem, if the user doesn’t reply with in the max reprompts allowed, alexa will end the session sending the skill a request of type SessionEndedRequest. Alexa doesn’t expect responses for this type of request.
And that seems to be a problem with AppDaemon API, I’m unable to “Not send a response”.

If I send any kind of response to the SessionEndedRequest, alexa will tell me there was a problem with the requested skill.
If I don’t send a response at all, the AppDaemon API will send some kind of default response (which will result in alexa telling me there’s something wrong with my skill again) and it will raise the following error (which makes sense since I’m not returning anything):

File "/usr/lib/python3.6/site-packages/appdaemon/api.py", line 40, in call_api
    ret, code = yield from ha.dispatch_app_by_name(app, args)
TypeError: 'NoneType' object is not iterable

Do you happen to know a way around this situation?
A way to “stop the current execution” when this type or request arrives?

Thank you!

At this point you know all I do - I experimented for a while to get that far but shelved it for other priorities. The good news is you have access to all you need in the App so you at least have the tools to figure it out if you are so inclined if you get to it before I do. If you do make any progress please let me know - my intention was to move these helper functions into AD when they are complete.

1 Like

I’m sure I’ll figure it out eventually and I will make sure to update you and this wonderful community.
Thank you so much!

If it helps anyone, this is the code I have so far.
I added a couple of helper functions to allow me to accomplish what I needed.

import appdaemon.appapi as appapi

class HomeControl(appapi.AppDaemon):

    def initialize(self):
        self.register_endpoint(self.api_call)

    def api_call(self, data):
        allowedApplications = []
        allowedDevices = []

        applicationId = data["context"]["System"]["application"]["applicationId"]
        if ("deviceId" in data["context"]["System"]["device"]):
            deviceId = data["context"]["System"]["device"]["deviceId"]
        else:
            deviceId = None

        if (allowedApplications and applicationId not in allowedApplications):
            self.log("unauthorized applicationId {}.".format(applicationId))
            return "", 404

        if (allowedDevices and deviceId and deviceId not in allowedDevices):
            self.log("unauthorized applicationId {}.".format(applicationId))
            return "", 404

        requestType = data["request"]["type"]

        if (requestType == "LaunchRequest"):
            response = self.LaunchRequest(data)
            return response, 200

        elif (requestType == "IntentRequest"):
            intent = self.get_alexa_intent(data)

            intents = {
                "LocatePhoneIntent": self.LocatePhoneIntent,
                "AMAZON.CancelIntent": self.CancelIntent,
                "AMAZON.HelpIntent": self.HelpIntent,
                "AMAZON.NoIntent": self.NoIntent,
                "AMAZON.StopIntent": self.StopIntent,
                "AMAZON.YesIntent": self.YesIntent,
                "IdentifyIntent": self.IdentifyIntent
            }

            if intent in intents:
                response = intents[intent](data)
            else:
                self.log("unknown intent {}.".format(intent))
                response = self.tellText(data, "Hmmm... I'm not familiar with the intent {}.".format(intent))

            return response, 200

        elif (requestType == "SessionEndedRequest"):
            if (data["request"]["reason"] == "error"):
                self.log("Alexa error encountered: {}".format(self.get_alexa_error(data)))

        else:
            self.log("unknown request type {}.".format(requestType))
            response = self.tellText(data, "Hmmm... I'm not familiar with the request type {}.".format(requestType))
            return response, 200

    #########################
    ######## Intents ########
    #########################
    def LaunchRequest(self, data):
        return self.askText(data, "Hey boss! How can I help you?", "Not that you need it. But you can ask me for help, if you want...")

    def LocatePhoneIntent(self, data):
        return self.tellText(data, "this is a test")

    def CancelIntent(self, data):
        return self.tellText(data, "this is a test")

    def HelpIntent(self, data):
        response = self.askSSML(data, "<speak>Try asking me to locate any member of your household.</speak>",
            "<speak>If you need any more help, please check Tomer <say-as interpret-as='spell-out'>fi</say-as> github repository for further instructions."+
            " The link is in a card in your alexa app.</speak>")

        self.setSimpleCard(response, "TomerFi's Github repository", "https://github.com/TomerFi")

        return response

    def NoIntent(self, data):
        return self.tellText(data, "this is a test")

    def StopIntent(self, data):
        return self.tellText(data, "this is a test")

    def YesIntent(self, data):
        return self.tellText(data, "this is a test")

    def IdentifyIntent(self, data):
        return self.tellSSML(data, "<speak>I'm a custom skill designed to help you control and monitor your Home Assistant environment."+
            " I can check stuff out for you<break time='200ms'/>, I can do stuff for you <break time='200ms'/>"+
            " and I can report back you with text supporting <say-as interpret-as='spell-out'>ssml</say-as> tags</speak>") 

    #########################
    ######## Helpers ########
    #########################
    def getSessionAttributes(self, data):
        if ("attributes" not in data["session"]):
            return {}
        else:
            return data["session"]["attributes"]

    def setSessionAttribute(self, response, key, value):
        response["sessionAttributes"][key] = value

    def setSimpleCard(self, response, title, content):
        response["response"]["card"] = {
            "type": "Simple",
            "title": title,
            "content": content
        }

    def setStandardCard(self, response, title, text, smallImageUrl, largeImageUrl):
        response["response"]["card"] = {
            "type": "Standard",
            "title": title,
            "text": text,
            "image": {
                "smallImageUrl": smallImageUrl,
                "largeImageUrl": largeImageUrl
            }
        }

    def tellText(self, data, prompt):
            response = {
                "outputSpeech": {
                    "type": "PlainText",
                    "text": prompt
                },
                "shouldEndSession": True
            }

            sessionAttributes = self.getSessionAttributes(data)

            return {
                "version": "1.0",
                "response": response,
                "sessionAttributes": sessionAttributes
            }

    def askText(self, data, prompt, reprompt):
            response = {
                "outputSpeech": {
                    "type": "PlainText",
                    "text": prompt
                },
                "reprompt": {
                    "outputSpeech": {
                        "type": "PlainText",
                        "text": reprompt
                    }
                },
                "shouldEndSession": False
            }

            sessionAttributes = self.getSessionAttributes(data)

            return {
                "version": "1.0",
                "response": response,
                "sessionAttributes": sessionAttributes
            }

    def tellSSML(self, data, prompt):
            response = {
                "outputSpeech": {
                    "type": "SSML",
                    "ssml": prompt
                },
                "shouldEndSession": True
            }

            sessionAttributes = self.getSessionAttributes(data)

            return {
                "version": "1.0",
                "response": response,
                "sessionAttributes": sessionAttributes
            }

    def askSSML(self, data, prompt, reprompt):
            response = {
                "outputSpeech": {
                    "type": "SSML",
                    "ssml": prompt
                },
                "reprompt": {
                    "outputSpeech": {
                        "type": "SSML",
                        "ssml": reprompt
                    }
                },
                "shouldEndSession": False
            }

            sessionAttributes = self.getSessionAttributes(data)

            return {
                "version": "1.0",
                "response": response,
                "sessionAttributes": sessionAttributes
            }

I just want to point out, everything is working great as long as the user replies!

BTW: I’m working with Hass.io. Not that it matters, just wanted to point it out in case someone else with the same installation will happen to see this post.

1 Like

i already have complete conversations, that can go on and on.
also i have al responses in args so they can be translated.

I don’t have a problem creating the multi step interactions with AppDaemon as long as the user responds to alexa’s question and takes part of the conversation.
It’s when the user doesn’t respond then the problem occurs.

How did you get passed it it you don’t mind me asking?

1 Like

hmm, i think i misunderstood that part.
when the user doesnt respond the conversation stops.
thats only logical, because a conversation is ababab and not aaaaa
why would you like alexa to respond 2 times without user interaction?