A way to call HA scripts from Google Assistant with parameters without IFTTT

Hey everyone,

wasn’t really sure where to put this, but I’ve been searching for a way to call HA scripts from my Google Home with parameters, but without IFTTT which always comes up as the solution and I haven’t been able to find anyone showing another way of doing it so I went digging on my own and here’s what I found and was able to do (spoiler: I got it mostly working). I hope someone finds it useful and possibly expands (or collaborates). Apologies for the very text-heavy post.

First off, why would I want to do this? Well, let’s say for example I want to create timers, when I’m putting a frozen pizza in the oven, when I’m boiling pasta, … Or possibly something non-cooking related :slight_smile: Google by default knows how to set a timer, however what will happen at the end is it will have that annoying chime and it will go until I dismiss it. I find it very annoying.
A better alternative would to have my lights flash when the timer is over, maybe even if I’m watching something pause that and so on.
So I can create a script in HA that takes in a parameter for the duration of the timer and call it from HA no problem. However if I expose that script to Google Assistant, there doesn’t seem to be a way to tell it how long the timer should be.
IFTTT does have a way to do this, however since they introduced the change to only allow 3 applets for free, so if you don’t want to pay for more IFTTT applets, you’re stuck.

So I initially had the idea of using DialogFlow to create a bot that would call a webhook that passes the parameter(s) to HA scripts. However, since the last time I played with DialogFlow, Google has gone the paid way as well and, while you can have a free trial of DialogFlow, you will ultimately again need to pay for it.

Instead, I saw that Google Actions actually has all the basics for free. You can create actions and give it a bunch of example phrases, define what’s the parameter in the phrase and it will happily employ it’s Natural Language Understanding to create your actions.

So I set off to create a Google Actions project that would handle my custom commands.
One of my concerns here was that, as I had already setup a Google Actions project for my regular HA integration so my Google Assistant and Home Assistant are connected, I called this project (you guessed it) Jarvis and, it is a very minor nitpick, but I didn’t want to change the name to something else, but I also didn’t want this other Google Actions project to be named anything other than Jarvis and you can’t merge a Smart Home project with another project.
Thankfully, this is not a problem, you can just name them both Jarvis and they will both work at the same time! Both my Smart Home Jarvis and Custom commands Jarvis are working at the same time :slight_smile:

Second thing that I was trying to “fix” and unfortunately I was not able to is Google’s intention of bot invocation being that you would say Hey Google, let me talk to Jarvis, then when it gets your bot you tell the bot your command, or you directly say Hey Google, ask Jarvis to [your custom command].
When you setup a command, there is a part to setup deeplink, however that seems to only be for the second style of invocation I mentioned here, even when you set it up you are still not able to just say something like Hey Google, [my custom command].
This makes it a deal breaker for some people and it’s not great for me either, however I have not find a way around it yet.
To rub salt on the wounds, even if you’re ok with the invocation part, Google will always respond with Getting the test version of Jarvis followed by a short pause, followed by a response from your Jarvis.
Would love to hear if someone has a workaround for this and if I find something will post back.

But, moving on to how I set this up.
If you go to Google Actions and create a new project (same as for the regular HA integration for Google Assistant) but when it asks the type of project, select Custom.
I’m not going to go into the detail here, but basically you need to create a custom command, and custom type for your parameters (unless your parameters are numbers or dates which Google has built in types for). If anyone wants details on this part I’m happy to provide the info, but to keep this post … well, it’s already not brief, but you know, not a kilometer of text :slight_smile: I will just say that basically there’s a few things there you can play with and ultimately, you set the webhook for what’s going to handle your requests.

Now comes the HA part where I also need help from someone more familiar with HA to more tightly integrate this into HA as I have it completely separate from HA.
For my webhook, for now, I set up a simple PHP website, as my HA is behind an Nginx that acts as a proxy, and I just added another website to the Nginx that handles Google Assistant commands now.

There are three parts to that webhook handler - the parsing of incoming data from Google Assistant, sending it to Home Assistant and finally responding to Google on what next.
Again, I’m going to omit details for the sake of brevity, but here’s the overview of these.

Google Assistant will pass in data to your webhook as a JSON string in the body of the request, it will have an identifier of which action was invoked, as well as the method of invocation, and it will have “params” JSON object that will have the parameters nicely parsed, in fact here’s what the params object I receive for my custom timer looks like:

"params": {
    "duration": {
        "original": "2 minutes",
        "resolved": "2 minutes"
    }
}

(I have only one parameter, called duration and I said Hey Google, ask Jarvis to set timer for 2 minutes)
It will also send you the full original query as well as a bunch of other useful and less useful info.

So at this point you need to call Home Assistant. In my above example, there’s no easy way for a random person to translate 2 minutes into 00:02:00 that would work in HA, although I suspect there will be a higher proportion of people who’ll be able to handle this seeing as HA does lean into Jinja and things like this.
But anyway, your webhook will need to either process things like this, or just hand off to HA to do the processing.
Here’s the part where I made it completely separate from HA. I have a separate app that calls the HA Rest Api, the URL being:
https://my_ha_url/api/services/script/name_of_my_script
and it’s a POST with a long-live access token in the Authorization header in the format Bearer myLongLivedToken, with the body being a JSON object with my script parameters.
This works, but I feel like it would be nicer if HA was able to handle this (after all, the Smart Home Google Assistant integration works in the same way, Google sends requests and HA responds). However, my Python is rusty as well as I’m still not well-versed in the HA architecture.
Note that this is a blocking call, the API will not return until the script is finished. I didn’t test, but I suspect same as calling from within HA, if you want a non-blocking script call you can use the service to turn on the script.

Anyway, now that you’ve called your HA script, your HA is doing exactly what you want, you’ve done it, right?
Well, almost, you also probably want Google to respond with something, in my example Setting timer for x minutes. Even if you don’t want anything though, you still need to respond to Google as it will expect this (most likely) to be a conversation and it will continue listening for input.
To handle this, you just need to return a JSON object in the body of your response.
Here’s an example of my response to the above mentioned request and I’ll explain below:

{
  "session": {
    "id": "[invocationSessionId]",
    "params": {}
  },
  "prompt": {
    "override": false,
    "firstSimple": {
      "speech": "Setting custom timer for 2 minutes.",
      "text": ""
    }
  },
  "scene": {
    "name": "[invocationSceneName]",
    "slots": {},
    "next": {
      "name": "actions.scene.END_CONVERSATION"
    }
  }
}

Again, I will just go through the most important bits here.
Remember how I said Google will send you some other useful information? Well, one of those is the session id, another one is the scene name.
The session id is just a session id, you take it from the original request and you put it in your response.
The scene name is kind of the same, however there are a couple extra things:

  • the name of the scene is going to be something like actions.scene.START_CONVERSATION so you’ll know what the scene you’re in is, although I don’t think that’s going to be super useful
  • the next object in there tells Google what to do next and, as you can see in my example, I request to end the conversation. This will close the microphone as well as the conversation with the bot. If you don’t do this, the Google Home might still have the bot answering the next query instead of Google Home and it will keep listening for input

Finally, the prompt object of the respone is what Google will say in response. I take the value of the parameter and append it to the phrase Setting custom timer for . Ideally this would actually make sure that’s what’s happening, but this is a POC :slight_smile:
The speech is going to be your spoken text. If you leave text empty, it will be the same as speech, however if you want text (on a display device) to be different than the spoken text you can set them both.

And that’s pretty much it.
Now would be a great time for a nice demo, but I fear no one is going to read to here even, let alone watch a demo which would be more work to do so I’m gonna go now :slight_smile: If anyone found this useful and would like more info, has questions, wants that demo :slight_smile: or anything else, please do let me know.
I think it would be cool if HA could handle calling scripts with parameters in the same “kind-of-out-of-the-box” way that it handles other Google Assistant integrations.

10 Likes

Using Dialogflow allows google assistant to send parameters directly to Home assistant via the JSON. It has built in parsing and is quite flexible.

Too bad that this mode will soon no longer be operational for Google’s choice:
“The Dialogflow Google Assistant integration will be sunsetting in 2023.”

As I wrote in another post:
I think that Google has no VALID reason to eliminate conversational actions… the only things that comes to mind are:

  • they want to OBLIGE you to use ONLY their products;
  • they want to increase CONTROL over the PERSONAL INFORMATIONS that users use/exchange (more and more “big brother” then!)
  • probably, one day, they will charge everything (firebase, and all their products included)…

I really believe that I will abandon my initial project and I will move towards something more useful/valid/less invasive using only Home Assistant and … I will invent something!

2 Likes

Naah, not that long text, you’re fine!

Very interesting actually, thanks.

I’m also interested in this, i’m planning to build an E-TKT, a custom label printer featured in hackaday, with old school embossed labels. Plan is to integrate it into HA with a custom card for printing and possibly some stats perhaps.

BUT then i thought, hey, voice interface could be useful/interesting, can google pass arbitrary strings as arguments to a custom service?
And here we are. But i was kind of hoping for something a bit more straightforward

Hi skrutt!
Do you agree with me?
It is now official: Dialogflow is no longer usable (as before)!

I had found an alternative with IFTTT but, just happen to be… Google has also changed, in August, the integration with IFTTT. I had to recreate all the applets I had made and connect the assistant to the new integration! Now Google forces you to say “OK Goole, ACTIVATE my app”. I found also a workaround for this problem too but…

Unfortunately interaction with voice commands and customizable strings ARE NO LONGER AVAILABLE because … Gooogle doesn’t like them!

Google be cursed!

Well i still have not gotten that far, but yeah I agree…

Seems like another instance of deleting useful features just because. Sad.

Also very strange, I’d think that arbitrary strings as parameter would be one cornerstone of keeping a useful and complete API for a product as a voice assistant.

Also I don’t really see that there is very much going on with Google assistant at all in terms of r&d/features so i wonder if it will be replaced right of if a new one comes in to play

The main problem is that Google wants to force you to use ONLY its products/services and in the way that its developers like it.
So if you don’t like how their voice assistant works (which isn’t the best in life) they stop you from ‘improving’ it and/or making improvements (some time ago it was possible)…

How disgusting!

If this is the only issue then there are ways around it.
You could actually use the native timer and have HA dismiss the timer and create a HA timer with the same time.

There is a HACS integration GitHub - leikoilja/ha-google-home: Home Assistant Google Home custom component that will get add the timer you created to a sensor.
You can use the ID to dismiss the timer before it has ran out, or as I have it set up, five seconds after it has ran out.
Gives me one ring on the speaker and then it’s quiet.

This can also be used to trigger other things like the lights and such that was mentioned.

1 Like

Yes, this also seems Google’s main goal to me.
We will lose voice command connection to IFTTT after August 31st. I need arbitrary strings to be passed as arguments and still do not have an alternative to IFTTT.
I hope that in the near feature there will be more open ASR altenatives to google devices.

1 Like

Typical. This looks like a really workable free solution to something I’ve been wanting to do for a while.
It looks like Google’s about to kill this and (although I haven’t read into it) isn’t this an example of a “conversation action”? (Which they’re also about to kill.)

I’d have been interested in a tutorial and I think you might be surprised at the takeup, if you can somehow make it continue to work!!

I’ve long since abandoned this, sorry, however the newest version of Home Assistant has a Google Assistant SDK integration which very possibly might be a good alternative!

Needless to reiterate that…
Google, in my opinion, has become a stinky SHIT!!!

Google Assistant SDK only allows you to:

  • Sending text commands to Google Assistant to control devices supported by Google Assistant but not by Home Assistant;
  • Broadcast messages to Google Assistant speakers and displays without interrupting music/video;
  • Playback Google Assistant audio response for any query on any media player;
  • Having a conversation with Google Assistant.

The last function, however, does not work well (just read the numerous posts on the subject and … it works even worse in languages ​​other than English).

Google Assistant SDK does not offer what was once the original question and/or answer interaction capabilities with configurable parameters that Dialogflow offered
(before Google decided it was too dangerous for them to continue to let users use it and… decided to completely remove it from games)