Setting up a 100% local smart speaker on an Android tablet using Tasker and Snowboy to handle wake word detection

Configuration Voice Assistant
1

While I eagerly await a built in wake word function, I decided to see what other options are out there. I started looking into open source hotword detection projects and found Snowboy. Combining a Hotword Plugin android app that runs Snowboy and integrates with Tasker (or Automate) allows an android tablet to run local wake word detection that drops seamlessly into the HA Assist service. (Edit on 10/30/2024 - now that built in wake word functionality has dropped I think there’s still value to having Tasker/Automate handle the wake word detection on the satellite device directly rather than having multiple audio streams constantly sending noise to the HA server, bogging down your network and consuming resources on HA server, so I’m updating this a little - also as of Feb 2024 there’s no way to run open wake word on unrooted android)

Creating custom wake word “pmdl” files is possible via multiple methods, but the easiest is to just use this web-based tool maintained by the hotword plugin dev: https://snowboy.jolanrensen.nl/
(The key to making them accurate is to record a lot of audio samples (20+) in a perfectly quiet environment.)

There are also a small selection of fairly accurate universal “umdl” wake words that include:
Hey/Ok Google
Alexa
Jarvis
Computer
Snowboy
Smart Mirror ← my personal favorite

There are a few other oddball umdl files that can be found here: https://github.com/poveteen/jarbas-wake-word-plugin-snowboy/tree/master/jarbas_wake_word_plugin_snowboy/resources/models
(There’s lots of “pmdl” wake words in this repository too, but accuracy on them is hit or miss, mostly miss in my testing, you’ll be better served to make your own)

In short, using any android device (an old phone or tablet - it should run at least android 10, earlier versions of android did a very poor job of handling audio input with multiple apps and the experience was much slower and inconsistent in my trials of devices running Android 9 and earlier), install “Hotword Plugin” which uses snowboy for the local wake word detection, and install “Tasker” or “Automate” to run the HA companion apps Assist service when a hotword is detected. Assuming you have the assist pipeline setup already on your HA server and the HA companion app installed on your device, just create a routine in Tasker/Automate that calls the HA apps assist service whenever Hotwork Plugin detects the wake word and triggers the routine.
Tip: In Tasker set it to launch a new assist service with every trigger so it will take consecutive commands. (Automate has a similar setting)

See below reply for detailed description of how to setup the Tasker routine.

Tasker: https://play.google.com/store/apps/details?id=net.dinglisch.android.taskerm
(It’s a paid app, but well worth it considering what you get for $3.50)

Automate: https://play.google.com/store/apps/details?id=com.llamalab.automate
(A free alternative to Tasker, seems to work just as well, credit to NervousApps for finding this alternative, see his post below for a screenshot of his setup)

HotwordPlugin: https://play.google.com/store/apps/details?id=nl.jolanrensen.hotwordPlugin
(there is an ad supported free version as well)

Snowboy: GitHub - seasalt-ai/snowboy: DNN based hotword and wake word detection toolkit (model generation included)
(Nothing to install here, just linking this for background info)

12 Likes
2

I followed your instructions but got stuck in "have Tasker call the HA apps assist service " part.
Could you elaborate more on this? Should I create a new task in Tasker with home assistant call service “conversation_process”? What should be service data then?

3

I’m assuming you’ve figured out how to setup a hotword in hotword plugin and you’re just struggling with Tasker? I’ve found their naming conventions to be less than intuitive, so here’s a step-by-step:

Create a new “profile” in tasker under the “event” category. For the event, choose “plugin” and select hotword plugin.

It’ll create a blank profile, hit the pencil button to edit the config and it’ll have you select which hot word(s) you want to use as the trigger. Hit the check mark and it’ll take you back to the event config, exit here and it’ll ask if you want to use a predefined task or create a new one.

Create a new task, give it a name when prompted and it’ll let you add tasks by pressing the + button. Add an “app” task, then select “launch app”. It’ll bring up a list of apps on your phone, find home assistant and long press it so it brings up the list of app related sub-tasks, select the one labelled “assist.AssistActivity”.

Check the box labelled “always start new copy”.

Back out of the config and hit the check mark to save and apply everything.

4 Likes
4

Also, just wanted to share that after running this for a while I’ve had the best luck with the “smart mirror” umdl file found here:

Using hotwords like “jarvis” or “hey computer” was producing too many false positives, and using “alexa” or “hey google” wasn’t appealing since the point (for me at least) was to get away from the corporate branding and control.

5

Thanks for detailed tutorial. I figured it out already and noticed the same behaviour (many false positives) with my custom hotword. I thought using “offical” ones would solve the issue, but your post suggest it isn’t so simple. I’ll keep using this solution until we have offical hotword from HA Assist, though.

6

I only recently started to mess around with the custom wake words, as I thought I’d need to run the training application myself, not realizing that the hotword plugin developer runs it and links to it in the app.

So far I haven’t done any custom words with more than a dozen samples, how many samples did you use for your custom hot words, and did you notice any accuracy increase with more samples?

7

I thought that there are only 3 samples possible…
With 3 samples my experience is very bad. I’ll probably stick to hey google hotword for now.
There are many PR’s in HA repo on github regarding wakeword, so I think it is coming soon.

8

Sir, you are a genius! I was searching for a similar solution for soooo long (I have many older android devices I want to repurpose), until I searched for “tasker wake word”. Thank you very much!

1 Like
9

So I was already using IP Camera app to get audio/video stream to frigate, then I found this but haven´t been able to get it working yet: Add wake word support · Issue #5 · AlexxIT/StreamAssist · GitHub

If that wakeword functionality is added, this could potentially enable full wakeword enabled assist on any tablet :smiley:

10

I got this to trigger the assistant, but it never stops listening when the assistant is triggered, it just keeps listening unless I press the microphone icon manually.

11

I had a similar problem at first, seems it was related to my whisper voice model. If there was any background noise like a fan or a dishwasher then the “tiny” and “base” models would assume that was close enough to speech and keep listening until it timed out. Increasing to the “small” voice model and increasing the beam size mostly eliminated that issue.

Can’t say if your issue is caused by the same thing, if yours won’t ever stop listening - even when there’s no background noise - then it may be something different like the app not having mic access permission.

12

I don’t think its mic permission because when I do press the mic button manually, after its been trigger by hot word, it picks up what I said just fine.

It also triggers all the time even when not saying the hot word lol…

I dig around in the settings today and see if I can adjust anything to fix both the issues I am having.

13

I found that using the universal “umdl” wake words cut down on a lot of the false positives, though even with those there’s some variability, I found that the “alexa” umdl had an occasional false positive whereas the “smart mirror” umdl had basically none (but also would frequently ignore my wife).

That being said I think I finally cracked the code on how to get semi-accurate custom “pmdl” wake words. I used the wake word creator here (https://snowboy.jolanrensen.nl/) and recorded 24 samples of myself in a completely silent environment. That seemed to do the trick, I’ve been running it for a few hours with people talking around it and it only had 1 false positive and responds about 90-95% of the time.

Edit a few days later - the custom wake words definitely have a higher false positive rate, I’d say 3-4 false positives per day, which is livable. I’m also running two custom wake words simultaneously, one for my wife and one for me, I don’t know if one or both are causing them, but having two undoubtedly increases the chances of a false positive match.

1 Like
14

You can use automate instead of tasker. It is free.

15

I’ve never used Automate before so if I’m doing something wrong please let me know, but it looks like the hotword detection in the app uses an old version of Google Now and only responds to “Ok Google”.

16

Screenshot_20231212-144931_Automate
Screenshot_20231212-144931_Automate1080×2280 119 KB

Here is my ,working flow, it uses only 3 blocks, others are just test.

3 Likes
17

Ahh, so you’re still using hotword plugin to handle the wake word detection, not the built in hotword detection block. Got it, thanks for the clarification, I’ll amend the main post to add Automate as an option in place of Tasker.

1 Like
18

Thank you so much, didn’t know about the subtasks of apps by holding on the app icon

19

Thanks for posting this. I am on the hunt for a way to replace my Amazon Echo devices using a tablet and a modified HA display that I’m working on. Wakeword on device is super important to me and this ‘project’ seems like it has a lot of potential and coincides with my wants. It’d be great if this functionality would be rolled into the companion app but who knows when or if that will happen.

Now that it’s been some time, can you update on your experience? Are you still using this daily? Do you find it to be reliable? Have you added AI to your pipeline for additional functionality?

1 Like
20

Yes I’m still using it daily, I have two devices running with this workaround, one tablet/dashboard in my main room, and an old phone in my kitchen that was supposed to also be a digital photo frame.

There’s two ways I could take your question, if you’re asking specifically about the wakeword piece I can tell you it works flawlessly. If you’re asking about the overall experience of running an HA voice assistant on an android device I have a few complaints.

It’s definitely reliable enough to be useful, and all of my issues have been specific to the HA app, not the wakeword automation. In a quiet environment whispers STT is fairly accurate (about 95% of commands are understood), when other people are talking in the background the wakeword will still work flawlessly but it’s a 50/50 chance whisper will understand it. Unless you have 3-5 seconds of silence after a command it may not be sure when to stop listening and eventually just times out. I haven’t used any other voice assistants extensively so maybe that’s to be expected. I’m using the small-int8 whisper model with beam size 3 for reference.

I found that running fotoo as the photo frame app in the foreground seemed to somehow cause HA to not be able to access the device audio. The wakeword would function normally while the photos were displaying, but the HA assist dialog would be unable to hear the command.

I haven’t set up AI yet, waiting for an implementation that’s friendlier to non-tech inclined people. That would certainly help resolve another issue I’ve had with using HA as a voice assistant, which is that the built in commands are limited and require specific verbiage. My wife has attempted to use it but was frustrated by the fact that commands have to be phrased particularly ways and she could never remember exactly how to ask it what she wanted.