Setting up a 100% local smart speaker on an Android tablet using Tasker and Snowboy to handle wake word detection

Configuration Voice Assistant
1

TL;DR - Using any android device that runs Android 10+:

  1. Install “Hotword Plugin” (link below)
  2. Install “Tasker” or “Automate” (link below)
  3. Create an automation routine to call the HA companion apps “assist” function whenever the wakeword is detected.

Long explanation - as of this June 2024 update, there is still no way to detect wakewords locally/on-device using the official HA companion app for android. The only option is to use a 3rd party app to handle wakeword detection and create an automation routine with Tasker/Automate to connect it to the HA companion app’s “assist” function.

It’s also worth noting that there are no 3rd party android apps that use openwakeword, so you won’t be able to integrate this method with openwakeword on your HA voice pipeline or use your .tflite wakewords. The method I’ve linked here uses Snowboy as the wakeword backend.

(Should a method for running openwakeword or wyoming-satellite locally on Android become available this post will be updated)

Combining the Hotword Plugin android app with Tasker (or Automate) allows an android tablet to run local wake word detection that drops seamlessly into the HA Assist service.

You can create custom wake words via multiple methods, the easiest being to just use this web-based tool maintained by the hotword plugin dev: https://snowboy.jolanrensen.nl/
(The key to making them accurate is to record a lot of audio samples (20+) in a perfectly quiet environment.)

There are a small selection of fairly accurate universal “.umdl” wake words that include:
Hey/Ok Google
Alexa
Jarvis
Computer
Snowboy
Smart Mirror ← my personal favorite
Etc
Link to wakeword repository: jarbas-wake-word-plugin-snowboy/jarbas_wake_word_plugin_snowboy/resources/models at master · poveteen/jarbas-wake-word-plugin-snowboy · GitHub
(The “.pmdl” file wake words in this repository may not be very high accuracy, you may want to make your own)

You can use any android device running at least android 10, earlier versions of android did not allow simultaneous mic access to multiple apps.

This method assumes you have the voice assistant pipeline setup already on your HA server and the HA companion app installed on your device. See the post below for details on how to setup a routine in Tasker/Automate that calls the HA apps assist service whenever Hotwork Plugin detects the wake word and triggers the routine.
Tip: In Tasker set it to launch a new assist service with every trigger so it will take consecutive commands. (Automate has a similar setting)

-Links-

Tasker: https://play.google.com/store/apps/details?id=net.dinglisch.android.taskerm
(It’s a paid app, but well worth it considering what you get for $3.50)

Automate: https://play.google.com/store/apps/details?id=com.llamalab.automate
(A free alternative to Tasker, seems to work just as well, credit to NervousApps for finding this alternative, see his post below for a screenshot of his setup)

HotwordPlugin: https://play.google.com/store/apps/details?id=nl.jolanrensen.hotwordPlugin
(there is an ad supported free version as well)

Snowboy: GitHub - seasalt-ai/snowboy: DNN based hotword and wake word detection toolkit (model generation included)
(Nothing to install here, just linking this for background info)

ALTERNATIVE ANDROID WAKEWORD OPTION:

This is an alternative method that bypasses the HA companion app and just streams continuous audio to your HA server, where it can be handled like any other voice satellite.

18 Likes
2

I followed your instructions but got stuck in "have Tasker call the HA apps assist service " part.
Could you elaborate more on this? Should I create a new task in Tasker with home assistant call service “conversation_process”? What should be service data then?

3

I’m assuming you’ve figured out how to setup a hotword in hotword plugin and you’re just struggling with Tasker? I’ve found their naming conventions to be less than intuitive, so here’s a step-by-step:

  1. Create a new “profile” in tasker under the “event” category. For the event, choose “plugin” and select hotword plugin.

  2. It’ll create a blank profile, hit the pencil button to edit the config and it’ll have you select which hot word(s) you want to use as the trigger.

  3. Hit the check mark and it’ll take you back to the event config, exit here and it’ll ask if you want to use a predefined task or create a new one.

  4. Create a new task, give it a name when prompted and it’ll let you add tasks by pressing the + button.

  5. Add an “app” task, then select “launch app”. It’ll bring up a list of apps on your phone, find home assistant and long press it so it brings up the list of app related sub-tasks, select the one labelled “assist.AssistActivity”.

  6. Check the box labelled “always start new copy”.

  7. Back out of the config and hit the check mark to save and apply everything.

8 Likes
4

Also, just wanted to share that after running this for a while I’ve had the best luck with the “smart mirror” umdl file found here:

Using hotwords like “jarvis” or “hey computer” was producing too many false positives, and using “alexa” or “hey google” wasn’t appealing since the point (for me at least) was to get away from the corporate branding and control.

5

Thanks for detailed tutorial. I figured it out already and noticed the same behaviour (many false positives) with my custom hotword. I thought using “offical” ones would solve the issue, but your post suggest it isn’t so simple. I’ll keep using this solution until we have offical hotword from HA Assist, though.

6

I only recently started to mess around with the custom wake words, as I thought I’d need to run the training application myself, not realizing that the hotword plugin developer runs it and links to it in the app.

So far I haven’t done any custom words with more than a dozen samples, how many samples did you use for your custom hot words, and did you notice any accuracy increase with more samples?

7

I thought that there are only 3 samples possible…
With 3 samples my experience is very bad. I’ll probably stick to hey google hotword for now.
There are many PR’s in HA repo on github regarding wakeword, so I think it is coming soon.

8

Sir, you are a genius! I was searching for a similar solution for soooo long (I have many older android devices I want to repurpose), until I searched for “tasker wake word”. Thank you very much!

1 Like
9

So I was already using IP Camera app to get audio/video stream to frigate, then I found this but haven´t been able to get it working yet: Add wake word support · Issue #5 · AlexxIT/StreamAssist · GitHub

If that wakeword functionality is added, this could potentially enable full wakeword enabled assist on any tablet :smiley:

10

I got this to trigger the assistant, but it never stops listening when the assistant is triggered, it just keeps listening unless I press the microphone icon manually.

11

I had a similar problem at first, seems it was related to my whisper voice model. If there was any background noise like a fan or a dishwasher then the “tiny” and “base” models would assume that was close enough to speech and keep listening until it timed out. Increasing to the “small” voice model and increasing the beam size mostly eliminated that issue.

Can’t say if your issue is caused by the same thing, if yours won’t ever stop listening - even when there’s no background noise - then it may be something different like the app not having mic access permission.

12

I don’t think its mic permission because when I do press the mic button manually, after its been trigger by hot word, it picks up what I said just fine.

It also triggers all the time even when not saying the hot word lol…

I dig around in the settings today and see if I can adjust anything to fix both the issues I am having.

13

I found that using the universal “umdl” wake words cut down on a lot of the false positives, though even with those there’s some variability, I found that the “alexa” umdl had an occasional false positive whereas the “smart mirror” umdl had basically none (but also would frequently ignore my wife).

That being said I think I finally cracked the code on how to get semi-accurate custom “pmdl” wake words. I used the wake word creator here (https://snowboy.jolanrensen.nl/) and recorded 24 samples of myself in a completely silent environment. That seemed to do the trick, I’ve been running it for a few hours with people talking around it and it only had 1 false positive and responds about 90-95% of the time.

Edit a few days later - the custom wake words definitely have a higher false positive rate, I’d say 3-4 false positives per day, which is livable. I’m also running two custom wake words simultaneously, one for my wife and one for me, I don’t know if one or both are causing them, but having two undoubtedly increases the chances of a false positive match.

1 Like
14

You can use automate instead of tasker. It is free.

15

I’ve never used Automate before so if I’m doing something wrong please let me know, but it looks like the hotword detection in the app uses an old version of Google Now and only responds to “Ok Google”.

16

Screenshot_20231212-144931_Automate
Screenshot_20231212-144931_Automate1080×2280 119 KB

Here is my ,working flow, it uses only 3 blocks, others are just test.

3 Likes
17

Ahh, so you’re still using hotword plugin to handle the wake word detection, not the built in hotword detection block. Got it, thanks for the clarification, I’ll amend the main post to add Automate as an option in place of Tasker.

1 Like
18

Thank you so much, didn’t know about the subtasks of apps by holding on the app icon

19

Thanks for posting this. I am on the hunt for a way to replace my Amazon Echo devices using a tablet and a modified HA display that I’m working on. Wakeword on device is super important to me and this ‘project’ seems like it has a lot of potential and coincides with my wants. It’d be great if this functionality would be rolled into the companion app but who knows when or if that will happen.

Now that it’s been some time, can you update on your experience? Are you still using this daily? Do you find it to be reliable? Have you added AI to your pipeline for additional functionality?

1 Like
20

Yes I’m still using it daily, I have two devices running with this workaround, one tablet/dashboard in my main room, and an old phone in my kitchen that was supposed to also be a digital photo frame.

There’s two ways I could take your question, if you’re asking specifically about the wakeword piece I can tell you it works flawlessly. If you’re asking about the overall experience of running an HA voice assistant on an android device I have a few complaints.

It’s definitely reliable enough to be useful, and all of my issues have been specific to the HA app, not the wakeword automation. In a quiet environment whispers STT is fairly accurate (about 95% of commands are understood), when other people are talking in the background the wakeword will still work flawlessly but it’s a 50/50 chance whisper will understand it. Unless you have 3-5 seconds of silence after a command it may not be sure when to stop listening and eventually just times out. I haven’t used any other voice assistants extensively so maybe that’s to be expected. I’m using the small-int8 whisper model with beam size 3 for reference.

I found that running fotoo as the photo frame app in the foreground seemed to somehow cause HA to not be able to access the device audio. The wakeword would function normally while the photos were displaying, but the HA assist dialog would be unable to hear the command.

I haven’t set up AI yet, waiting for an implementation that’s friendlier to non-tech inclined people. That would certainly help resolve another issue I’ve had with using HA as a voice assistant, which is that the built in commands are limited and require specific verbiage. My wife has attempted to use it but was frustrated by the fact that commands have to be phrased particularly ways and she could never remember exactly how to ask it what she wanted.

1 Like