Year of the Voice - Chapter 4: Wake words

Blog
1

This year is Home Assistant’s Year of the Voice. It is our goal for 2023 to let users control Home Assistant by speaking in their own language.

We’ve got great news: wake words are finally here! After 4 chapters, we now have the final building block for voice in Home Assistant.

In Chapter 1, we started with text commands such as “turn on the kitchen light” and “open garage door”. We now support 56 languages and have 188 contributors helping to translate common smart home commands for everyone.

Chapter 2 introduced audio for voice commands: both speech-to-text and text-to-speech. This included local options for maximum privacy as well as support for Home Assistant Cloud for incredible speed and language coverage. Lastly in Chapter 3, we added the ability to set Home Assistant as your default assistant on Android phones and watches.

For Chapter 4, we’ve now added wake word processing inside Home Assistant. Wake words are special words or phrases that tell a voice assistant that a command is about to be spoken. Examples are: Hey Google, Hey Siri or Alexa.

Home Assistant’s wake words are leveraging a new project called openWakeWord by David Scripka. This project has real-world accuracy, runs on commodity hardware and anyone can train a basic model of their own wake word in an hour, for free.

To try wake words today, follow our updated guide to the $13 voice assistant.

To watch the video presentation of this blog post, including live demos, check the recording of our live stream.

Wake words in Home Assistant

Wake words are hard to build. They are based on AI, there is little room for false positives, and they need to run extremely fast: as fast audio as comes in. You can’t have a voice assistant start listening 5 seconds after a wake word is spoken. Voice satellite hardware generally does not have a lot of computing power so wake word engines need hardware experts to optimise the models to run smoothly.

We didn’t want to limit ourselves to a single type of hardware, so we decided to change the approach: we do the wake word detection inside Home Assistant. Voice satellite devices will constantly sample current audio in your room for voice. When it detects voice, the satellite will send audio to Home Assistant where it will check if the wake word was said and handle the command that followed it.

Overview of the wake word architecture

The advantage of this approach is that any device that streams audio can be turned into a voice satellite, even if it doesn’t have enough power to do wake word detection locally. It also allows our developer community to easily experiment with new wake word models as they don’t have to first shrink it to be able to run on a low-powered voice satellite device.

To try it out, follow our updated tutorial to create your own $13 voice assistant.

There are downsides to this approach. The first is that the quality of the captured audio differs. A speakerphone with multiple microphones and audio processing chips captures voice very cleanly. A device with a single microphone and no post-processing? Not so much. We compensate for poor audio quality with audio post-processing inside Home Assistant and users can use better speech-to-text models to improve accuracy like the one included with Home Assistant Cloud.

The other downside of this approach is that each satellite requires ongoing resources inside Home Assistant when it’s streaming audio. With our current approach, users can run 5 voice satellites without overwhelming a Raspberry Pi 4 (assuming all satellites are streaming at the same time). To scale up, we’ve updated the Wyoming protocol to allow users to run wake word detection on an external server.

Wyoming is our protocol allowing to run parts of a voice assistant in other programs and/or computers

Users can pick per configured voice assistant what wake word to listen for

openWakeWord

For the built-in wake words, we rely on openWakeWord by David Scripka. It’s a technological marvel that is created with 4 goals in mind:

  • Be fast enough for real-world usage
  • Be accurate enough for real-world usage
  • Have a simple model architecture and inference process
  • Require little to no manual data collection to train new models

To achieve its goals, openWakeWord is built around an open source audio embedding model trained by Google and fine-tuned using our text-to-speech system Piper. Piper is used to generate many thousands of audio clips for each wake word using a unique approach that creates endless variations of different speakers. These audio clips are then augmented to sound as if they were spoken in multiple kinds of rooms, at specific distances from a microphone, and with varying speeds. Finally, the clips are mixed with background noise like music, environmental sounds, and conversation before being fed into the training process to generate the wake word model.

Overview of the openWakeWord training pipeline.

Home Assistant runs openWakeWord as an add-on and comes with various wake word models by default, including our “Okay Nabu” model. Click the button below to install it.

![](upload://uPXPlUVHs5hskZPLetZxJcw5Bhx.svg)

Once installed, the add-on will be discovered via the Wyoming integration.

OpenWakeWord currently only works for English wake words. This is because we lack models of other languages with many different speakers. Similar models for other languages can be trained as more multi-speaker models per language become available.

If you’re not running Home Assistant OS, openWakeWord is also available as a Docker container. Once the container is running, you will need to add the Wyoming integration and point it at its IP address and port (typically 10400).

Make your own wake word

What makes openWakeWord unique is its ability to fine tune Google’s model, trained on clips from real voices, with fake voice clips generated by Piper. This makes it possible to create your own wake words without collecting samples from real people (though real samples can improve the outcome).

David created a Google Collab notebook to create your own openWakeWord model. Enter your desired wake word and an hour later you get your own wake word (using the free computing available to all Google Collab users).

To get started, see our new “create your own wake word”-tutorial.

The models generated with the notebook will perform reasonably well. They will not perform as well as the ones bundled with Home Assistant which have received a lot of extensive training.

Screenshot of the wake word generation notebook

Other wake word engines

In Home Assistant, we ship our defaults but allow a user to configure each part of their voice assistants. This also applies to our wake words.

Wake word engines can integrate with Home Assistant by adding them as an integration or run them as a standalone program that communicates with Home Assistant via the Wyoming protocol.

How wake words integrate into Home Assistant

As an example, we’re also making the Porcupine (v1) wake word engine available. It supports 29 wake words across English, French, Spanish and German, including Computer, Framboise, Manzana and Stachelschwein.

![](upload://uPXPlUVHs5hskZPLetZxJcw5Bhx.svg)

Reuse and repurpose: different ways to create a voice satellite

We’re building our voice assistant based on our Open Home vision: a smart home that values privacy, choice and sustainability. Two words that are often mentioned as part of sustainability are reuse and repurpose.

Since our voice satellite is only responsible for capturing audio, a lot of devices one might have in their “old tech” drawer can be given a new life and purpose as a voice satellite.

When audio is captured via USB, we recommend using a USB speakerphone because they contain audio processing chips that clean up the audio and enhance voices. They also come with a speaker and look a bit like one expects a voice satellite to look. We had great results in our testing with the Anker PowerConf S330. It did require a firmware update before it could be used with Home Assistant.

Some USB speakerphones will require a powered USB hub because of power limits on the Raspberry Pi’s USB ports.

Turn Home Assistant into a voice satellite

You can configure your device running Home Assistant to capture audio and turn it into a voice assistant. To do this, you need to plug in a USB microphone or speakerphone and configure the Assist microphone add-on. Your Home Assistant device may need to be rebooted before the microphone is usable.

![](upload://uPXPlUVHs5hskZPLetZxJcw5Bhx.svg)

Home Assistant Blue with a speakerphone

Turn any ESP32 into a voice satellite using ESPHome

ESPHome is our firmware to allow users to easily create devices for their smart home. In Year of the Voice - Chapter 2, we’ve added support for ESPHome to accept voice commands when a user pushes a button.

Today, that support is extended to allow any ESP32 device with an i2s microphone to become a voice satellite for Home Assistant.

Voice assistant on a breadboard.

Recommended parts:

Example configuration

This method requires users to have basic experience with configuring ESPHome devices.

Turn any old Raspberry Pi into a voice satellite

We’ve made homeassistant-satellite available that allows you to connect a USB microphone or speakerphone to an old Raspberry Pi, or any other Linux computer, and turn it into a voice satellite for Home Assistant.

While any Linux computer works, we recommend limiting it to ARM-based processors because they use a lot less energy.

This method requires users to know how to install applications on a Linux system.

Voice office hours for scientists

We want Home Assistant to be used as a platform for scientists that are developing new wake word, speech-to-text and text-to-speech engines. Working with Home Assistant allows you to try your model as part of a voice assistant in a real world scenario. The Home Assistant community loves new technology and will be great in testing it out and providig feedback.

Engines can be plugged Home Assistant’s voice pipelines using the Wyoming protocol. While small, the Wyoming protocol can be tricky to get right for first time integrators. If you’re such a person, reach out to us at [email protected] and we’ll help you integrate.

What’s next

Now that the foundation is in place for all parts of a voice assistant, it will be easier for us to share what we are going to work on next.

We want to work towards supporting the most common tasks that people use with other voice assistants. This includes support for multiple shopping lists, timers and weather forecasts.

To improve accuracy, openWakeWord allows further fine-tuning of the model with recordings made by the user via their own voice satellite. We want users to be able to easily record themselves and let Home Assistant create this improved model.

On the voice satellite side we’re going to integrate more advanced audio processing to improve wake word and Speech-to-Text accuracy. We will also do another attempt at getting wake words running inside ESPHome.

The voice satellite improvements will require more advanced hardware and we’re aiming for the ESP32 S3 Box 3. This is the new variant of the now discontinued ESP32 S3 Box (and lite version). Espressif told us that it will be in stock soon.

If you already have an ESP32 S3 Box variant, you can install our ESPHome configuration to receive these updates as they come available.

That’s a wrap!

We hope that you enjoy the wake words and that you set up voice satellites around your house. Let us know how it goes and share your experience with us.

See you soon in chapter 5!

Thank You

A big thanks to David Scripka for openWakeWord. Thanks to Jesse Hills for his patience and support while Mike and I explored wake word architectures and help ESPHome fit in. Big thanks to everyone at Nabu Casa who has helped make and review today’s content.

Thank you to the Home Assistant community for subscribing to Home Assistant Cloud to support Year of the Voice and development of Home Assistant, ESPHome and other projects in general.

Thanks to our language leaders for extending the sentence support to all the various languages.

This is a companion discussion topic for the original entry at https://www.home-assistant.io/blog/2023/10/12/year-of-the-voice-chapter-4-wakewords/
18 Likes
2

First. Grate job. Really big improvement this year.

And now the quiestion.
Is it possible to set Google speakers as satelites?
First reason - I have one in every room.
Second - atom echo are out of stock for some time in my country and other satelites are DIY Look sollutions so WAF will be zero.

Thanks.

3 Likes
3

This is awesome!

Not sure if it was mentioned in the video, for work reasons I was only able to listen to part of it and skim the rest.

Will the wake word work on phones? This would be awesome if it did.

Also being able to use multiple sources like Google Home / Nest does to help triangulate your voice in order to get the clearest sounding one (including your phone). But that would be a much more difficult task I assume that would be a later thing xD

1 Like
4

The companion app satellite would be awesome for tablet kiosk users.

11 Likes
5

It would also be awesome if we could use camera audio feeds already existing in home assistant.

10 Likes
6

This is awesome!!
One question: does it work with many satellites with the same wake word? in that case, how does it works? which one responds? With Alexa, it detects the nearest Alexa and knowing in which room it is, you can activate the lights without saying which light (eg: Alexa, turn on the lights, and it turns on the lights of the room where the speaker is).

Some tutorials to make a DIY device with esp would be great too.

:grin::grin::grin:

1 Like
7

The satellite that detects the wake word first will be the one that responds. We will use the device id in the near future to know which room the satellite is in so we can turn on the right lights :slightly_smiling_face:

18 Likes
8

Stay tuned :wink:

12 Likes
9

Happy to download and install python and jupyter notebook/anaconda, but anyone know how to get the liked google collab notebook to run on my AMD GPU rather than CPU for faster training?

10

Amazing work, thank you guys :pray:t2:

Question, I saw that you will be working on timers which is amazing news. Are you also planning to work on an “announce” feature similar to Alexa to send a message to all smart speakers in the house?

11

Will this do to get openwakeword as a container? (docker-compose.yaml)

version: "3"
services:
  openwakeword:
    container_name: openwakeword
    image: rhasspy/wyoming-openwakeword:latest
    command: [  "--preload-model", "ok_nabu"  ]
    volumes:
      - "./openwakeword-data/custom-models:/custom"
    environment:
      - "TZ=Europe/Amsterdam"
    restart: unless-stopped
    ports:
      - "10400:10400"

removed --model because it’s depricated

2 Likes
12

Looks like CloudFlare has nerfed the email address in the “Voice office hours for scientists” section.

Also, it’d be nice if there could be some kind of fallback / passthrough option available. We’ve got a bunch of Google Home units at our place, and we pay for YouTube Music, so it’d be nice to have basic stuff fulfilled locally (e.g. starting timers, asking about the weather etc.) with any requests that can’t be fulfilled by Home Assistant being sent on to the Google Assistant SDK, or wherever you want, to be fulfilled and played back on the speaker(s) of your choosing.

1 Like
13

Fantastic announcement - congrats and thanks to all involved

1 Like
14

This is a great project. I am so happy that we have wake words now and a couple different ways to caputure the wake words.
Quick question, I was training my own wake word using the notebook. I followed the instructions and it was running thru and then at the end where it looks like it would download the files I need, it is throwing errors. The last line is:FileNotFoundError: [Errno 2] No such file or directory: 'my_custom_model/Carson.onnx'
There is more error code above that as well. Thoughts?

15

Nice!
I don’t think it’s quite in the state where I can transition my existing voice control to Assist, but it’s getting close. The missing parts for me right now are:

  • non-English custom wake word (Russian in my case, I’m currently using Snowboy, so adaptation of that to Wyoming would work for me as well)
  • non-voice wake mechanisms, e. g. a service call or a button (might be there already and I’m not seeing it?) — I have a few scenarios where I switch voice waking off (“silence for X minutes” or when a media player is playing in the room) and a hardware override of some sort (physical button?) would be useful

As for the rest, that I think I can adapt myself already.

16

I was able to install the M5Stack Atom Echo Smart Speaker and it seems to be working however I see I have an update available within esphome however when I try to upgrade I get the following error that I can’t get past. Any ideas what I need to do?

**edit - spoke too soon it seems to falling offline not staying connected

I didn’t see it mentioned but I figured out it needs at least the beta version of esphome 2023.10.b. Unfortunately it is still falling offline and I see this error in the logs → WARNING m5stack-atom-echo-8a1a74.local: Connection error occurred: [Errno 104] Connection reset by peer

Any thoughts on why this might be happening?

17

@cadster Looks like this is also needed https://github.com/guillaumekln/faster-whisper/

1 Like
18

This is fantastic! I’m excited we’re as close as we are to getting a fully integrated local voice assistant! The one question I had that I haven’t seen mentioned yet is the ability to replicate the Alexa interactive notifications where the system can prompt and ask questions and the answers can kick off branches of an automation.

Is that something we can already do in the existing architecture or is that something that will have to be developed in the future? I’ve had some trouble following all the voice developments so if I missed it, I apologize in advance!

Thanks again for a great series of developments!

19

This is something I’ve wanted to do for a long time. I’d love to have my voice assistant ask a question when my kitchen motion sensor detects me in the morning and then have it wait for my yes/no response so it can complete an automation.

20

I got openwakeword running in a docker container, and used the example configuration for my ESP32-S3-Box to test this out, and it’s working great! That was really easy overall.

Not perfect of course, but it’s so much better than it used to be already. Can’t wait for more! Great work!

1 Like