This is a demo of a voice assistant made in Home Assistant with visual responses played on a tablet on which I installed fully kiosk browser and browser mod.
EDIT UPDATE
Due to the fact that AlexxIT managed to implement Wake Word detection in the StreamAssist integration, I managed to make some changes in this integration to be able to play these random visual responses without the need for an Esp32 satellite, using:
When the wake word is detected, the MUTE switch on the Esp32 satellite turns on. This is a REAL MUTE SWITCH made by connecting the L/R pin on the microphone to a digital pin on the Esp32 through which the microphone records only when there is voltage on this pin, and in the code, the microphone is set to the left channel.
When the wake word is detected, a video and audio response is streamed through the browser mode media player on the tablet’s display. You can make several answers that can be played randomly.
After this answer is finished playing, the mute switch goes back to off and the listening starts.
When the streaming of the tts response starts, it is played through the fully kiosk media player on the tablet.
At the same time, a no_sound_speech video is sent, i.e. simulation of speech but without sound, through the browser mode media player on the tablet display.
Several voice assistants can be set, each with individual responses and pipelines that can be exchanged through voice commands.
Very well done this, especially the changing of languages and assistants.
Can you share a bit more on the creation of the assistant characters (Javis & Sheila)?
As I understand it, there is no support for Wake Words in the browser and you would need to manually interact with the tablet to start Assist, plus voice input doesn’t work if you don’t have https set up on your Home Assistant server. That’s why the ESP32 board is needed.
Indeed, that is true. Now I’m working to replace the Esp32 satellite with AlexxIT’s Stream Assist integration in this project and I think I’ll be able to finish this soon.
With a script and automation, I will upload them to my GitHub. The Esphome satellite integration exposes several services: next pipeline, first pipeline, and last pipeline. An SSH script overwrites the GIFs for speech and listening. Additionally, an automation triggered by conversation executes them.
Thx! So you just switch the select of the pipeline entity of the esphome device. I didnt think of that. nice!
Any idea how we could enable different wakewords on the same device in parallel? Should be simple with remote wakeword-detection to send the audio to two different pipelines
i have a fire tablet installed all the requirements and am running fully kiosk, however when I attempt to use the wake work i see rtp mic moving but nothing else happens. Along with fully kiosk do i also need to have the home assistant companion app installed??
No. You can’t use google speakers built in microphone to comand Home Assistant Assist. On google nest mini you can replace internall pcb with a onju voice pcb to use internal microphone. Onju pcb is an esp32 sattelite. Search onju voice on youtube for more informations.
