I used the ESPHome yaml from the Seeed website Voice Assistant System for Home Assitant | Seeed Studio Wiki.
I’ll check out the voice-kit GitHub and see how that compares.
1 Like
Tbf, the op can do whatever they want with their thread.
2 Likes
I would argue that both ESPHome voice-kit development and other hardware using ESP32 and XMOS xCORE chips combination for audio AI acceleration is on-topic as without those these “ReSpeaker Lite” kits will in practice never be useful in the future for anyone who is not a ESPHome developer themselves.
The “bigger picture” to make all these eventually become useful to the avérage Home Assistant user they not only need to be fully supported in the upstream ESPHome project, they also need to standardize those devices in both ESPHome and the Home Assistant core, and ESHome developers are now working on several new components related to this, including a new entity component as assist_satellite platform for that which will represent a standard VoIP-based voice satellite for Home Assistant Assist voice control. Check out this architecture discussion (which sounds like it has essentially been approved)
Initial entity component for this new assist_satellite platform been merged to Home Assistant core:
Bigger picture:
- Standardize how voice satellites expose their capabilities (ie, available wake words)
- Standardize how voice satellites are configured (which wake word to listen to)
- Automate based on the state of the satellite’s pipeline
- Control the behavior of a voice satellite from HA during the setup wizard
- Skip wake word and listen for a command (with or without executing it)
- Listen for a specific wake word (without running a pipeline)
- Control a voice satellite from HA using service calls
- Announce text using the TTS portion of the satellite’s pipeline
Also see related patches with new features submitted to both ESPHome and the Home Assistant core:
2 Likes
Hi,
I’ve just received mine and I’m trying to modify the yaml to use micro wake word.
It seems to detect when I say the wake word but it stops streaming the audio just after switching to this state.
Is anyone succeeded on making it work ?
“mine” which ? ReSpeaker Lite, or one of the other similar devices mentioned through this thread ?
It would also make it easier for others to assist if you give a bit more information about your configuration (hardware and software). Eg, is the yaml in the ESPHome configuration for the device ?
1 Like
Have you tried using the experimental ”voice-kit” fork repo project from ESPHome instead of the main esphome repo?
Okay, I’m a bit confused. I already had 2 XIOS S3’s so I just ordered the board. How is the XIAO soldiered to the board? Taking a look at all the pics here and seseds website shows the XIAO sitting on the respeaker lite but I’m not seeing anything soldiered in any pictures.
They also have you flashing 2 different things, one for just the board and another for the i2s. Then on the page with instructions and YAML for ESPHome it says a Sonoff smart switch is required although outside adding it to HA, it’s not referenced anywhere.
It’s using Openeakeword with the default YAML, this is just our laziness by Seeed. I’m pretty sure the DFU driver is for when an ESP32-S3 XIAO isn’t used and using this with a PC. Then the i2s driver is able to be downloaded but the flashing instructions link takes you to a non existent page.
All in all Seeed has done a terrible job with documentaries and broken links on how to flash new firmware is just plain odd.
The frustrating part is they have an example of using a wake work using tensorflow lite but only.for Arduino…
Anyone who has the pre soldiered version could you post a pic of how it’s actually attached to the board? Below is the yaml from Seeed.
esphome:
name: esp32s3
friendly_name: ReSpeakerv3
platformio_options:
board_build.flash_mode: dio
board_build.mcu: esp32s3
# on_boot:
# then:
# - if:
# condition:
# switch.is_on:
# then:
# - voice_assistant.start_continuous:
esp32:
board: esp32-s3-devkitc-1
variant: esp32s3
framework:
type: esp-idf
version: recommended
logger:
level: VERY_VERBOSE
api:
wifi:
ssid: "Wi-Fi name"
password: "Wi-Fi password"
captive_portal:
external_components:
- source: github://QingWind6/ESPHome_XIAO-ESP32S3
i2s_audio_xiao:
i2s_lrclk_pin: GPIO7
i2s_bclk_pin: GPIO8
i2s_mclk_pin: GPIO9
microphone:
- platform: i2s_audio_xiao
id: xiao_mic
adc_type: external
i2s_din_pin: GPIO44
pdm: false
bits_per_sample: 32bit
channel: left
speaker:
- platform: i2s_audio_xiao
id: xiao_speaker
dac_type: external
i2s_dout_pin: GPIO43
mode: stereo
voice_assistant:
microphone: xiao_mic
use_wake_word: true
noise_suppression_level: 0
auto_gain: 0dBFS
volume_multiplier: 1
speaker: xiao_speaker
id: assist
on_listening:
- light.turn_on:
id: led
blue: 100%
red: 0%
green: 0%
effect: "Slow Pulse"
on_stt_vad_end:
- light.turn_on:
id: led
blue: 100%
red: 0%
green: 0%
effect: "Fast Pulse"
on_tts_start:
- light.turn_on:
id: led
blue: 100%
red: 0%
green: 0%
brightness: 100%
effect: none
on_end:
- delay: 100ms
- wait_until:
not:
speaker.is_playing:
- script.execute: reset_led
on_error:
- light.turn_on:
id: led
red: 100%
green: 0%
blue: 0%
brightness: 100%
effect: none
- delay: 1s
- script.execute: reset_led
on_client_connected:
- if:
condition:
switch.is_on: use_wake_word
then:
- voice_assistant.start_continuous:
- script.execute: reset_led
on_client_disconnected:
- if:
condition:
switch.is_on: use_wake_word
then:
- voice_assistant.stop:
- light.turn_off: led
light:
- platform: esp32_rmt_led_strip
id: led
name: None
disabled_by_default: true
entity_category: config
pin: GPIO1
default_transition_length: 0s
chipset: ws2812
num_leds: 1
rgb_order: grb
rmt_channel: 0
effects:
- pulse:
name: "Slow Pulse"
transition_length: 250ms
update_interval: 250ms
min_brightness: 50%
max_brightness: 100%
- pulse:
name: "Fast Pulse"
transition_length: 100ms
update_interval: 100ms
min_brightness: 50%
max_brightness: 100%
output:
- platform: ledc
id: light_output
pin: GPIO21
inverted: true
script:
- id: reset_led
then:
- if:
condition:
- switch.is_on: use_wake_word
- switch.is_on: use_listen_light
then:
- light.turn_on:
id: led
effect: none
else:
- light.turn_off:
id: led
switch:
- platform: template
name: Use wake word
id: use_wake_word
optimistic: true
restore_mode: RESTORE_DEFAULT_ON
entity_category: config
on_turn_on:
- lambda: id(assist).set_use_wake_word(true);
- if:
condition:
not:
- voice_assistant.is_running
then:
- voice_assistant.start_continuous
on_turn_off:
- voice_assistant.stop
- lambda: id(assist).set_use_wake_word(false);
- platform: template
name: Use Listen Light
id: use_listen_light
optimistic: true
restore_mode: RESTORE_DEFAULT_ON
entity_category: config
on_turn_on:
- script.execute: reset_led
on_turn_off:
- script.execute: reset_led
You’ll need hot air gun and solder paste to solder the board smd style
Using the normal 7 pin headers that used to come with any esp32 XIAO board I was able to get power to the S3 and ping it when only the respeaker lite board USB C connected. Upon further inspection, I noticed that it appears to be an issue with the code it’s pointed to in the git url. When looking at the repository, nothing has been updated in 6 to 9 months and it’s specifically saying it doesn’t like something in the git url. Even if you don’t have one you can reproduce this using the option to download the bin file… I thought this was a relatively new product? Also, how would the Adafruit QT Py fit if you had to use solder paste and hot air gun? Maybe I am missing something but plug and play with PC. Can’t get the microphones to work at all though on ESP32. Those compile errors below happen regardless if I create a new device and copy/paste seeed’s code in. Another let down from Seeed IMO that will be returned.
lite/src/esphome/components/i2s_audio_xiao/microphone/i2s_audio_microphone.o
src/esphome/components/i2s_audio_xiao/microphone/i2s_audio_microphone.cpp: In member function 'void esphome::i2s_audio_xiao::I2SAudioMicrophone::start_()':
src/esphome/components/i2s_audio_xiao/microphone/i2s_audio_microphone.cpp:63:3: warning: missing initializer for member 'i2s_driver_config_t::fixed_mclk' [-Wmissing-field-initializers]
};
^
src/esphome/components/i2s_audio_xiao/microphone/i2s_audio_microphone.cpp:63:3: warning: missing initializer for member 'i2s_driver_config_t::mclk_multiple' [-Wmissing-field-initializers]
src/esphome/components/i2s_audio_xiao/microphone/i2s_audio_microphone.cpp:63:3: warning: missing initializer for member 'i2s_driver_config_t::chan_mask' [-Wmissing-field-initializers]
src/esphome/components/i2s_audio_xiao/microphone/i2s_audio_microphone.cpp:63:3: warning: missing initializer for member 'i2s_driver_config_t::total_chan' [-Wmissing-field-initializers]
src/esphome/components/i2s_audio_xiao/microphone/i2s_audio_microphone.cpp:63:3: warning: missing initializer for member 'i2s_driver_config_t::left_align' [-Wmissing-field-initializers]
src/esphome/components/i2s_audio_xiao/microphone/i2s_audio_microphone.cpp:63:3: warning: missing initializer for member 'i2s_driver_config_t::big_edin' [-Wmissing-field-initializers]
src/esphome/components/i2s_audio_xiao/microphone/i2s_audio_microphone.cpp:63:3: warning: missing initializer for member 'i2s_driver_config_t::bit_order_msb' [-Wmissing-field-initializers]
src/esphome/components/i2s_audio_xiao/microphone/i2s_audio_microphone.cpp:63:3: warning: missing initializer for member 'i2s_driver_config_t::skip_msk' [-Wmissing-field-initializers]
Compiling .pioenvs/respeaker-
You asked specifically about the S3 and referenced the kit they sell, I simply gave you the information you requested. Most other platforms will access the respeaker via USB.
You can also break out the pads using short wires, solder wires onto the respeaker pads for the s3 and solder whatever you want onto the other end for testing/debugging. AFAICT this is basically a HW dev kit.
I am waiting on my hot air station, solder paste and flux before I go down the path of making it work. You will notice in previous replies that a lot of the esphome stuff is being worked on to get this product working well.
As for disappointment with docs, from their track record, you should not be surprised. End of the day, we have access to XMOS chips and multiple outfits working towards the same goal.
1 Like
I do appreciate your answer, which is 100 percent the truth. I apologize if any of my frustration seemed aimed towards you, it wasn’t, just frustration at Seeed and time wasted mostly. I do question why they put the holes on the board to fit as one of the mics is directly under the S3, which is also the side with status led.
I know the mic can probably withstand a good bit of heat for the six middle contacts on the XIOS S3/respeaker lite, but obviously I don’t want to damage it as I am not great with paste/hot air. I have only done that twice before and that middle square pad almost right where the second mic is on the other side. Thanks for the tips though as those are methods I’m more comfortable with personally.
You will need to flash different firmware to the ESP32 SoC chip and the XMOS XU316 (DSP) SoC chip. I understand it as you needing to flash the ESPHome first, and then you need to flash the XMOS chip from ESPHome via I2C DFU.
Such tasks are just one of the things that Nabu Casa and ESPHome developers are looking at simplifying for non-developers. Again, the ReSpeaker Lite with ESP32 is a development board and this is all very advanced cutting-edge stuff, but do check out their experimental voice-kit fork of ESPHome if have the skill and time.
Right now they have an open pull request to that experimental fork that will eventuelly have ESPHome automatically get XMOS firmware OTA image from a URL (via example the ESPHome addon) and flash it on boot.
1 Like
Thanks for the explanation, that makes more sense, especially in the long run…
I am glad that they are implementing the OTA/http_request functionality so non technical users who don’t want to learn/troubleshoot YAML can add devices to HA without the ESPHome add on installed and still do OTA updates with the user having zero understanding of yaml configurations. It will all be super user friendly on e it’s done.
It’s great to know how this will probably end up working.and that there are a lot of people making a huge effort to make this work as seamless as possible .
From the link you provided above one of the files is checking the DFU version via I2C to make sure it’s a valid version and will throw an error if it isn’t but not actually updating it although I’m not a developer so I could be wrong but that’s what it appears to be doing in this file and other files in that directory. If they can check the version then updating it would be possible through the ESP32 using I2C. Once again, I know it’s a work in progress. I also didn’t go through every file so it may be doing the DFU firmware update over I2C already.
dfu-util is just a CLI utility to update device firmware via USB so Seeed probably tells you to use it because it exists already and is free. It will be nice to see this functionality in ESPHome eventually . I did find the Arduino sketch to stream audio via MQTT informative and possibly useful to do in HA in specific use cases. The below is from seeeds docs, which is what I was going by.
Flash the I2S firmware
Before we start this chapter, please make sure the ReSpeaker Lite’s firmware is the I2S version.I2S Firmware Download
Check Get Started to flash the firmware.Software Preparation
Install Arduino IDE
Download and Install the stable version of Arduino IDE
Sorry my notification was not set and I forgot to check the thread. I’ve received the respeaker lite with Xiao esp32 s3.
Mic seems to work as it waits for wake word but once wake word is detected, the esp32 crash
Received mine and the hot air station (AliExpress 858D station - around $40 CAD). These are the steps I took using a Linux PC.
-
SMD soldered the s3 to the respeaker board:
- Using Sn42 Bi58 and flux paste @ 190 C
- Flux first, slightly melt
- Paste second, place S3 on pads and start gently warming the entire respeaker PCB and S3
- After sufficiently warming up respeaker and S3, focus the heat on the S3 and wait for the paste to melt onto the pads
- Use the hot air station to gently cool down the respeaker and S3
- Clean with IPA (IsoPropyl Alcohol) and let dry
- Don’t plug into PC, yet
-
Uploaded the I2S firmware using Linux and
dfu-util.- Install
dfu-util - Add UDEV rule:
sudo nano /etc/udev/rules.d/77-mm-usb-device-blacklist.rules - Add this line; you may need to change
plugdevto the group that your distro uses (uucpon Arch):ATTRS{idProduct}=="d058", ATTRS{idVendor}=="2b04", MODE="664", GROUP="plugdev"
- Plug in the respeaker board
- Connect a USB-C data cable to the respeaker USB-C port (not the esp32 S3)
- Run flashing command:
dfu-util -e -a 1 -D /path/to/ffva_i2s_v1.0.5.bin - Troubleshooting:
LIBUSB_ERROR_ACCESS→ usesudo- Any others, try rebooting
- After successful flash, unplug and replug the respeaker board
- Audio input/output is now accessed using I2S interface.
- Install
-
Using this guide to configure esphome
- This time, use the S3 USB-C port to upload the esphome firmware onto the S3
- You can ignore the SONOFF device, it is just a switch, so you can use the respeaker board to toggle a device. You can expose any device to your voice assistant pipeline and use respeaker to interact with it.
- The juicy part is the esphome YAML config.
- Also, the voice assistant pipeline config, piper, whisper, openwakeword stuff (if you don’t have that configured).
- There will/may be issues using their base esphome code, it’s basically just an example to get you pointed in the right direction. Watch the above repos for insight into how esphome is trying to get voice assistants working well.
Firmware / YAML
I backed up the UA and I2S firmware as well as the esphome YAML config they show, just in case 
.
Differences
The UA firmware is for accessing the microphone/speaker via USB, the I2S firmware is to access via I2S.
- UA firmware is for RPI (with USB), PC, etc.
- I2S is for esp32, PyQT, RPI non-USB, etc.
Obviously, if you’re not using the XIAO S3, you’d need to break the I2S pads out with short wires from the S3 PCB traces. The respeaker board has headers and extra 5v/GND pads for powering external devices.
Example output of dfu-util
Success
❯ sudo dfu-util -e -a 1 -D ~/Downloads/ffva_i2s_v1.0.5.bin
dfu-util 0.11
Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2021 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/
dfu-util: Warning: Invalid DFU suffix signature
dfu-util: A valid DFU suffix will be required in a future dfu-util release
Opening DFU capable USB device...
Device ID 2886:0019
Run-Time device DFU version 0101
Claiming USB DFU Interface...
Setting Alternate Interface #1 ...
Determining device status...
DFU state(2) = dfuIDLE, status(0) = No error condition is present
DFU mode device DFU version 0101
Device returned transfer size 4096
Copying data from PC to DFU device
Download [=========================] 100% 262144 bytes
Download done.
DFU state(7) = dfuMANIFEST, status(0) = No error condition is present
DFU state(2) = dfuIDLE, status(0) = No error condition is present
Done!
LIBUSB_ERROR_ACCESS
❯ dfu-util -e -a 1 -D ~/Downloads/ffva_ua_v2.0.5.bin
dfu-util 0.11
Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2021 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/
dfu-util: Warning: Invalid DFU suffix signature
dfu-util: A valid DFU suffix will be required in a future dfu-util release
dfu-util: Cannot open DFU device 2886:0019 found on devnum 9 (LIBUSB_ERROR_ACCESS)
dfu-util: No DFU capable USB device available
Windows
I don’t use windows 
, but the driver they talk about is strictly for interacting with the USB interface to allow uploading firmware to the respeaker. After the UA firmware is flashed and the board is restarted, the respeaker should present itself as an audio input.
2 Likes
@baudneo Hi! If you don’t mind, i have couple of questions.
(I got bundle with soldered S3 and speaker)
- Did you observe speaker squeaking and distorting constantly on boot and couple seconds after? I have this behaviour on both my hats with example YAML from Seeed…
- Did you try to use hat with Gnumpi component GitHub - gnumpi/esphome_audio: Custom audio components for ESPHome? It’s really working good with simple mic/amp setup, so i wanted to check if it works with this one too. So far no success - seems like mics and speaker both use single I2S bus, will dif specs.
- For me on example YAML it’s working to some extent, but very often sets LED to red and stops responding. Do you see same behaviour?
- I am using the headphone jack without anything noticeable.
- No, I haven’t. i2s is half duplex, so on a single bus, you’ll only really get send/recv at one time. From all video examples ive seen of seeeed, they dont use the board as a music player and an assistant. Unless they accounted for this by using 2 i2c busses, we would need to hack something together.
- Yes, it has issues for me as well.
I haven’t had much time to play with it yet. Hoping to spend some time this weekend.
Thank you!
Do you know specifics of duplex? It should have device addresses there. Any help appreciated.
Haven’t researched it but from a quick glance. You use 1 is2 bus for speaker and 1 i2s bus for the microphone. This allows send/recv at the same time.
This is all speculative though, seeed may have both busses wired up, or maybe it’s 1 bus that the xmos chip handles, idk.
It’s full duplex, confirmed by Seeed. So I guess out and in should have different I2C addresses in order to be able to process data. Will dig more.