"ReSpeaker Lite" - new Seeed Studio Voice Assistant Development Kit hardware combine ESP32 with XMOS XU316 DSP chip for advanced audio processing as a ESPHome-based Home Assistant Assist Satellite voice devkit

Mrsash · October 23, 2024, 6:18am

I think something weird is going on…

INFO ESPHome 2024.10.1
INFO Reading configuration /config/esphome/respeaker.yaml...
INFO Updating https://github.com/esphome/voice-kit@dev
INFO Updating https://github.com/esphome/esphome.git@pull/7605/head
INFO Updating https://github.com/formatBCE/Respeaker-Lite-ESPHome-integration@dev
WARNING GPIO3 is a strapping PIN and should only be used for I/O with care.
Attaching external pullup/down resistors to strapping pins can cause unexpected failures.
See https://esphome.io/guides/faq.html#why-am-i-getting-a-warning-about-strapping-pins
INFO Starting log output from respeaker-satellite.local using esphome API
WARNING Can't connect to ESPHome API for respeaker-satellite.local: Error resolving IP address: [Errno -5] No address associated with hostname (APIConnectionError)
INFO Trying to connect to respeaker-satellite.local in the background
INFO Successfully connected to respeaker-satellite @ 192.168.0.60 in 0.100s
INFO Successful handshake with respeaker-satellite @ 192.168.0.60 in 0.192s
[17:09:45][I][app:100]: ESPHome version 2024.10.1 compiled on Oct 23 2024, 15:28:36
[17:09:45][C][wifi:600]: WiFi:
[17:09:45][C][wifi:428]:   Local MAC: 64:E8:33:50:C0:7C
[17:09:45][C][wifi:433]:   SSID: [redacted]
[17:09:45][C][wifi:436]:   IP Address: 192.168.0.60
[17:09:45][C][wifi:440]:   BSSID: [redacted]
[17:09:45][C][wifi:441]:   Hostname: 'respeaker-satellite'
[17:09:45][C][wifi:443]:   Signal strength: -34 dB ▂▄▆█
[17:09:45][C][wifi:447]:   Channel: 3
[17:09:45][C][wifi:448]:   Subnet: 255.255.255.0
[17:09:45][C][wifi:449]:   Gateway: 192.168.0.1
[17:09:45][C][wifi:450]:   DNS1: 192.168.0.1
[17:09:45][C][wifi:451]:   DNS2: 0.0.0.0
[17:09:45][C][logger:185]: Logger:
[17:09:45][C][logger:186]:   Level: DEBUG
[17:09:45][C][logger:188]:   Log Baud Rate: 115200
[17:09:45][C][logger:189]:   Hardware UART: USB_SERIAL_JTAG
[17:09:45][C][i2c.idf:075]: I2C Bus:
[17:09:45][C][i2c.idf:076]:   SDA Pin: GPIO5
[17:09:45][C][i2c.idf:077]:   SCL Pin: GPIO6
[17:09:45][C][i2c.idf:078]:   Frequency: 50000 Hz
[17:09:45][C][i2c.idf:084]:   Recovery: bus successfully recovered
[17:09:45][I][i2c.idf:094]: Results from i2c bus scan:
[17:09:45][I][i2c.idf:100]: Found i2c device at address 0x18
[17:09:45][C][switch.gpio:068]: GPIO Switch 'mute_toggle'
[17:09:45][C][switch.gpio:091]:   Restore Mode: always OFF
[17:09:45][C][switch.gpio:031]:   Pin: GPIO4
[17:09:45][C][esp32_rmt_led_strip:187]: ESP32 RMT LED Strip:
[17:09:45][C][esp32_rmt_led_strip:188]:   Pin: 1
[17:09:45][C][esp32_rmt_led_strip:189]:   Channel: 0
[17:09:45][C][esp32_rmt_led_strip:214]:   RGB Order: GRB
[17:09:45][C][esp32_rmt_led_strip:215]:   Max refresh rate: 0
[17:09:45][C][esp32_rmt_led_strip:216]:   Number of LEDs: 1
[17:09:45][C][gpio.binary_sensor:015]: GPIO Binary Sensor 'User button'
[17:09:45][C][gpio.binary_sensor:016]:   Pin: GPIO3
[17:09:46][C][light:103]: Light 'respeaker-satellite'
[17:09:46][C][light:105]:   Default Transition Length: 0.0s
[17:09:46][C][light:106]:   Gamma Correct: 2.80
[17:09:46][C][template.switch:068]: Template Switch 'Mic mute'
[17:09:46][C][template.switch:070]:   Icon: 'mdi:microphone-off'
[17:09:46][C][template.switch:091]:   Restore Mode: always OFF
[17:09:46][C][template.switch:057]:   Optimistic: NO
[17:09:46][C][template.switch:068]: Template Switch 'timer_ringing'
[17:09:46][C][template.switch:091]:   Restore Mode: always OFF
[17:09:46][C][template.switch:057]:   Optimistic: YES
[17:09:46][C][psram:020]: PSRAM:
[17:09:46][C][psram:021]:   Available: YES
[17:09:46][C][psram:024]:   Size: 8191 KB
[17:09:46][C][safe_mode.button:024]: Safe Mode Button 'Safe Mode Boot'
[17:09:46][C][safe_mode.button:024]:   Icon: 'mdi:restart-alert'
[17:09:46][C][factory_reset.button:011]: Factory Reset Button 'Factory reset'
[17:09:46][C][factory_reset.button:011]:   Icon: 'mdi:restart-alert'
[17:09:46][C][restart.button:017]: Restart Button 'Restart'
[17:09:46][C][restart.button:017]:   Icon: 'mdi:restart'
[17:09:46][C][captive_portal:089]: Captive Portal:
[17:09:46][C][mdns:116]: mDNS:
[17:09:46][C][mdns:117]:   Hostname: respeaker-satellite
[17:09:46][C][esphome.ota:073]: Over-The-Air updates:
[17:09:46][C][esphome.ota:074]:   Address: respeaker-satellite.local:3232
[17:09:46][C][esphome.ota:075]:   Version: 2
[17:09:46][C][esphome.ota:078]:   Password configured
[17:09:46][C][safe_mode:018]: Safe Mode:
[17:09:46][C][safe_mode:020]:   Boot considered successful after 60 seconds
[17:09:46][C][safe_mode:021]:   Invoke after 10 boot attempts
[17:09:46][C][safe_mode:023]:   Remain in safe mode for 300 seconds
[17:09:46][C][api:140]: API Server:
[17:09:46][C][api:141]:   Address: respeaker-satellite.local:6053
[17:09:46][C][api:143]:   Using noise encryption: YES
[17:09:46][C][micro_wake_word:072]: microWakeWord:
[17:09:46][C][micro_wake_word:073]:   models:
[17:09:46][C][micro_wake_word:015]:     - Wake Word: Okay Nabu
[17:09:46][C][micro_wake_word:016]:       Probability cutoff: 0.97
[17:09:46][C][micro_wake_word:017]:       Sliding window size: 5
[17:09:46][C][micro_wake_word:015]:     - Wake Word: Stop
[17:09:46][C][micro_wake_word:016]:       Probability cutoff: 0.50
[17:09:46][C][micro_wake_word:017]:       Sliding window size: 5
[17:09:46][C][micro_wake_word:021]:     - VAD Model
[17:09:46][C][micro_wake_word:022]:       Probability cutoff: 0.50
[17:09:46][C][micro_wake_word:023]:       Sliding window size: 5
[17:09:46][D][micro_wake_word:379]: Starting wake word detection
[17:09:46][D][light:036]: 'respeaker-satellite' Setting:
[17:09:46][D][light:047]:   State: OFF
[17:09:46][D][light:109]:   Effect: 'None'
[17:09:46][D][esp-idf:000][microphone_task]: I (75732) I2S: DMA Malloc info, datalen=blocksize=1280, dma_buf_count=4

[17:09:47][D][i2s_audio.microphone:377]: Starting I2S Audio Microphne
[17:09:47][D][i2s_audio.microphone:381]: Started I2S Audio Microphone
[17:09:47][D][micro_wake_word:418]: State changed from IDLE to DETECTING_WAKE_WORD
[17:14:47][D][micro_wake_word:357]: Detected 'Okay Nabu' with sliding average probability is 0.99 and max probability is 1.00
[17:14:47][D][voice_assistant:516]: State changed from IDLE to START_MICROPHONE
[17:14:47][D][voice_assistant:522]: Desired state set to START_PIPELINE
[17:14:47][D][voice_assistant:225]: Starting Microphone
[17:14:47][D][ring_buffer:034]: Created ring buffer with size 16384
[17:14:47][D][voice_assistant:516]: State changed from START_MICROPHONE to STARTING_MICROPHONE
[17:14:47][D][voice_assistant:516]: State changed from STARTING_MICROPHONE to START_PIPELINE
[17:14:47][D][voice_assistant:280]: Requesting start...
[17:14:47][D][voice_assistant:516]: State changed from START_PIPELINE to STARTING_PIPELINE
[17:14:47][W][respeaker_lite:063]: Request mic mute state failed
[17:14:47][W][respeaker_lite:063]: Request mic mute state failed
[17:14:47][W][respeaker_lite:063]: Request mic mute state failed
[17:14:47][W][respeaker_lite:063]: Request mic mute state failed
WARNING respeaker-satellite @ 192.168.0.60: Connection error occurred: [Errno 104] Connection reset by peer
INFO Processing unexpected disconnect from ESPHome API for respeaker-satellite @ 192.168.0.60
WARNING Disconnected from API
INFO Successfully connected to respeaker-satellite @ 192.168.0.60 in 0.495s
INFO Successful handshake with respeaker-satellite @ 192.168.0.60 in 0.222s

formatBCE · October 23, 2024, 9:27pm

Just updated repo.

New things:

Fixed stop-word behaviour
Fixed loud noise on OTA/software restart
Fixed random mute turn on on restart. Now it will be unmuted after restart

**Decided not to expose speaker mute hardware switch to the HA, since it would require sync with media_player state

P.S. merge quirks broke main branch, now everything is intact.

AndyCap · October 24, 2024, 2:47pm

Thanks for the update, I’m getting the following error though:

PackageException: Package version 1.0.0+sha.2597ba8 doesn't satisfy requirements 0.1.0 based on PackageMetadata <type=library name=esp-audio-libs version=1.0.0+sha.2597ba8 spec={'owner': None, 'id': None, 'name': 'esp-audio-libs', 'requirements': '0.1.0', 'uri': 'git+https://github.com/esphome/esp-audio-libs'}

Any ideas?

Thanks

Andy

formatBCE · October 24, 2024, 3:00pm

That’s not really related to the repo. I believe it’s related to some troubles with ESPHome. Try restarting add-on/container, and make sure you cleaned build files on your configuration. Also please make sure you have enough CPU/RAM on machine you’re running ESPHome on - it gives problems often, especially when building for esp-idf.

AndyCap · October 24, 2024, 3:03pm

Thanks, it seems unlucky timing. They updated the version to 1.0.0 an hour ago:

formatBCE · October 24, 2024, 4:03pm

Does it install now?

Charly_X · October 24, 2024, 5:51pm

@formatBCE Hi, I’m trying to follow this for a few days. English is not my native language. Correct me if I’m wrong, but I can’t find a simple diy how to install this to a Respeaker lite kit.
Do I just have to find the yaml and install to the ESP?
Hope you can help me to start playing .
Thanks in advance.

AndyCap · October 24, 2024, 6:35pm

Nope, I upgraded to latest esphome (2024.10.2) but still same problem.

Something is broken:)

KevinA · October 24, 2024, 6:43pm

I broke it… sorry! It should be fixed, as I just bumped the version in our repo.

formatBCE · October 24, 2024, 8:54pm

It didn’t help. I guess I can try bumping it to 1.0.1, but I’ll better wait.
Ok, last PR fixed it, it’s installing!

formatBCE · October 24, 2024, 9:01pm

Hi and welcome!

If you go to README in my repo, there’s simple instructions how to start. The step with YAML you have to do in ESPHome add-on/container. Create new device, copy the code from repository and adjust to your environment (WiFi settings and so on). Right now there’s no simple installer.
I’d say take someone familiar with ESPHome and sit with them for couple of hours (or use ChatGPT).
At this stage, using Respeaker Lite is more of a pain. Sorry, the board is great, but very new and unpolished…

Charly_X · October 24, 2024, 9:53pm

Thank you so much!
I’ll give it a try.
Do I understand correctly your yaml is for the ESP32 S3, so suitable for the Respeaker lite?

formatBCE · October 24, 2024, 9:56pm

That repo is dedicated for Respeaker Lite. YAML fits the kit with S3, soldered to Respeaker Lite. But it’s not full, as it doesn’t contain things like WiFi configuration. Probably I will make web-installer with improv at some point, but now it’s under active development so there’s no point for it.

Charly_X · October 24, 2024, 9:58pm

Thanks a lot. The wifi part I can change. I’m going to give it a try .

formatBCE · October 24, 2024, 10:30pm

If it gives errors that you cannot understand, try clean build files.

And flash Respeaker board first with DFU util, you may download latest 1.0.9 firmware for it from my repo - it’s only 1.0.8 on Seeed site as far as I know.

AndyCap · October 25, 2024, 4:54am

Thanks guys, seems to be building here now.

AndyCap · October 25, 2024, 7:12am

I spoke too soon, I am getting distorted/corrupted audio.

I have tested the 1.0.9 firmware with some code I am working on and the audio out is fine there, so something to do with ESPHome/firmware/HA ?

Everything is updated to the newest versions.

The corruption is not altered by the media player volume.

I have uploaded two mp3 files (both files are normalised).

Media player playing, 80% volume : Dropbox

Media player playing, 0% volume. Halfway through I say “Ok Nabu” so you can hear the corruption changing: Dropbox

formatBCE · October 25, 2024, 12:48pm

This is something very strange. I have 4 devices and didn’t have this once…
Could it be the hardware problem with DAC, or did you have good results before?

AndyCap · October 25, 2024, 7:50pm

Good to know.

I had good results before (I only rebuilt it as I had changed the wifi password!) also perfect audio from my own i2c code for something I have been working on outside of HA.

I have another respeaker, I will try that tomorrow…

Edit: Also I have only tried from the headphone out, will try the speaker tomorrow. I guess there is little chance that will make a difference though.

AndyCap · October 26, 2024, 3:05pm

Ok I have done some investigation, two different respeakers, both 1.0.7 and 1.0.9.

First this is only effecting the headphone out, the speaker out is fine (apart from a bit of hiss if I hold the speaker to my ear).

I used the YAML from seed and @formatBCE and also some test code which I am running outside of HA.

First with my test code I output silence: There is a general noise level at around -70db at -10dBV, digital distortion, seems to mainly cycle at around 390hz. This is probably just poor electronics!

Noise floor spectrum:

With the Seed YAML I get constant broadband noise at about -50db, If I use the wake word and the re speaker is waiting for a response from the server this broadband noise goes and returns when it gets the response.

With The YAML from @formatBCE, I do not get this broadband noise when it isn’t doing anything, I do get the odd bit of corruption though. The moment I say “Ok Nabu” (slow purple LED) though I get the broadband noise, this goes when it is waiting for the server (fast purple LED), returns just before the audio answer (cyan) and then into the start of the audio, then back to the normal noise floor and odd bit of corruption.

It does this with 1.0.7 and 1.0.9 on both respeakers.

Broadband noise spectrum:

Broadband noise wave (amplified):

Before I rebuilt everything the headphone out was fine as I was using it for testing rather rather than the speakers.

The test code I have doesn’t have this issue with the headphone out.

So I’m guessing something is different somewhere in the HA stuff?