ESP32 S3 Box 3: Why is this so difficult?!

drewsky208 · May 16, 2024, 12:12pm

I’m not sure I’ve done any project in HA that has created more teeth gritting than getting this box to work! I’m working on getting voice assistant cranking. I’ve made lots of progress, even getting it working flawlessly with ChatGPT! I just need to start on a dedicated voice assistant box. I’m starting with this Box 3 before I move into the more exciting realm of ripping apart my google home pucks.

I got the box bootstrapped and on my wifi by the usual means with no real fuss. I installed the Box 3 specific firmware with the short-and-sweet yaml that has the include from a github and my wifi secrets linking.

The box was deaf out of the gate. Microphone wasn’t working correctly. I wandered through searches and found some things to add in after bringing the whole config yaml down local to me. I got it to successfully compile, AND for the microphone to work once! I thought I was on my way. Then on a reboot, no more microphone.

I did much more toiling about for days with little change. I am also plagued with the very routine occurrence of giving the install command and the compile crashing. I do nothing except hit retry and the compile happily proceeds.

An ESPHome update came out yesterday with lots of fixes so I thought I’d give it a shot since I tried everything else. Viola! The microphone works! It hears me, processes the S2T and responds…but…now there is no speaker! I see it sending the reply .wav and the Box 3 playing it but no sound comes out. I reverted back to the “pure stock” version that is on github but same result.

Given the relatively small number of conversations on this, others seem to have gotten this sucker working. Can anyone give me some advice? I’m at my wits end. I’m not a newbie at esphome, i am running esp32 powermon and ratgdo without a single issue, it’s just the Box that is getting me!

I’m sure the first question will be “what’s your config yaml?” So here it is:

substitutions:
  name: esp32-s3-box-3-5aacd8
  friendly_name: ESP32 S3 Box 3
packages:
  esphome.voice-assistant: github://esphome/firmware/wake-word-voice-assistant/esp32-s3-box-3.yaml@main
esphome:
  name: ${name}
  name_add_mac_suffix: false
  friendly_name: ${friendly_name}

(followed by the api key and wifi references of course)

Any advice appreciated!

smcnaught · May 16, 2024, 11:51pm

I did have it working but after the latest ESPHome upgrade to 2024.5.0, it has broken. I have not troubleshooted the issue yet and but when I find the solution I will be providing update here.

smcnaught · May 16, 2024, 11:59pm

Others are reporting the issues as well with no documented solution as of yet.

github.com/esphome/issues

2024.5.0 Voice assistant, no speaker sounds

opened 10:09AM - 15 May 24 UTC

hugobloem

### The problem I am using the ESP32-S3-BOX (non 3) firmware from esphome/firmw…are. However, after updating to esphome 2024.5 I get no voice return. The text on the display does come up correctly and opening the audio link in a browser plays the audio as normal. ### Which version of ESPHome has the issue? 2024.5.0 ### What type of installation are you using? Home Assistant Add-on ### Which version of Home Assistant has the issue? 2024.5 ### What platform are you using? ESP32 ### Board _No response_ ### Component causing the issue _No response_ ### Example YAML snippet _No response_ ### Anything in the logs that might be useful for us? ```txt [11:05:18][D][voice_assistant:591]: Speech recognised as: "Tell me a joke." [11:05:18][D][text_sensor:064]: 'text_request': Sending state 'Tell me a joke.' [11:05:18][W][component:237]: Component voice_assistant took a long time for an operation (240 ms). [11:05:18][W][component:238]: Components should block for at most 30 ms. [11:05:18][D][voice_assistant:563]: Event Type: 5 [11:05:18][D][voice_assistant:596]: Intent started [11:05:19][D][voice_assistant:563]: Event Type: 6 [11:05:19][D][voice_assistant:563]: Event Type: 7 [11:05:19][D][voice_assistant:619]: Response: "I'm here to assist with your smart home. How can I help you today?" [11:05:19][D][text_sensor:064]: 'text_response': Sending state 'I'm here to assist with your smart home. How can I help you today?' [11:05:19][D][voice_assistant:563]: Event Type: 98 [11:05:19][D][voice_assistant:704]: TTS stream start [11:05:19][D][esp-idf:000][speaker_task]: I (258604) I2S: DMA Malloc info, datalen=blocksize=2048, dma_buf_count=8 [11:05:19][D][esp-idf:000][speaker_task]: I (258612) I2S: I2S0, MCLK output by GPIO2 [11:05:19][D][esp-idf:000][speaker_task]: I (258618) ESP32_S3_BOX: I2S0, MCLK output by GPIO0 [11:05:19][D][esp-idf:000][speaker_task]: I (258622) AUDIO_PIPELINE: link el->rb, el:0x3d85c2c8, tag:raw, rb:0x3d85c438 [11:05:19][D][esp-idf:000][speaker_task]: I (258629) AUDIO_ELEMENT: [raw-0x3d85c2c8] Element task created [11:05:19][D][esp-idf:000][speaker_task]: I (258635) AUDIO_ELEMENT: [i2s-0x3d85c024] Element task created [11:05:19][D][esp-idf:000][speaker_task]: I (258640) AUDIO_PIPELINE: Func:audio_pipeline_run, Line:359, MEM Total:8064151 Bytes, Inter:63740 Bytes, Dram:63740 Bytes [11:05:19][D][esp-idf:000][i2s]: I (258646) AUDIO_ELEMENT: [i2s] AEL_MSG_CMD_RESUME,state:1 [11:05:19][D][esp-idf:000][i2s]: I (258648) I2S_STREAM: AUDIO_STREAM_WRITER [11:05:19][D][esp-idf:000][speaker_task]: I (258652) AUDIO_PIPELINE: Pipeline started [11:05:20][W][component:237]: Component voice_assistant took a long time for an operation (280 ms). [11:05:20][W][component:238]: Components should block for at most 30 ms. [11:05:20][D][voice_assistant:563]: Event Type: 8 [11:05:20][D][voice_assistant:639]: Response URL: "http://192.168.1.102:8123/api/tts_proxy/8e80ff9caa1ef21e0bcaaea38ac66211b3483bab_en-gb_2cdeae300d_tts.microsoft.wav" [11:05:20][D][voice_assistant:439]: State changed from AWAITING_RESPONSE to STREAMING_RESPONSE [11:05:20][D][voice_assistant:445]: Desired state set to STREAMING_RESPONSE [11:05:20][D][voice_assistant:563]: Event Type: 2 [11:05:20][D][voice_assistant:653]: Assist Pipeline ended [11:05:21][D][esp-idf:000][speaker_task]: W (260212) AUDIO_PIPELINE: There are no listener registered [11:05:21][D][esp-idf:000][speaker_task]: I (260219) AUDIO_PIPELINE: audio_pipeline_unlinked [11:05:21][D][esp-idf:000][speaker_task]: W (260226) AUDIO_ELEMENT: [i2s] Element has not create when AUDIO_ELEMENT_TERMINATE [11:05:21][D][esp-idf:000][speaker_task]: I (260235) I2S: DMA queue destroyed [11:05:21][D][esp-idf:000][speaker_task]: W (260243) AUDIO_ELEMENT: [filter] Element has not create when AUDIO_ELEMENT_TERMINATE [11:05:21][D][esp-idf:000][speaker_task]: W (260251) AUDIO_ELEMENT: [raw] Element has not create when AUDIO_ELEMENT_TERMINATE [11:05:21][D][esp-idf:000][speaker_task]: I (260291) I2S: DMA Malloc info, datalen=blocksize=2048, dma_buf_count=8 [11:05:21][D][esp-idf:000][speaker_task]: I (260299) I2S: I2S0, MCLK output by GPIO2 [11:05:21][D][esp-idf:000][speaker_task]: I (260309) ESP32_S3_BOX: I2S0, MCLK output by GPIO0 [11:05:21][D][esp-idf:000][speaker_task]: I (260317) AUDIO_PIPELINE: link el->rb, el:0x3d85c2c8, tag:raw, rb:0x3d85c438 [11:05:21][D][esp-idf:000][speaker_task]: I (260325) AUDIO_ELEMENT: [raw-0x3d85c2c8] Element task created [11:05:21][D][esp-idf:000][speaker_task]: I (260333) AUDIO_ELEMENT: [i2s-0x3d85c024] Element task created [11:05:21][D][esp-idf:000][speaker_task]: I (260338) AUDIO_PIPELINE: Func:audio_pipeline_run, Line:359, MEM Total:8064243 Bytes, Inter:63832 Bytes, Dram:63832 Bytes [11:05:21][D][esp-idf:000][i2s]: I (260345) AUDIO_ELEMENT: [i2s] AEL_MSG_CMD_RESUME,state:1 [11:05:21][D][esp-idf:000][i2s]: I (260348) I2S_STREAM: AUDIO_STREAM_WRITER [11:05:21][D][esp-idf:000][speaker_task]: I (260350) AUDIO_PIPELINE: Pipeline started [11:05:25][D][voice_assistant:563]: Event Type: 99 [11:05:25][D][voice_assistant:712]: TTS stream end [11:05:25][D][voice_assistant:310]: End of audio stream received [11:05:25][D][voice_assistant:439]: State changed from STREAMING_RESPONSE to RESPONSE_FINISHED [11:05:25][D][voice_assistant:445]: Desired state set to RESPONSE_FINISHED [11:05:27][D][esp-idf:000][speaker_task]: W (266700) AUDIO_PIPELINE: There are no listener registered [11:05:27][D][esp-idf:000][speaker_task]: I (266707) AUDIO_PIPELINE: audio_pipeline_unlinked [11:05:27][D][esp-idf:000][speaker_task]: W (266716) AUDIO_ELEMENT: [i2s] Element has not create when AUDIO_ELEMENT_TERMINATE [11:05:27][D][esp-idf:000][speaker_task]: I (266723) I2S: DMA queue destroyed ``` ### Additional information _No response_

drewsky208 · May 17, 2024, 12:10pm

Thanks for the reply, late yesterday I started to see people with same problem. That actually makes me happy because I know it’s not something that I borked. In that regard, misery loves company. I’ll hang out and wait for updates.

Stiltjack · May 17, 2024, 12:32pm

I used the Willow add-on. Problem free so far.

jazzmonger · May 17, 2024, 2:59pm

I don’t know why you guys are wasting your time on this. It’s all good if you like science experiments. The HA voice stuff just is NOT ready for prime time. Try Willow. I wrote up a detailed doc explaining how to get it working and IT WORKS. Literally 15 minutes to set it up…

Use the link Stiltjack posted.

drewsky208 · May 18, 2024, 6:59pm

I think you are misunderstanding what my ask is. Willow still needs a voice “box” for you to talk to. I already have the voice assistant part of HA working just fine, I’m struggling with that box.

Stiltjack · May 18, 2024, 7:03pm

ESP32 S3 Box 3 is the voice box. Speech recognition is very good ideed.

JeeBee · May 18, 2024, 8:30pm

No speaker sound here as well, just a “click”.
Watch the log while compiling, there are a lot of issue’s to be solved so it seems .

smcnaught · May 21, 2024, 3:10am

Short term fix - downgrading to an older version of esp-idf:

esp32:
  board: esp32s3box
  flash_size: 16MB
  framework:
    type: esp-idf
    version: 4.4.6

tim.plas · May 21, 2024, 3:54am

I had a startlingly easy go of it with my BOX unit (non S3). A couple weeks ago, I did the following:

Plugged device into a PC with USB OTG cable
Pointed web browser to Ready-Made Projects — ESPHome

Selected my device type
Hit Connect, selected proper COM port, & hit Install.

A couple minutes later, I had a working voice assist satellite device, with working mic, speaker, and cutesy display.

I immediately asked it “Hey Nabu, turn off XX lights”
And much to my surprise, without doing any other config, IT DID IT!
(+ other voice requests)

I haven’t really played with it much since, & don’t know if it’ll have any issues with newest ESPHome code.

But I was absolutely flabbergasted that it came up so easily.

I’m still doing HA voice control here with Alexa and/or Siri. But I’m now convinced that HA Voice Assist is much farther along than I had thought, & I can seriously think about starting to move to it anytime.

You can find all the YAML code that it’s using, via the GitHub link near bottom of page – firmware/voice-assistant at main · esphome/firmware · GitHub

tim.plas · May 21, 2024, 4:16am

Follow-up: If anyone uses the Ready Made Projects to build a voice satellite with M5Stack Atom Echo device, I’d have to agree that unit is more toy like. Almost un-hearable speaker volume. And no wake word – you have to push its button for Push-to-Talk functionality (apparently not enough horsepower to process wake word on device). And mic didn’t always seem to pick up voices clearly.

But even so, it IS a functional voice assist satellite, which worked immediately after loading firmware. (And is cheap way to experiment)

drewsky208 · May 21, 2024, 5:50pm

Thanks for that! I tried that but that ended up breaking the microphone.

tfcarb · May 22, 2024, 3:51pm

I’m in the death spiral of attempting to install as well. I’ve follow the instructions here:

ESP32-S3-BOX voice assistant - Home Assistant (home-assistant.io)

which don’t work as there is never an option to set the WIFI credentials.

I tried this: firmware/wake-word-voice-assistant/esp32-s3-box-3.yaml at main · esphome/firmware · GitHub but it was rejected as being too large.

Anyone with a suggestion on how to proceed with the ESP32-S3-BOX-3 variant?

I agree, it shouldn’t be this hard. This is a bad look for the Home Assistant/ESPHome ecosystem.

pepe59 · May 22, 2024, 4:07pm

Look here

chaotik_penguin · May 23, 2024, 11:27am

I had a similar issue on Arch Linux. So this may not be relevant to you. I found that after the device got flashed the permissions on the USB reverted back and I didn’t have access anymore. As a hack I opened a background terminal and did:

while :; do chmod 666 /dev/ttyACM0; sleep 2;done

Replace ACM0 with your device name. Keep it running until you’re all done then ctrl+c

tfcarb · May 24, 2024, 4:36am

FWIW, I couldn’t get this to work using Windows 11…it did work using using @pepe59’s suggestion on Ubuntu/Chrome. No clue as as to why.

D3SOX · May 24, 2024, 6:38am

I think you need to pin ESPHome to 2024.4.x to fix it.
It works in my branch under GitHub - D3SOX/ESPHome-firmware: Holds firmware configuration files for projects that the ESPHome team provides.

JeeBee · May 25, 2024, 7:09pm

This is really disappointing, there is a lot of blabla going on about voice assist, so I bought this very nice exp32 S3 box 3(the m5stack echo was a rather disappointing experience), got the yaml code into esphome in HA, and I end up with a non-working speaker. It does process voice commands, not as good as Google, but that’s oke for now.
While compiling the code, the log screen is flooded with messages / warnings / errors. How can this be unnoticed by the creators ?
I followed the advice from @smcnaught to switch back to the esp framework version: 4.4.6, that works for me, for now

Lesboy72 · May 27, 2024, 2:37pm

Hello, where exactly should I write these few lines that restore ESPHOME to the previous version?
Does /homeassistant/esphome/esp32-s3-box-3-5a93fc.yaml match the beginning or the end?