try adding this one to the others
CONFIG_ESP32_S3_BOX_BOARD: “y”
Nope, powering everything off the ONN speaker power board, through common six-pin bus bars. The end configuration is to power ALL of the ESP32, microphone, and amplifier from those bars. I’m only using the USB power right now, to load firmware and get logs through the hardwired connection, instead of trying WiFi logs. But the MAX98357 amp is grounded to the ESP32 through a GND pin, not the bus bar. That was an earlier resolution for the speaker noise. I’d guess soldering that jumper could also eliminate that noise while properly powering the amp. But the bus-bar is the better solution, because it can provide more current (2000 mA) than can the ESP32’s 1117c voltage regulator (800 mA).
ok , we will rule that one out. if you check for voltage on the lrclk and sclk when audio should be outputting do you get around 1,5v (give or take) probing the pins on the MAX board ?
just for reference and interest . I have done current draw tests on the s3 with an inmp441, max98357 operating. These are the results. Yellow=V - purple/pink = W and green = A - the area circled in red is when there was input from the mic and output through the DAC. The board is powered by USB.
other stats were …
2 Likes
Before I went to bed it appeared the voice assistant was hung up. I added a couple of buttons so I could change a few things without having to reboot the ESP and then went to bed. 7 1/2 hours later I tried the wake word and on the second attempt I got a response. So with a quite house the thing stayed up overnight. I’m going to turn the TV on and see if it hangs after a few hours and if it does attempt a little more debugging.
It turns out that I got an error condition, which I knew would be a problem if it happened after I stopped the voice assistant code. I’ve since added another change to restart the esp voice assistant code when an error is reported. I’ve seen multiple errors since putting this in place and the system recovered. The full code seems to be working without issue. Hopefully it continues to run for days without hanging now. I’ll post an update if I see it hang at any point. If you try this code, please provide feedback if it is solid for you. That is, the ESP keeps working, accepting new request and giving back some kind of response. Sometimes, I do have to give the wake word twice before the ESP is responsive.
substitutions:
voice_assist_idle_phase_id: '1'
voice_assist_listening_phase_id: '2'
voice_assist_thinking_phase_id: '3'
voice_assist_replying_phase_id: '4'
voice_assist_not_ready_phase_id: '10'
voice_assist_error_phase_id: '11'
voice_assist_muted_phase_id: '12'
esphome:
name: onn-groove-voice-sat
friendly_name: onn groove voice sat
on_boot:
priority: 600
then:
- script.execute: control_led
- delay: 10s
- if:
condition:
lambda: return id(init_in_progress);
then:
- lambda: id(init_in_progress) = false;
- script.execute: control_led
- switch.turn_on: use_wake_word
esp32:
board: esp32dev
framework:
type: esp-idf
# Enable logging
logger:
level: info
# Enable Home Assistant API
api:
encryption:
key: !secret api_key
ota:
password: !secret ota_password
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
# Enable fallback hotspot (captive portal) in case wifi connection fails
ap:
ssid: "FR-Lamp-Touch"
password: !secret wifi_password
captive_portal:
# Change 1 to provide way to restarting some things through the web server via buttons
# Enable Web server.
web_server:
port: 80
button:
- platform: restart
name: "Reboot"
- platform: template
name: "Restart voice assistant"
on_press:
- if:
condition: voice_assistant.is_running
then:
- logger.log:
format: "Stopping voice assistant"
level: INFO
- voice_assistant.stop:
- delay: 1s
- logger.log:
format: "Start voice assistant"
level: INFO
- voice_assistant.start_continuous:
else:
- logger.log:
format: "Start voice assistant"
level: INFO
- voice_assistant.start_continuous:
esp_adf:
external_components:
- source: github://pr#5230
components:
- esp_adf
refresh: 0s
light:
- platform: esp32_rmt_led_strip
rgb_order: GRB
pin: GPIO18
num_leds: 3
rmt_channel: 0
chipset: WS2812
name: "Status LED"
id: led
default_transition_length: 0s
effects:
- pulse:
name: "extra_slow_pulse"
transition_length: 800ms
update_interval: 800ms
min_brightness: 0%
max_brightness: 30%
- pulse:
name: "slow_pulse"
transition_length: 250ms
update_interval: 250ms
min_brightness: 50%
max_brightness: 100%
- pulse:
name: "fast_pulse"
transition_length: 100ms
update_interval: 100ms
min_brightness: 50%
max_brightness: 100%
i2s_audio:
- id: i2s_in
i2s_lrclk_pin: GPIO25
i2s_bclk_pin: GPIO26
- id: i2s_out
i2s_lrclk_pin: GPIO32
i2s_bclk_pin: GPIO13
microphone:
platform: i2s_audio
id: external_microphone
adc_type: external
i2s_audio_id: i2s_in
i2s_din_pin: GPIO34
pdm: false
bits_per_sample: 32bit
speaker:
platform: i2s_audio
id: external_speaker
dac_type: external
i2s_audio_id: i2s_out
i2s_dout_pin: GPIO12
mode: mono
voice_assistant:
id: va
microphone: external_microphone
speaker: external_speaker
use_wake_word: true
noise_suppression_level: 2
auto_gain: 31dBFS
volume_multiplier: 2.5
on_listening:
- lambda: id(voice_assistant_phase) = ${voice_assist_listening_phase_id};
- script.execute: control_led
on_stt_vad_end:
- lambda: id(voice_assistant_phase) = ${voice_assist_thinking_phase_id};
# change 2: stop voice assistant speach is no longer being processed to text
- voice_assistant.stop
- script.execute: control_led
on_tts_stream_start:
- lambda: id(voice_assistant_phase) = ${voice_assist_replying_phase_id};
- script.execute: control_led
on_tts_stream_end:
- lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
- script.execute: control_led
# change 3: restart voice after it was stopped for wake word
- delay: 1s
- voice_assistant.start
on_error:
- if:
condition:
lambda: return !id(init_in_progress);
then:
- lambda: id(voice_assistant_phase) = ${voice_assist_error_phase_id};
- script.execute: control_led
- delay: 1s
- if:
condition:
switch.is_on: use_wake_word
then:
- lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
else:
- lambda: id(voice_assistant_phase) = ${voice_assist_muted_phase_id};
- script.execute: control_led
# change 4: restart voice assistant on any error
- logger.log:
format: "Error so stopping voice assistant"
level: INFO
- voice_assistant.stop:
- delay: 1s
- logger.log:
format: "Re-starting voice assistant after error detected"
level: INFO
- voice_assistant.start
on_client_connected:
- if:
condition:
switch.is_on: use_wake_word
then:
- voice_assistant.start_continuous
- lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
else:
- lambda: id(voice_assistant_phase) = ${voice_assist_muted_phase_id};
- script.execute: control_led
on_client_disconnected:
- lambda: id(voice_assistant_phase) = ${voice_assist_not_ready_phase_id};
- script.execute: control_led
switch:
- platform: template
name: Use Wake Word
id: use_wake_word
optimistic: true
restore_mode: RESTORE_DEFAULT_ON
on_turn_on:
- if:
condition:
lambda: return !id(init_in_progress);
then:
- lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
- if:
condition:
not:
- voice_assistant.is_running
then:
- voice_assistant.start_continuous
- script.execute: control_led
on_turn_off:
- if:
condition:
lambda: return !id(init_in_progress);
then:
- voice_assistant.stop
- lambda: id(voice_assistant_phase) = ${voice_assist_muted_phase_id};
- script.execute: control_led
globals:
- id: init_in_progress
type: bool
restore_value: no
initial_value: 'true'
- id: voice_assistant_phase
type: int
restore_value: no
initial_value: ${voice_assist_not_ready_phase_id}
script:
- id: control_led
then:
- if:
condition:
lambda: return !id(init_in_progress);
then:
- if:
condition:
wifi.connected:
then:
- if:
condition:
api.connected:
then:
- lambda: |
switch(id(voice_assistant_phase)) {
case ${voice_assist_listening_phase_id}:
id(led).turn_on().set_rgb(0, 0, 1).set_brightness(1.0).set_effect("none").perform();
break;
case ${voice_assist_thinking_phase_id}:
id(led).turn_on().set_rgb(0, 1, 0).set_effect("slow_pulse").perform();
break;
case ${voice_assist_replying_phase_id}:
id(led).turn_on().set_rgb(0, 0, 1).set_brightness(1.0).set_effect("fast_pulse").perform();
break;
case ${voice_assist_error_phase_id}:
id(led).turn_on().set_rgb(1, 1, 1).set_brightness(.5).set_effect("none").perform();
break;
case ${voice_assist_muted_phase_id}:
id(led).turn_off().perform();
break;
case ${voice_assist_not_ready_phase_id}:
id(led).turn_on().perform();
break;
default:
id(led).turn_on().set_rgb(1, 0, 0).set_brightness(0.2).set_effect("none").perform();
break;
}
else:
- light.turn_off:
id: led
else:
- light.turn_off:
id: led
else:
- light.turn_on:
id: led
blue: 50%
red: 50%
green: 50%
effect: "fast_pulse"
Here is my working config for these S3 Dev boards. Audio quality seems better on these boards, although the speaker buffer issue has happened a couple of times. Im still not convinced that is a hardware issue anyway…:
substitutions:
voice_assist_idle_phase_id: '1'
voice_assist_listening_phase_id: '2'
voice_assist_thinking_phase_id: '3'
voice_assist_replying_phase_id: '4'
voice_assist_not_ready_phase_id: '10'
voice_assist_error_phase_id: '11'
voice_assist_muted_phase_id: '12'
esphome:
name: master-bedroom-voice-assistant
friendly_name: Master Bedroom Voice assistant
platformio_options:
board_build.flash_mode: dio
on_boot:
priority: 600
then:
- script.execute: control_led
- delay: 30s
- if:
condition:
lambda: return id(init_in_progress);
then:
- lambda: id(init_in_progress) = false;
- script.execute: control_led
psram:
mode: octal
speed: 80MHz
esp32:
board: esp32-s3-devkitc-1
variant: esp32s3
framework:
type: esp-idf
version: recommended
components:
- name: esphome_board
source: github://jesserockz/esphome-esp-adf-board@main
refresh: 0s
sdkconfig_options:
CONFIG_ESP32_S3_BOX_BOARD: "y"
CONFIG_ESP32S3_DEFAULT_CPU_FREQ_240: "y"
CONFIG_ESP32S3_DATA_CACHE_64KB: "y"
CONFIG_ESP32S3_DATA_CACHE_LINE_64B: "y"
CONFIG_AUDIO_BOARD_CUSTOM: "y"
# Enable logging
logger:
# Enable Home Assistant API
api:
encryption:
key: "snip"
on_client_connected:
- script.execute: control_led
on_client_disconnected:
- script.execute: control_led
ota:
password: "snip"
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
manual_ip:
static_ip: *
gateway: *
subnet: *
# Enable fallback hotspot (captive portal) in case wifi connection fails
ap:
ssid: "Master-Bedroom-Voice-Assistant"
password: "snip"
esp_adf:
external_components:
- source: github://pr#5230
components:
- esp_adf
refresh: 0s
captive_portal:
light:
- platform: esp32_rmt_led_strip
rgb_order: GRB
pin: GPIO18
num_leds: 3
rmt_channel: 0
chipset: WS2812
name: "Status LED"
id: led
default_transition_length: 0s
effects:
- pulse:
name: "extra_slow_pulse"
transition_length: 800ms
update_interval: 800ms
min_brightness: 0%
max_brightness: 30%
- pulse:
name: "slow_pulse"
transition_length: 250ms
update_interval: 250ms
min_brightness: 50%
max_brightness: 100%
- pulse:
name: "fast_pulse"
transition_length: 100ms
update_interval: 100ms
min_brightness: 50%
max_brightness: 100%
i2s_audio:
- id: i2s_in
i2s_lrclk_pin: GPIO3 ##INMP441-WS
i2s_bclk_pin: GPIO2 ##INMP441-SCK
- id: i2s_out
i2s_lrclk_pin: GPIO6 ##Max98357 - LRC
i2s_bclk_pin: GPIO20 ###Max98357 - BCLK
## l/R pin on ##INMP441 is connected to ground
microphone:
platform: i2s_audio
id: external_microphone
adc_type: external
i2s_audio_id: i2s_in
i2s_din_pin: GPIO4
channel: left
pdm: false
speaker:
platform: i2s_audio
id: external_speaker
dac_type: external
i2s_audio_id: i2s_out
i2s_dout_pin: GPIO8 ###Max98357 - DIN
mode: mono
voice_assistant:
id: va
microphone: external_microphone
speaker: external_speaker
use_wake_word: true
noise_suppression_level: 2
auto_gain: 31dBFS
volume_multiplier: 2.5
on_listening:
- lambda: id(voice_assistant_phase) = ${voice_assist_listening_phase_id};
- script.execute: control_led
on_stt_vad_end:
- lambda: id(voice_assistant_phase) = ${voice_assist_thinking_phase_id};
- script.execute: control_led
on_tts_stream_start:
- lambda: id(voice_assistant_phase) = ${voice_assist_replying_phase_id};
- script.execute: control_led
on_tts_stream_end:
- lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
- script.execute: control_led
on_error:
- if:
condition:
lambda: return !id(init_in_progress);
then:
- lambda: id(voice_assistant_phase) = ${voice_assist_error_phase_id};
- script.execute: control_led
- delay: 1s
- if:
condition:
switch.is_on: use_wake_word
then:
- lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
else:
- lambda: id(voice_assistant_phase) = ${voice_assist_muted_phase_id};
- script.execute: control_led
on_client_connected:
- if:
condition:
switch.is_on: use_wake_word
then:
- voice_assistant.start_continuous
- lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
else:
- lambda: id(voice_assistant_phase) = ${voice_assist_muted_phase_id};
- script.execute: control_led
on_client_disconnected:
- lambda: id(voice_assistant_phase) = ${voice_assist_not_ready_phase_id};
- script.execute: control_led
switch:
- platform: template
name: Use Wake Word
id: use_wake_word
optimistic: true
restore_mode: RESTORE_DEFAULT_ON
on_turn_on:
- if:
condition:
lambda: return !id(init_in_progress);
then:
- lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
- if:
condition:
not:
- voice_assistant.is_running
then:
- voice_assistant.start_continuous
- script.execute: control_led
on_turn_off:
- if:
condition:
lambda: return !id(init_in_progress);
then:
- voice_assistant.stop
- lambda: id(voice_assistant_phase) = ${voice_assist_muted_phase_id};
- script.execute: control_led
globals:
- id: init_in_progress
type: bool
restore_value: no
initial_value: 'true'
- id: voice_assistant_phase
type: int
restore_value: no
initial_value: ${voice_assist_not_ready_phase_id}
script:
- id: control_led
then:
- if:
condition:
lambda: return !id(init_in_progress);
then:
- if:
condition:
wifi.connected:
then:
- if:
condition:
api.connected:
then:
- lambda: |
switch(id(voice_assistant_phase)) {
case ${voice_assist_listening_phase_id}:
id(led).turn_on().set_rgb(0, 0, 1).set_brightness(1.0).set_effect("none").perform();
break;
case ${voice_assist_thinking_phase_id}:
id(led).turn_on().set_rgb(0, 1, 0).set_effect("slow_pulse").perform();
break;
case ${voice_assist_replying_phase_id}:
id(led).turn_on().set_rgb(0, 0, 1).set_brightness(1.0).set_effect("fast_pulse").perform();
break;
case ${voice_assist_error_phase_id}:
id(led).turn_on().set_rgb(1, 1, 1).set_brightness(.5).set_effect("none").perform();
break;
case ${voice_assist_muted_phase_id}:
id(led).turn_off().perform();
break;
case ${voice_assist_not_ready_phase_id}:
id(led).turn_on().perform();
break;
default:
id(led).turn_on().set_rgb(1, 0, 0).set_brightness(0.2).set_effect("none").perform();
break;
}
else:
- light.turn_off:
id: led
else:
- light.turn_off:
id: led
else:
- light.turn_on:
id: led
blue: 50%
red: 50%
green: 50%
effect: "fast_pulse"
1 Like
Finally got that to compile and burn to the board. It ends up in this reboot loop
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x7 (TG0WDT_SYS_RST),boot:0x28 (SPI_FAST_FLASH_BOOT)
Saved PC:0x400454d5
SPIWP:0xee
mode:QIO, clock div:1
load:0x3fce3808,len:0x162c
ets_loader.c 78
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x7 (TG0WDT_SYS_RST),boot:0x28 (SPI_FAST_FLASH_BOOT)
Saved PC:0x400454d5
SPIWP:0xee
mode:QIO, clock div:1
load:0x3fce3808,len:0x162c
ets_loader.c 78
Now nothing is connected to the pins on the chip. I’m not sure if that could some how cause the boot issue? Will probably try and connect up all the pins to the microphone and amp just to make sure that isn’t causing this loop.
What board are you using?
For the one that isn’t working I’m using the M5Stamps3. For the one that is working I’m using the ESP32 you indicate you’re using. I just have a few of the M5stampS3 sitting around and was hoping they might provide a more powerful CPU option. I’m thinking I might have to pick up the other S3 mentioned in this thread if I want the more powerful board. I’m assuming you had the hanging issues with the original configuration you provide. Any chance you’ll try the mods I posted?
Im rarely getting the speaker buffer error anymore. I did put your mods in place on one for testing.
What mods did you make to clean things up? Prior to trying your approach, I tried the ATOM Echo Smart Speaker and it hangs. I tried the ESP32 S3 Box3, which was a lot better than the ATOM, but it would hang periodically. My initial build based on your approach would hang, much like the ATOM. I’ve been hoping to get something clean. The modes I provided cleaned up the hangs and cleared the studders up a lot. I still have some minor studder, but it’s not bad. The studder typically happen at the start of the response play back. Longer response phases play pretty clean. You mentioned openAI, I’m curious if the openAI works together with the HA voice assistant? I’m planning on looking at that next, assuming the HA voice assistant stays alive over 24 hours.
Thes s3 boards are just not cutting it. They just die after a bit and need to be unpluged, then plugged back in. The ones from the older boards are pretty reliable.
I really hate hearing that. 
It’s disappointing.
But not entirely surprising, either. </skip_rant>
I believe it was an unrelated issue. I’m testing them now for stability.
I’ll be honest, I took a day off from fighting with it yesterday. 
Integrating the changes supplied by @bkprath and @robgough1970 I would hope for a more stable experience with this board. I think the idea of stopping and restarting the voice_assistant is a good move, so long as it doesn’t introduce long delays.
Going to add wires soldered to the D+ and D- signals on that Onn power/USB board, specifically to enable data connections using the external USB-C connector. Ordered a few of these USB-C connectors to use.
There is a delay between completion of response and start of detection of the next wake word, as a lot of times I have to say the wake word twice. However, I do not know if that was the result of my change or just the way it would work. A lot of times I have to say the wake word twice when initiating the first command. For me stability was a big issue. With the change I don’t have any lost in space moments, the ones that required restart of the board to get things going again. And audio is a lot better.
Starting and stopping the voice assistant was the issue I was having with the S3 boards. They would work at first, then just not respond after a bit of time.
I removed that code from one of them and left it in the other last night. This morning, the one I removed that code from responded and the one that had the code did not.
This doesn’t seem to affect the other boards but I removed the code anyway.
Here is a vid of me showing the 4 I have built right after each other. Got a bit of choppiness in one. I talk a bit about what I did that I think helps that speaker buffer issue.
5 Likes
Nice video. You mention that you turn off wake word detection when you’re out of the room. Turning off/on wake word detection would actually be similar to my cycling the ESP voice assistant code during command processing. Since I’ve had they system hang problems with the other platforms people are using it makes sense that some network latency could be what triggers the issue within the ESP voice assistant code. I’m going to see if I get better performance simply by putting my one unit closer to my WiFi router. Which ESP32S3 are you using, I looked into the code and M5StampS3 I tried to use are not supported by the code base?
Any chance you have reference for doing the openAI intergration if HA doesn’t get an answer?