ESp32 Voice Assistant - Help (No microphone seemingly..)

Configuration Voice Assistant
1

I’m at a serious loss here - I’ve tried everything, does anyone have an esp32 (the one I’m using,…) (https://www.amazon.com/gp/product/B08D5ZD528/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&th=1)
working??

I’m using the INMP441 MEMS microphone - and NO speaker at this time (I’d just like to see the thing do SOMETHING).

I’ve tried I don’t know how many yaml files/pins you name it - I never see anything about it picking up a wake word.

Here’s my current non-working YAML. I have 100% confirmed the mic is working by flashing this with the Audioreactive fork of WLED and it reacts to music correctly.

I run HASS in a docker container, so I also have openwakeword, esphome, whisper and piper all in containers.

SO frustrated. </vent off>

esphome:
  name: esphome-voice-main-bedroom
  friendly_name: Voice main bedroom
  on_boot:
     - priority: -100
       then:
         - wait_until: api.connected
         - delay: 1s
         - if:
             condition:
               switch.is_on: use_wake_word
             then:
               - voice_assistant.start_continuous:

esp32:
  board: esp32dev
  framework:
    type: esp-idf
    version: recommended

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: "/L0DhB1BoUfurwkwKgctKVZ16/9BPtRaFU+y/khaNko="

ota:


wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password


  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Esphome-voice-main-bedroom"
    password: "" # VGiTqrtnA52n


i2s_audio:
  i2s_lrclk_pin: GPIO27
  i2s_bclk_pin: GPIO22

microphone:
  - platform: i2s_audio
    id: mic_i2s
    adc_type: external
    i2s_din_pin: GPIO21
    pdm: false

speaker:
  - platform: i2s_audio
    id: speaker_i2s
    dac_type: external
    i2s_dout_pin: GPIO18
    mode: mono

voice_assistant:
  microphone: mic_i2s
  speaker: speaker_i2s
  use_wake_word: true
  noise_suppression_level: 4
  auto_gain: 15dBFS
  volume_multiplier: 1.0
  id: assist
  on_end:
  - light.turn_off:
      id: led_ring
  on_wake_word_detected:
  - light.addressable_set:
      id: led_ring
      range_from: 16
      range_to: 17
      red: 0%
      green: 0%
      blue: 100%
  - delay: 0.03s
  - light.addressable_set:
      id: led_ring
      range_from: 15
      range_to: 18
      red: 0%
      green: 0%
      blue: 100%
  - delay: 0.03s
  - light.addressable_set:
      id: led_ring
      range_from: 14
      range_to: 19
      red: 0%
      green: 0%
      blue: 100%
  - delay: 0.03s
  - light.addressable_set:
      id: led_ring
      range_from: 13
      range_to: 20
      red: 0%
      green: 0%
      blue: 100%
  - delay: 0.03s
  - light.addressable_set:
      id: led_ring
      range_from: 12
      range_to: 21
      red: 0%
      green: 0%
      blue: 100%
  - delay: 0.03s
  - light.addressable_set:
      id: led_ring
      range_from: 11
      range_to: 22
      red: 0%
      green: 0%
      blue: 100%
  - delay: 0.03s
  - light.addressable_set:
      id: led_ring
      range_from: 10
      range_to: 23
      red: 0%
      green: 0%
      blue: 100%
  - delay: 0.03s
  - light.addressable_set:
      id: led_ring
      range_from: 9
      range_to: 1
      red: 0%
      green: 0%
      blue: 100%
  - delay: 0.03s
  - light.addressable_set:
      id: led_ring
      range_from: 9
      range_to: 24
      red: 0%
      green: 0%
      blue: 100%
  - delay: 0.03s
  - light.addressable_set:
      id: led_ring
      range_from: 7
      range_to: 8
      red: 0%
      green: 50%
      blue: 50%
  - light.addressable_set:
      id: led_ring
      range_from: 0
      range_to: 1
      red: 0%
      green: 50%
      blue: 50%
  - delay: 0.1s
  - light.addressable_set:
      id: led_ring
      range_from: 2
      range_to: 6
      red: 0%
      green: 100%
      blue: 00%

switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(assist).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(assist).set_use_wake_word(false);

light:
  - platform: esp32_rmt_led_strip
    rgb_order: GRB
    pin: GPIO23
    num_leds: 24
    rmt_channel: 0
    chipset: ws2812
    name: "led_ring"
    id: led_ring

captive_portal:
2

If you add this to configuration.yaml. And create the linked folder, you will be able to play back what your mic is recording, this can help understand why wake word is not responding. My experince was distorted sound. you will need to restart HA and do not leave this in the setting once you have tested or it will fill up you storage quickly.

assist_pipeline:
  debug_recording_dir: /share/assist_pipeline
3

If you have grounded the L/R pin on the INMP441 microphone, then it is configured for the LEFT channel. In your microphone: block, add the following:

  channel: left

and that should help immensely. :slight_smile:

4

@FredTheFrog - sad tuba.mp3 - no difference. Thanks for the suggestion though!

5

@Arh

Thanks for the thought - no dice - I get nothing.

Now to be sure - I run HASS in a Docker container, so I created a ‘Temp’ folder in the root of my mounted folder (where the configuration.yaml is loaded from) and I’ve yet to see anything show up in the temp folder.

image
image2288×147 7.36 KB

so in my docker_files folder under homassistant (on the local ubuntu server) I mount that as /config in the container. So I created a temp folder in there and listed it as /config/temp in the configuration.yaml - so it ‘should’ work

…update…

ok as I was typing this all out - I got this in the folder. I downloaded it but it plays nothing at all.

image
image795×71 2.94 KB

Also screenshot of esphome logs.

image
image757×797 25.5 KB

6

Whenever using the espressif esp-idf platform, it is crucial to clean all build files before rebuilding your project. Try that before doing a clean build.

7

@FredTheFrog - tried that - build clean and reinstalled the current config. Same issue.

I get random files but there is no sound in the wav file at all.

8

I would suggest a faulty mic then. Try another one.

9

@Arh

ok - took another mic off an esp32 I was/am using for WLED - so I know it works.

Plugged it in - and rebooted the esp32 running esphome -

Got this:
2 wav files - same name - 44 bytes - but no sound.

image
image716×273 6.01 KB

It’s ‘almost’ like it created the files itself - I could be wrong - but those seemed to be there on boot of the esphome esp32 bceause I cleared the files before plugging it in and on speaking to it - I get no more files.

10

That’s frustrating. :frowning: I can tell you this, the inexpensive INMP441 microphones from Amazon are no match for the higher-priced Adafruit SPH0645LM microphones. Better pin layout, better quality control with Adafruit.

1 Like
11

@FredTheFrog - yeah, you might be right on this one. I mean they have been working for WLED soundreactive esp devices, but maybe no bueno for the HASS voice? It’s a consideration for sure. Thanks for taking the time to respond to my thread!

12

As someone who is a former Rhasspy user, and having experienced many of these frustrations building HA Voice Assistants in the last few weeks, I’m VERY happy to share my experiences. Currently lamenting the lack of response to my posts, so I’m trying to remedy that where I can.

13

@FredTheFrog - yes, I’ve noticed this - a lot fo threads, not a lot of input.

Do you have a working voice assistant yet at all?

If so, not including the mic you just mentioned, would you mind sharing which esp32 board you’re using as well (if you have a working setup…).

If not - no worries.

Thanks!

14

I had the unfortunate experience of receiving several bad ESP32-S3-N16R8 boards from one Amazon vendor (AITRIP) in my first purchase. Switched to DWEII vendor on Amazon using this ESP32-S3 board. Better quality in my experience, even though the boards looked 100% identical. Very likely from the same factory, just different batches.

The little round mics kinda worked okay, but were a PAIN during soldering and unit assembly. The two rows of three pins and requirement for the L/R pin to be grounded made firm, reliable fixture of the mic a serious issue. The Adafruit mics have six pins in a straight line down one edge of the board, making things a bit easier. They also default to LEFT placement, unless you tie the L/R pin to 3V3 power.

I have two major concerns with the HA voice assist devices right now:

  1. reliability/repeatability of the ESPhome voice_assistant code. It often doesn’t respond every time. Whether that’s down to the wake word processing or code/loop timing, I just don’t know. It also sometimes gets stuck in ‘thinking’ mode when things get out of sync with the HA server.
  2. output responses stuttering/choppy/incomplete - occasionally, they are almost perfect (but not quite) but most of the time, they are terrible. :frowning:
1 Like
15

@FredTheFrog - thanks, so to ask one more question have you been able to make a working voice assistant with that mic and those esp32 boards (I don’t mind fighting with yaml/configs, but I like to know if someone at least got the hardware running…)

16

Just added to the end of my previous post.
Yes, it ‘works’ (kind of).
No, it’s not nearly where I’d like it to be.

1 Like
17

Ah - had not seen your response yet. - thanks for the input, I appreciate it!!!

1 Like
18

I have 2 voice assist working very well using the SPH0645LM4H mic and MAX98357. On an esp32 wroom board. They have been working for some time without issue.

This is the code I use.

esphome:
  name: assistant
  friendly_name: Assistant
  on_boot:
     - priority: -100
       then:
         - wait_until: api.connected
         - delay: 1s
         - if:
             condition:
               switch.is_on: use_wake_word
             then:
               - voice_assistant.start_continuous:

esp32:
  board: esp32dev
  framework:
    type: esp-idf
    version: recommended

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: "stuff"
ota:
  password: "stuff"

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password



  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Esp32-Mic-Speaker"
    password: "9vYvAFzzPjuc"

improv_serial:

i2s_audio:
  i2s_lrclk_pin: GPIO27
  i2s_bclk_pin: GPIO26

microphone:
  - platform: i2s_audio
    id: echo_microphone
    i2s_din_pin: GPIO13
    adc_type: external
    pdm: false

speaker:
  - platform: i2s_audio
    id: echo_speaker
    i2s_dout_pin: GPIO25
    dac_type: external
    mode: mono

voice_assistant:
  id: va
  microphone: echo_microphone
  speaker: echo_speaker
  noise_suppression_level: 2
  #auto_gain: 31dBFS
  volume_multiplier: 2.0
  vad_threshold: 3
  on_listening:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 0%
        brightness: 10%
        effect: pulse
  on_tts_start:
    - light.turn_on:
        id: led
        blue: 0%
        red: 0%
        green: 100%
        brightness: 100%
        effect: pulse
  on_end:
    - delay: 100ms
    - wait_until:
        not:
          speaker.is_playing:
    - script.execute: reset_led
  on_error:
    - light.turn_on:
        id: led
        blue: 0%
        red: 100%
        green: 0%
        brightness: 100%
        effect: none
    - delay: 1s
    - script.execute: reset_led
    - script.wait: reset_led
    - lambda: |-
        if (code == "wake-provider-missing" || code == "wake-engine-missing") {
          id(use_wake_word).turn_off();
        }

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO39
      inverted: true
    name: Button
    disabled_by_default: true
    entity_category: diagnostic
    id: echo_button
    on_click:
      - if:
          condition:
            switch.is_off: use_wake_word
          then:
            - if:
                condition: voice_assistant.is_running
                then:
                  - voice_assistant.stop:
                  - script.execute: reset_led
                else:
                  - voice_assistant.start:
          else:
            - voice_assistant.stop
            - delay: 1s
            - script.execute: reset_led
            - script.wait: reset_led
            - voice_assistant.start_continuous:

light:
  - platform: esp32_rmt_led_strip
    id: led
    name: None
    disabled_by_default: true
    entity_category: config
    pin: GPIO14
    default_transition_length: 0s
    chipset: ws2812
    num_leds: 1
    rgb_order: grb
    rmt_channel: 0
    effects:
      - pulse:
          transition_length: 250ms
          update_interval: 250ms

script:
  - id: reset_led
    then:
      - if:
          condition:
            - switch.is_on: use_wake_word
            - switch.is_on: use_listen_light
          then:
            - light.turn_on:
                id: led
                blue: 100%
                red: 100%
                green: 0%
                brightness: 30%
                effect: none
          else:
            - light.turn_off: led

switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(va).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
      - script.execute: reset_led
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(va).set_use_wake_word(false);
      - script.execute: reset_led
  - platform: template
    name: Use Listen Light
    id: use_listen_light
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - script.execute: reset_led
    on_turn_off:
      - script.execute: reset_led

external_components:
  - source: github://pr#5230
    components:
      - esp_adf
    refresh: 0s

esp_adf:

I am also playing with some esp32 s3 n16r8 boards using micro wake word. These are using imp441 mics and don’t seem to be as responsive as the other mics. But this is very much a work in progress right now.

See this for code samples, none of it is my work esphome:

1 Like
19

@Arh - thanks for all that info very helpful! Would you mind to link the hardware you bought and are using? If not it’s ok, but thought I’d ask. I’m specifically very interested in which esp32 you’re using as it seems some variants work well when others do not (seemingly to me anyway…)

Thanks for taking the time to respond/help!

20

This is the n16r8. I am using for the micro wake word experiments.

As for the boards in my working voice assistants i not sure where they came from as I have had them for some time, but they look.like these.

1 Like