ESp32 Voice Assistant - Help (No microphone seemingly..)

Whenever using the espressif esp-idf platform, it is crucial to clean all build files before rebuilding your project. Try that before doing a clean build.

@FredTheFrog - tried that - build clean and reinstalled the current config. Same issue.

I get random files but there is no sound in the wav file at all.

I would suggest a faulty mic then. Try another one.

@Arh

ok - took another mic off an esp32 I was/am using for WLED - so I know it works.

Plugged it in - and rebooted the esp32 running esphome -

Got this:
2 wav files - same name - 44 bytes - but no sound.

It’s ‘almost’ like it created the files itself - I could be wrong - but those seemed to be there on boot of the esphome esp32 bceause I cleared the files before plugging it in and on speaking to it - I get no more files.

That’s frustrating. :frowning: I can tell you this, the inexpensive INMP441 microphones from Amazon are no match for the higher-priced Adafruit SPH0645LM microphones. Better pin layout, better quality control with Adafruit.

1 Like

@FredTheFrog - yeah, you might be right on this one. I mean they have been working for WLED soundreactive esp devices, but maybe no bueno for the HASS voice? It’s a consideration for sure. Thanks for taking the time to respond to my thread!

As someone who is a former Rhasspy user, and having experienced many of these frustrations building HA Voice Assistants in the last few weeks, I’m VERY happy to share my experiences. Currently lamenting the lack of response to my posts, so I’m trying to remedy that where I can.

@FredTheFrog - yes, I’ve noticed this - a lot fo threads, not a lot of input.

Do you have a working voice assistant yet at all?

If so, not including the mic you just mentioned, would you mind sharing which esp32 board you’re using as well (if you have a working setup…).

If not - no worries.

Thanks!

I had the unfortunate experience of receiving several bad ESP32-S3-N16R8 boards from one Amazon vendor (AITRIP) in my first purchase. Switched to DWEII vendor on Amazon using this ESP32-S3 board. Better quality in my experience, even though the boards looked 100% identical. Very likely from the same factory, just different batches.

The little round mics kinda worked okay, but were a PAIN during soldering and unit assembly. The two rows of three pins and requirement for the L/R pin to be grounded made firm, reliable fixture of the mic a serious issue. The Adafruit mics have six pins in a straight line down one edge of the board, making things a bit easier. They also default to LEFT placement, unless you tie the L/R pin to 3V3 power.

I have two major concerns with the HA voice assist devices right now:

  1. reliability/repeatability of the ESPhome voice_assistant code. It often doesn’t respond every time. Whether that’s down to the wake word processing or code/loop timing, I just don’t know. It also sometimes gets stuck in ‘thinking’ mode when things get out of sync with the HA server.
  2. output responses stuttering/choppy/incomplete - occasionally, they are almost perfect (but not quite) but most of the time, they are terrible. :frowning:
1 Like

@FredTheFrog - thanks, so to ask one more question have you been able to make a working voice assistant with that mic and those esp32 boards (I don’t mind fighting with yaml/configs, but I like to know if someone at least got the hardware running…)

Just added to the end of my previous post.
Yes, it ‘works’ (kind of).
No, it’s not nearly where I’d like it to be.

1 Like

Ah - had not seen your response yet. - thanks for the input, I appreciate it!!!

1 Like

I have 2 voice assist working very well using the SPH0645LM4H mic and MAX98357. On an esp32 wroom board. They have been working for some time without issue.

This is the code I use.

esphome:
  name: assistant
  friendly_name: Assistant
  on_boot:
     - priority: -100
       then:
         - wait_until: api.connected
         - delay: 1s
         - if:
             condition:
               switch.is_on: use_wake_word
             then:
               - voice_assistant.start_continuous:

esp32:
  board: esp32dev
  framework:
    type: esp-idf
    version: recommended

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: "stuff"
ota:
  password: "stuff"

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password



  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Esp32-Mic-Speaker"
    password: "9vYvAFzzPjuc"

improv_serial:

i2s_audio:
  i2s_lrclk_pin: GPIO27
  i2s_bclk_pin: GPIO26

microphone:
  - platform: i2s_audio
    id: echo_microphone
    i2s_din_pin: GPIO13
    adc_type: external
    pdm: false

speaker:
  - platform: i2s_audio
    id: echo_speaker
    i2s_dout_pin: GPIO25
    dac_type: external
    mode: mono

voice_assistant:
  id: va
  microphone: echo_microphone
  speaker: echo_speaker
  noise_suppression_level: 2
  #auto_gain: 31dBFS
  volume_multiplier: 2.0
  vad_threshold: 3
  on_listening:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 0%
        brightness: 10%
        effect: pulse
  on_tts_start:
    - light.turn_on:
        id: led
        blue: 0%
        red: 0%
        green: 100%
        brightness: 100%
        effect: pulse
  on_end:
    - delay: 100ms
    - wait_until:
        not:
          speaker.is_playing:
    - script.execute: reset_led
  on_error:
    - light.turn_on:
        id: led
        blue: 0%
        red: 100%
        green: 0%
        brightness: 100%
        effect: none
    - delay: 1s
    - script.execute: reset_led
    - script.wait: reset_led
    - lambda: |-
        if (code == "wake-provider-missing" || code == "wake-engine-missing") {
          id(use_wake_word).turn_off();
        }

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO39
      inverted: true
    name: Button
    disabled_by_default: true
    entity_category: diagnostic
    id: echo_button
    on_click:
      - if:
          condition:
            switch.is_off: use_wake_word
          then:
            - if:
                condition: voice_assistant.is_running
                then:
                  - voice_assistant.stop:
                  - script.execute: reset_led
                else:
                  - voice_assistant.start:
          else:
            - voice_assistant.stop
            - delay: 1s
            - script.execute: reset_led
            - script.wait: reset_led
            - voice_assistant.start_continuous:

light:
  - platform: esp32_rmt_led_strip
    id: led
    name: None
    disabled_by_default: true
    entity_category: config
    pin: GPIO14
    default_transition_length: 0s
    chipset: ws2812
    num_leds: 1
    rgb_order: grb
    rmt_channel: 0
    effects:
      - pulse:
          transition_length: 250ms
          update_interval: 250ms

script:
  - id: reset_led
    then:
      - if:
          condition:
            - switch.is_on: use_wake_word
            - switch.is_on: use_listen_light
          then:
            - light.turn_on:
                id: led
                blue: 100%
                red: 100%
                green: 0%
                brightness: 30%
                effect: none
          else:
            - light.turn_off: led

switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(va).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
      - script.execute: reset_led
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(va).set_use_wake_word(false);
      - script.execute: reset_led
  - platform: template
    name: Use Listen Light
    id: use_listen_light
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - script.execute: reset_led
    on_turn_off:
      - script.execute: reset_led

external_components:
  - source: github://pr#5230
    components:
      - esp_adf
    refresh: 0s

esp_adf:

I am also playing with some esp32 s3 n16r8 boards using micro wake word. These are using imp441 mics and don’t seem to be as responsive as the other mics. But this is very much a work in progress right now.

See this for code samples, none of it is my work esphome:

2 Likes

@Arh - thanks for all that info very helpful! Would you mind to link the hardware you bought and are using? If not it’s ok, but thought I’d ask. I’m specifically very interested in which esp32 you’re using as it seems some variants work well when others do not (seemingly to me anyway…)

Thanks for taking the time to respond/help!

This is the n16r8. I am using for the micro wake word experiments.

As for the boards in my working voice assistants i not sure where they came from as I have had them for some time, but they look.like these.

1 Like

Cool - your 2nd link is what I’m working with basically. That helps. Thanks again for all the help/response!

Tossing this into the mix:

I know all the containers are running - and I think the piper one only became available once I did a test via the HASS gui. There is a github issue opened for this - but nothing has come of it. Considering this, would this cause what I’m seeing as well?

image


image

Hello @vexter0944 ,
I have exactly the same issues as you with a similar setup. HA, OpenWakeWord, Whisper, Piper work in Helm charts on my NAS via TrueCharts. My pipeline works perfectly in debug mode from my PC, the microphone works with an ESP32-S3 and microwakeword (but still generates the same 44 octets file for Whisper as well).
Have you solved your issues? I think that voice assist is not meant to work anywhere other than on HAOS.

@egreh4trj896436sfz

I unfortunately have not solved my issues. Thanks for jumping in with your results, appreciate the input. I feel like the whole voice assist is about half baked right now, so I’m going to hold off for more developments and see where it goes.

1 Like

Thank you for your feedback. I hope the issues are resolved quickly. I’m not sure how to address the problem or even participate in any way to its resolution.