EspHome - INMP441 microphone for voice control - pipeline error

Configuration Voice Assistant
1

Hi,
I’m trying to configure INMP441 microphone with EspHome. I’m using ESP32 Devkit v1 with this configuration:

esphome:
  name: ha-microphone
  friendly_name: HA-Microphone

esp32:
  board: esp32dev
  framework:
    type: arduino

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: ""

ota:
  password: ""

wifi:
  ssid: !secret wifi_ssid_espdev
  password: !secret wifi_password_espdev

i2s_audio:
  i2s_lrclk_pin: GPIO33 #ws
  i2s_bclk_pin: GPIO19 #clk

microphone:
- platform: i2s_audio
  i2s_din_pin: GPIO32 #sd
  id: adc_mic
  adc_type: external
  pdm: false

voice_assistant:
  microphone: adc_mic

binary_sensor:
  - platform: gpio
    name: "Voice"
    pin:
      number: GPIO0
      inverted: True
    on_press:
      - voice_assistant.start:
    on_release:
      - voice_assistant.stop:

I have installed Whisper and Wyoming Protocol with this config and log:

model: tiny-int8
language: cs
beam_size: 1

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service whisper: starting
s6-rc: info: service whisper successfully started
s6-rc: info: service discovery: starting
WARNING:wyoming_faster_whisper.download:Model hashes do not match
WARNING:wyoming_faster_whisper.download:Expected: {'config.json': 'e5a2f85afc17f73960204cad2b002633', 'model.bin': '9674f22b7dee7b4d321a46f235ea6c7f', 'vocabulary.txt': 'c1120a13c94a8cbb132489655cdd1854'}
WARNING:wyoming_faster_whisper.download:Got: {'model.bin': '', 'config.json': '', 'vocabulary.txt': ''}
INFO:__main__:Downloading FasterWhisperModel.TINY_INT8 to /data
INFO:__main__:Ready
[14:22:54] INFO: Successfully send discovery information to Home Assistant.
s6-rc: info: service discovery successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started

When I press button for voice record I get in EspHome this, but the pipeline is not connected:

12:45:59][D][binary_sensor:036]: 'Voice': Sending state ON
[12:45:59][D][voice_assistant:132]: Requesting start...
[12:45:59][D][voice_assistant:111]: Starting...
[12:46:02][D][binary_sensor:036]: 'Voice': Sending state OFF
[12:46:02][D][voice_assistant:144]: Signaling stop...

This is log from HomeAssistant:

Logger: homeassistant
Source: components/assist_pipeline/pipeline.py:728
First occurred: 10:55:45 (11 occurrences)
Last logged: 12:15:10

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/esphome/voice_assistant.py", line 317, in run_pipeline
    await async_pipeline_from_audio_stream(
  File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/__init__.py", line 79, in async_pipeline_from_audio_stream
    await pipeline_input.validate()
  File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/pipeline.py", line 728, in validate
    raise PipelineRunValidationError(
homeassistant.components.assist_pipeline.pipeline.PipelineRunValidationError: the pipeline does not support text-to-speech

Also from Homeassistant when I allow Text-to-speech with Google Translate

Logger: homeassistant
Source: components/assist_pipeline/pipeline.py:575
First occurred: 12:36:25 (1 occurrences)
Last logged: 12:36:25

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/esphome/voice_assistant.py", line 317, in run_pipeline
    await async_pipeline_from_audio_stream(
  File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/__init__.py", line 79, in async_pipeline_from_audio_stream
    await pipeline_input.validate()
  File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/pipeline.py", line 747, in validate
    await asyncio.gather(*prepare_tasks)
  File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/pipeline.py", line 575, in prepare_text_to_speech
    raise TextToSpeechError(
homeassistant.components.assist_pipeline.error.TextToSpeechError: Pipeline error code=tts-not-supported, message=Text-to-speech engine google_translate does not support language cs or options {'audio_output': 'mp3'}

System version:

Home Assistant 2023.6.1
Supervisor 2023.06.1
Operating System 10.2
Frontend 20230608.0 - latest

I used actual stable EspHome and also tried development version with the same result.
Wiring between ESP32 and INMP441 is well, I also tried another one with ESP32-S3, but it has the same result.

Home Assistant is not using ssl secure for https, but I thought that this is only necessary for use microphone with Web Browser and isn’t necessary with HA and EspHome.

Assist is working if I write text in HA.

Do you have any idea where can be mistake?

Thank you for any point.

2 Likes
2

Have you enabled to expose to Assist in the Wyoming whisper entity?

image
image1332×584 29.7 KB

3

Hi AnaCoda,

Thank you for answer. I didn’t have it allowed, so i try, but with the same result. Also tried latest esphome update version 2023.10.1 and disabled tts. I will wait for some time and then try again. :slight_smile:

4

I’ve got it working on an esp32_devkitc_v4
First it didn’t work with the esphome project m5stack.
I adopted the firmware that came from the web installer and did a few overrides:

substitutions:
  name: m5stack-atom-echo-d3187c
  friendly_name: M5Stack Atom Echo d3187c
  led_brightness: 10%
packages:
  m5stack.atom-echo-voice-assistant: github://esphome/firmware/voice-assistant/m5stack-atom-echo.yaml@main
esphome:
  name: ${name}
  name_add_mac_suffix: false
  friendly_name: ${friendly_name}
api:
  encryption:
    key: GEHEIM
esp32:
  board: esp32dev

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

# changes start here, above everything is original

i2s_audio:
  i2s_lrclk_pin: GPIO33
  i2s_bclk_pin: GPIO18

microphone:
  - id: !extend echo_microphone
    pdm: false

light:
  - id: !extend led
    chipset: ws2812

binary_sensor:
  - id: !extend echo_button
    pin:
      number: GPIO14

The binary sensor doesn’t matter… the culprit for me was pin 19 for i2s_bclk_pin which i changed to pin 18 and probably the pdm boolean which i changed to false.

According to this pinout: ESP32 DevKitC v4 high resolution pinout and specs – Renzo Mischianti
pin 18 is the SCK pin on this particular board… not an expert though but it works at leat :wink:

Next i have to add and test the dac and speaker to get the same thing running like stated on the Chapter 4 blog post. Both are waiting in my mailbox.

Hope this helps :slight_smile:

Edit:

I have to add that the reception does not work as great as demoed in the videos. Maybe it’s worth waiting for “official” firmware and schematics.

5

For Reference:

Hardware:

  • ESP32: DevKitC V4
  • Mic: INMP441
  • DAC: MAX98357

Firmware

see above

Pinout:

Microphone

Board → ESP32 (label on ESP)

  • VDD → 3.3V
  • GND → GND
  • SD → GPIO23 (23)
  • L/R → GND
  • WS → GPIO33 (33)
  • SCK → GPIO18 (18)
DAC

Board → ESP32 (label on ESP)

  • LRC → GPIO33 (33)
  • BCLK → GPIO18 (18)
  • DIN → GPIO22 (22)
  • GAIN → Unused
  • SD → Unused
  • GND → GND
  • Vin → 3.3V
6

Today I finally got a setup to work reliably I followed Lewis’ guide.

Removing the button and LED from my original setup and using the pins suggested in the video. My speaker is still a bit crackly but it is working as well as my echo atom.

1 Like
7

I had problems with the button. It switched constantly on and off so i activated the internal pulldown to stop the gpio from floating.

@Arh which microphone are you using? Do you have good speech recognition from a distance? I use INMP441 and… well impressing enough but not usable. It does not work reliably from a meter distance. If you use the ICS-43434, where did you buy it?

8

1 of the 2 I am playing with uses an INMP441 the other an SPH0645. Both work well after removing this line

auto_gain: 31dBFS

adding the entry into your config to enable saving the recordings to disk will help you understand what the recordings re like

assist_pipeline:
  debug_recording_dir: /share/assist_pipeline

Its still early days with mine as I have only been playing around for an hour or 2 so far.

1 Like
9

Great, i changed it to the default (0dBFS) and it works way better now. Thanks

I can read the logs of the docker container i run whiser in. I installed it to my homegrown NAS which has way more CPU Power than my HA machine.

Commercial:
Please upvote feature request Assist: "Triggered by" for sentences for having an even better experience :wink:

10

If you guys also experience crackling speech output with a MAX98357 amp you can try to mess around with the gain settings.

  • 15dB if a 100K resistor is connected between GAIN and GND
  • 12dB if GAIN is connected directly to GND
  • 9dB if GAIN is not connected to anything (this is the default)
  • 6dB if GAIN is connected directly to Vin
  • 3dB if a 100K resistor is connected between GAIN and Vin

My speaker works well with the 6dB setting.

2 Likes
11

Whoa, it’s great, finally I used config from this guy and it is working. Used classic ESP32. I didn’t try it with ESP32 S3, maybe later. :slight_smile: Also it works with Wake word. Just need to do something with a few second delay between voice command and realise, but I will need to add more CPU power to virtual PC.

Secondly it will me teach to correctly speak in English like I’m not native speaker and I don’t want use native language because I thought that it will working worse than English.

Great work, thank you. :slight_smile:

12

Guys, I’m losing my Mind…
I’ve messed around with this for 2 days straight now, but i can’t get anything to work. When i connect the Amp, my ESP does not boot anymore, doesn’t matter if i connect it to 3v3 or 5v. Can’t get any logs out of it as soon as it is connected. If i only connect the Mic, it boots, but I don’t think i was able to trigger a wake word (looked at the ESP Logs aswell as the Binary Sensor “Assist in Use”. I Tried two brand new Amp Boards (These), Two Mics (These), and even tried a new ESP (Here) I also tried switching the Pins around from those used in the Video (13, 25, 26, 27) to those used in the initial post (18, 22, 23, 33), but it’s always the same. Am I doing anything wrong? Are my Amps broken?

13

Post your yaml.

mine looks like this

esphome:
  name: assistant
  friendly_name: Assistant
  on_boot:
     - priority: -100
       then:
         - wait_until: api.connected
         - delay: 1s
         - if:
             condition:
               switch.is_on: use_wake_word
             then:
               - voice_assistant.start_continuous:

esp32:
  board: esp32dev
  framework:
    type: esp-idf
    version: recommended

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: "HKyfJKTonftR4oSfmBmnJzz01HkOwX1SVz23dRzwrUM="
ota:
  password: "65c5ad2a5e6322259f99d99e297199c3"

wifi:
  ssid: !secret wifi_ssid_4
  password: !secret wifi_password_4

  manual_ip:
    static_ip: 192.168.1.240
    gateway: 192.168.1.254
    subnet: 255.255.255.0



  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Esp32-Mic-Speaker"
    password: "9vYvAFzzPjuc"

improv_serial:

i2s_audio:
  i2s_lrclk_pin: GPIO27
  i2s_bclk_pin: GPIO26

microphone:
  - platform: i2s_audio
    id: echo_microphone
    i2s_din_pin: GPIO13
    adc_type: external
    pdm: false

speaker:
  - platform: i2s_audio
    id: echo_speaker
    i2s_dout_pin: GPIO25
    dac_type: external
    mode: mono

voice_assistant:
  id: va
  microphone: echo_microphone
  speaker: echo_speaker
  noise_suppression_level: 2
  #auto_gain: 31dBFS
  volume_multiplier: 2.0
  vad_threshold: 3
  on_listening:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 0%
        brightness: 10%
        effect: pulse
  on_tts_start:
    - light.turn_on:
        id: led
        blue: 0%
        red: 0%
        green: 100%
        brightness: 100%
        effect: pulse
  on_end:
    - delay: 100ms
    - wait_until:
        not:
          speaker.is_playing:
    - script.execute: reset_led
  on_error:
    - light.turn_on:
        id: led
        blue: 0%
        red: 100%
        green: 0%
        brightness: 100%
        effect: none
    - delay: 1s
    - script.execute: reset_led
    - script.wait: reset_led
    - lambda: |-
        if (code == "wake-provider-missing" || code == "wake-engine-missing") {
          id(use_wake_word).turn_off();
        }

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO39
      inverted: true
    name: Button
    disabled_by_default: true
    entity_category: diagnostic
    id: echo_button
    on_click:
      - if:
          condition:
            switch.is_off: use_wake_word
          then:
            - if:
                condition: voice_assistant.is_running
                then:
                  - voice_assistant.stop:
                  - script.execute: reset_led
                else:
                  - voice_assistant.start:
          else:
            - voice_assistant.stop
            - delay: 1s
            - script.execute: reset_led
            - script.wait: reset_led
            - voice_assistant.start_continuous:

light:
  - platform: esp32_rmt_led_strip
    id: led
    name: None
    disabled_by_default: true
    entity_category: config
    pin: GPIO14
    default_transition_length: 0s
    chipset: ws2812
    num_leds: 1
    rgb_order: grb
    rmt_channel: 0
    effects:
      - pulse:
          transition_length: 250ms
          update_interval: 250ms

script:
  - id: reset_led
    then:
      - if:
          condition:
            - switch.is_on: use_wake_word
            - switch.is_on: use_listen_light
          then:
            - light.turn_on:
                id: led
                blue: 100%
                red: 100%
                green: 0%
                brightness: 30%
                effect: none
          else:
            - light.turn_off: led

switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(va).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
      - script.execute: reset_led
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(va).set_use_wake_word(false);
      - script.execute: reset_led
  - platform: template
    name: Use Listen Light
    id: use_listen_light
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - script.execute: reset_led
    on_turn_off:
      - script.execute: reset_led

external_components:
  - source: github://pr#5230
    components:
      - esp_adf
    refresh: 0s

esp_adf:
2 Likes
15
16

Hello everyone!

I have what I believe is a pebkac error, but I’ve been working on this off and on for weeks and finally decided to ask for help from people smarter then I.

The problem - No sound from INMP441 using ESP23-S3-devkit-C-1. I have used 3 different mics and 2 boards. I have tried several different ESPHome configs (I’ll post the two main ones) and an Arduino config meant to stream the mic (will also post). I’ve also tried different pins on the ESP, and I’ve used the debug_recording_dir but the recordings all have no sound. What I’m hoping for is some way to troubleshoot why I’m not getting any sound from this mic, I’m just past my current knowledge levels and feel a bit lost.

Thank you for reading and any advice, I rely on these forums for almost everything I do. Though usually someone already asked the question I have!

esphome:
  name: ha-microphone
  friendly_name: HA-Microphone

esp32:
  board: esp32-s3-devkitc-1
  framework:
    type: arduino

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: ""

ota:
  password: ""

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

i2s_audio:
  i2s_lrclk_pin: GPIO09 #ws
  i2s_bclk_pin: GPIO12 #clk

microphone:
- platform: i2s_audio
  i2s_din_pin: GPIO10 #sd
  id: adc_mic
  adc_type: external
  pdm: false

voice_assistant:
  microphone: adc_mic

binary_sensor:
  - platform: gpio
    name: "Voice"
    pin:
      number: GPIO0
      inverted: True
    on_press:
      - voice_assistant.start:
    on_release:
      - voice_assistant.stop:

esphome:
  name: bedroom-assist
  friendly_name: bedroom_assist

esp32:
  board: esp32-s3-devkitc-1
  framework:
    type: arduino

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: ""

ota:
  password: ""

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Bedroom-Assist Fallback Hotspot"
    password: ""

captive_portal:

##commented out because I read one in one random forum it caused problems for someone. Will try once mic works.
#light:
#  - platform: esp32_rmt_led_strip
#    rgb_order: GRB
#    pin: GPIO48
#    num_leds: 30
#    rmt_channel: 0
#    chipset: ws2812
#    name: "Bedroom Assit LED"

i2s_audio:
  id: bedroom_i2s_in
  i2s_lrclk_pin: GPIO9 #WS / LRC
  i2s_bclk_pin: GPIO12 #SCK / BCLK
  
microphone:
  - platform: i2s_audio
    id: bedroom_mic
    i2s_din_pin: GPIO10 #SD
    adc_type: external
    pdm: false
    channel: right
    bits_per_sample: 32bit
    i2s_audio_id: bedroom_i2s_in

speaker:
  - platform: i2s_audio
    id: bedroom_speaker
    dac_type: external
    i2s_dout_pin: GPIO07 #DIN
    mode: mono
    i2s_audio_id: bedroom_i2s_in

voice_assistant:
  microphone: bedroom_mic
  id: bedroom_va
  noise_suppression_level: 2
  auto_gain: 31dBFS
  volume_multiplier: 2.0
  # use_wake_word: true
  speaker: bedroom_speaker

  on_error:
    - if:
        condition:
          switch.is_on: bedroom_use_wake_word
        then:
          - switch.turn_off: bedroom_use_wake_word
          - switch.turn_on: bedroom_use_wake_word

  on_client_connected:
    - if:
        condition:
          switch.is_on: bedroom_use_wake_word
        then:
          - voice_assistant.start_continuous:

  on_client_disconnected:
    - if:
        condition:
          switch.is_on: bedroom_use_wake_word
        then:
          - voice_assistant.stop:

binary_sensor:
  - platform: status
    name: Bedroom API Connection
    id: bedroom_api_connection
    filters:
      - delayed_on: 1s
    on_press:
      - if:
          condition:
            switch.is_on: bedroom_use_wake_word
          then:
            - voice_assistant.start_continuous:
    on_release:
      - if:
          condition:
            switch.is_on: bedroom_use_wake_word
          then:
            - voice_assistant.stop:

switch:
  - platform: template
    name: Bedroom Use Wake Word
    id: bedroom_use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(bedroom_va).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(bedroom_va).set_use_wake_word(false);

/////////////////////////////////////////////////////////////////
/*
  Broadcasting Your Voice with ESP32-S3 & INMP441
  For More Information: https://youtu.be/qq2FRv0lCPw
  Created by Eric N. (ThatProject)
*/
/////////////////////////////////////////////////////////////////

/*
- Device
ESP32-S3 DevKit-C
https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/hw-reference/esp32s3/user-guide-devkitc-1.html

- Required Library
Arduino ESP32: 2.0.9

Arduino Websockets: 0.5.3
https://github.com/gilmaimon/ArduinoWebsockets
*/

#include <driver/i2s.h>
#include <WiFi.h>
#include <ArduinoWebsockets.h>

#define I2S_SD 10
#define I2S_WS 9
#define I2S_SCK 12
#define I2S_PORT I2S_NUM_0

#define bufferCnt 10
#define bufferLen 1024
int16_t sBuffer[bufferLen];

const char* ssid = "";
const char* password = "";

const char* websocket_server_host = "";
const uint16_t websocket_server_port = 8888;  // <WEBSOCKET_SERVER_PORT>

using namespace websockets;
WebsocketsClient client;
bool isWebSocketConnected;

void onEventsCallback(WebsocketsEvent event, String data) {
  if (event == WebsocketsEvent::ConnectionOpened) {
    Serial.println("Connnection Opened");
    isWebSocketConnected = true;
  } else if (event == WebsocketsEvent::ConnectionClosed) {
    Serial.println("Connnection Closed");
    isWebSocketConnected = false;
  } else if (event == WebsocketsEvent::GotPing) {
    Serial.println("Got a Ping!");
  } else if (event == WebsocketsEvent::GotPong) {
    Serial.println("Got a Pong!");
  }
}

void i2s_install() {
  // Set up I2S Processor configuration
  const i2s_config_t i2s_config = {
    .mode = i2s_mode_t(I2S_MODE_MASTER | I2S_MODE_RX),
    //.sample_rate = 44100,
    .sample_rate = 16000,
    .bits_per_sample = i2s_bits_per_sample_t(16),
    .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,
    .communication_format = i2s_comm_format_t(I2S_COMM_FORMAT_STAND_I2S),
    .intr_alloc_flags = 0,
    .dma_buf_count = bufferCnt,
    .dma_buf_len = bufferLen,
    .use_apll = false
  };

  i2s_driver_install(I2S_PORT, &i2s_config, 0, NULL);
}

void i2s_setpin() {
  // Set I2S pin configuration
  const i2s_pin_config_t pin_config = {
    .bck_io_num = I2S_SCK,
    .ws_io_num = I2S_WS,
    .data_out_num = -1,
    .data_in_num = I2S_SD
  };

  i2s_set_pin(I2S_PORT, &pin_config);
}

void setup() {
  Serial.begin(115200);

  connectWiFi();
  connectWSServer();
  xTaskCreatePinnedToCore(micTask, "micTask", 10000, NULL, 1, NULL, 1);
}

void loop() {
}

void connectWiFi() {
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
}

void connectWSServer() {
  client.onEvent(onEventsCallback);
  while (!client.connect(websocket_server_host, websocket_server_port, "/")) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("Websocket Connected!");
}


void micTask(void* parameter) {

  i2s_install();
  i2s_setpin();
  i2s_start(I2S_PORT);

  size_t bytesIn = 0;
  while (1) {
    esp_err_t result = i2s_read(I2S_PORT, &sBuffer, bufferLen, &bytesIn, portMAX_DELAY);
    if (result == ESP_OK && isWebSocketConnected) {
      client.sendBinary((const char*)sBuffer, bytesIn);
    }
  }
}```
17

do you have the LR pin on the mic connected to gnd or vcc ?

18

Here is my working config for s3-dev kit board You need to set the mic to left to work on these boards:

substitutions:
  voice_assist_idle_phase_id: '1'
  voice_assist_listening_phase_id: '2'
  voice_assist_thinking_phase_id: '3'
  voice_assist_replying_phase_id: '4'
  voice_assist_not_ready_phase_id: '10'
  voice_assist_error_phase_id: '11'
  voice_assist_muted_phase_id: '12'
esphome:
  name: master-bedroom-voice-assistant
  friendly_name: Master Bedroom Voice assistant
  platformio_options:
    board_build.flash_mode: dio


  on_boot:
      priority: 600
      then:
        - script.execute: control_led
        - delay: 30s
        - if:
            condition:
              lambda: return id(init_in_progress);
            then:
              - lambda: id(init_in_progress) = false;
              - script.execute: control_led

psram:
  mode:  octal
  speed: 80MHz
  

esp32:
  board:   esp32-s3-devkitc-1
  variant: esp32s3
  framework:
    type: esp-idf
    version: recommended
    components:
      - name:    esphome_board
        source:  github://jesserockz/esphome-esp-adf-board@main
        refresh: 0s
    sdkconfig_options:
      CONFIG_ESP32_S3_BOX_BOARD: "y"
      CONFIG_ESP32S3_DEFAULT_CPU_FREQ_240: "y"
      CONFIG_ESP32S3_DATA_CACHE_64KB:      "y"
      CONFIG_ESP32S3_DATA_CACHE_LINE_64B:  "y"
      CONFIG_AUDIO_BOARD_CUSTOM:           "y"

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: "snip"

  on_client_connected:
    - script.execute: control_led
  on_client_disconnected:
    - script.execute: control_led

ota:
  password: "snip"

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  manual_ip:
    static_ip: *
    gateway: *
    subnet: *

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Master-Bedroom-Voice-Assistant"
    password: "snip"


esp_adf:
external_components:
  - source: github://pr#5230
    components:
    - esp_adf 
    refresh: 0s

captive_portal:

light:
  - platform: esp32_rmt_led_strip
    rgb_order: GRB
    pin: GPIO18
    num_leds: 3
    rmt_channel: 0
    chipset: WS2812
    name: "Status LED"
    id: led
    default_transition_length: 0s
    effects:
      - pulse:
          name: "extra_slow_pulse"
          transition_length: 800ms
          update_interval: 800ms
          min_brightness: 0%
          max_brightness: 30%
      - pulse:
          name: "slow_pulse"
          transition_length: 250ms
          update_interval: 250ms
          min_brightness: 50%
          max_brightness: 100%
      - pulse:
          name: "fast_pulse"
          transition_length: 100ms
          update_interval: 100ms
          min_brightness: 50%
          max_brightness: 100%

i2s_audio:
  - id: i2s_in
    i2s_lrclk_pin: GPIO3  ##INMP441-WS
    i2s_bclk_pin: GPIO2  ##INMP441-SCK
  - id: i2s_out
    i2s_lrclk_pin: GPIO6  ##Max98357 - LRC
    i2s_bclk_pin: GPIO20   ###Max98357 - BCLK
    ## l/R pin on ##INMP441 is connected to ground

microphone:
  platform: i2s_audio 
  id: external_microphone 
  adc_type: external 
  i2s_audio_id: i2s_in
  i2s_din_pin: GPIO4
  channel: left
  pdm: false

speaker:
  platform: i2s_audio 
  id: external_speaker 
  dac_type: external
  i2s_audio_id: i2s_out
  i2s_dout_pin: GPIO8  ###Max98357 - DIN
  mode: mono 

voice_assistant:
  id: va
  microphone: external_microphone 
  speaker: external_speaker
  use_wake_word: true
  noise_suppression_level: 2
  auto_gain: 31dBFS
  volume_multiplier: 2.5



  on_listening:
    - lambda: id(voice_assistant_phase) = ${voice_assist_listening_phase_id};
    - script.execute: control_led

  on_stt_vad_end:
    - lambda: id(voice_assistant_phase) = ${voice_assist_thinking_phase_id};
    - script.execute: control_led

  on_tts_stream_start:
    - lambda: id(voice_assistant_phase) = ${voice_assist_replying_phase_id};
    - script.execute: control_led

  on_tts_stream_end:
    - lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
    - script.execute: control_led

  on_error: 
    - if:
        condition:
          lambda: return !id(init_in_progress);
        then:
          - lambda: id(voice_assistant_phase) = ${voice_assist_error_phase_id};
          - script.execute: control_led
          - delay: 1s
          - if:
              condition:
                switch.is_on: use_wake_word
              then:
                - lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
              else:
                - lambda: id(voice_assistant_phase) = ${voice_assist_muted_phase_id};
          - script.execute: control_led

  on_client_connected: 
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.start_continuous
          - lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
        else:
          - lambda: id(voice_assistant_phase) = ${voice_assist_muted_phase_id};
    - script.execute: control_led          

  on_client_disconnected: 
    - lambda: id(voice_assistant_phase) = ${voice_assist_not_ready_phase_id};
    - script.execute: control_led

switch:
  - platform: template
    name: Use Wake Word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    on_turn_on:
      - if:
          condition:
            lambda: return !id(init_in_progress);
          then:
            - lambda: id(voice_assistant_phase) = ${voice_assist_idle_phase_id};
            - if:
                condition:
                    not:
                      - voice_assistant.is_running
                then:
                  - voice_assistant.start_continuous
            - script.execute: control_led          
 
    on_turn_off:
      - if:
          condition:
            lambda: return !id(init_in_progress);
          then:
            - voice_assistant.stop
            - lambda: id(voice_assistant_phase) = ${voice_assist_muted_phase_id};
            - script.execute: control_led          

globals:
  - id: init_in_progress
    type: bool
    restore_value: no
    initial_value: 'true'
  - id: voice_assistant_phase
    type: int
    restore_value: no
    initial_value: ${voice_assist_not_ready_phase_id}
  
script:
  - id: control_led
    then:
      - if:
          condition:
            lambda: return !id(init_in_progress);
          then:
            - if:
                condition:
                    wifi.connected:
                then:
                  - if:
                      condition:
                          api.connected:
                      then:
                        - lambda: |
                            switch(id(voice_assistant_phase)) {
                              case ${voice_assist_listening_phase_id}:
                                id(led).turn_on().set_rgb(0, 0, 1).set_brightness(1.0).set_effect("none").perform();
                                break;
                              case ${voice_assist_thinking_phase_id}:
                                id(led).turn_on().set_rgb(0, 1, 0).set_effect("slow_pulse").perform();
                                break;
                              case ${voice_assist_replying_phase_id}:
                                id(led).turn_on().set_rgb(0, 0, 1).set_brightness(1.0).set_effect("fast_pulse").perform();
                                break;
                              case ${voice_assist_error_phase_id}:
                                id(led).turn_on().set_rgb(1, 1, 1).set_brightness(.5).set_effect("none").perform();
                                break;
                              case ${voice_assist_muted_phase_id}:
                                id(led).turn_off().perform();
                                break;
                              case ${voice_assist_not_ready_phase_id}:
                                id(led).turn_on().perform();
                                break;
                              default:
                                id(led).turn_on().set_rgb(1, 0, 0).set_brightness(0.2).set_effect("none").perform();
                                break;
                            }
                      else:
                        - light.turn_off:
                            id: led
                else:
                  - light.turn_off:
                      id: led
          else:
            - light.turn_on:
                id: led
                blue: 50%
                red: 50%
                green: 50%
                effect: "fast_pulse"

Video of them working:

19

robgough1970

I have it connected to ground.

Rich37804

You found my pebkac error! I knew it had to use left, I thought I had it on left, but I clearly had it on right. Changed it to left and it is working! I have tweaking to do to get the light working the way I want (your example config is great!) and my speaker currently doesn’t sound very good. Once I have everything working up to par I’ll post my final config to provide one more working example. I don’t have as much time as I’d like so it might take a bit.

Thank you all again!

20

Anyone ever got this to work on a ESP32-C3 board?

Yaml 
esphome:
  name: test
  friendly_name: TEST
  platformio_options:
    board_build.flash_mode: dio
    board_build.f_cpu: 80000000L
  on_boot:
     - priority: -100
       then:
         - wait_until: api.connected
         - delay: 1s
         - if:
             condition:
               switch.is_on: use_wake_word
             then:
               - voice_assistant.start_continuous:

esp32:
  board: esp32-c3-devkitm-1
  variant: esp32c3
  framework:
    type: esp-idf


# Enable logging cuz board has UART
logger:
  baud_rate: 115200
  hardware_uart: UART0
  level: debug


#Pin25 GND
#Pin26 VCC
#Pin27 aka GPIO05 SD Mic
#Pin23 GPIO07 SCK
#Pin22 GPIO06 WS
 
debug:

# Enable Home Assistant API
api:
  encryption:
    key: "3gY/="

ota:
  password: ""

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  manual_ip:
    static_ip: 192.168.1.192
    gateway: 192.168.1.1
    subnet: 255.255.255.0
  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Test Fallback Hotspot"
    password: "*"

web_server:
  port: 80

captive_portal:

i2s_audio:
    i2s_lrclk_pin: GPIO6  ##INMP441-WS
    i2s_bclk_pin: GPIO7  ##INMP441-SCK

microphone:
  platform: i2s_audio 
  id: external_microphone 
  adc_type: external 
  i2s_din_pin: GPIO5
  channel: left
  pdm: false

voice_assistant:
  microphone: external_microphone
  use_wake_word: false
  noise_suppression_level: 2
  auto_gain: 31dBFS
  volume_multiplier: 2.0
  id: assist
  on_tts_end:
    - homeassistant.service:
        service: media_player.play_media
        data:
          media_content_id: !lambda 'return x;'
          media_content_type: audio/mpeg
          entity_id: media_player.windows

switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(assist).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(assist).set_use_wake_word(false);
Log on the Device

INFO ESPHome 2024.2.1
INFO Reading configuration /config/esphome/test.yaml…
INFO Starting log output from 192.168.1.192 using esphome API
INFO Successfully connected to test @ 192.168.1.192 in 0.109s
INFO Successful handshake with test @ 192.168.1.192 in 0.087s
[16:24:10][I][app:102]: ESPHome version 2024.2.1 compiled on Mar 4 2024, 16:20:37
[16:24:10][C][wifi:577]: WiFi:
[16:24:10][C][wifi:409]: Local MAC: 54:32:04:47:D3:E4
[16:24:10][C][wifi:414]: SSID: [redacted]
[16:24:10][C][wifi:415]: IP Address: 192.168.1.192
[16:24:10][C][wifi:417]: BSSID: [redacted]
[16:24:10][C][wifi:418]: Hostname: ‘test’
[16:24:10][C][wifi:420]: Signal strength: -50 dB ▂▄▆█
[16:24:10][C][wifi:424]: Channel: 12
[16:24:10][C][wifi:425]: Subnet: 255.255.255.0
[16:24:10][C][wifi:426]: Gateway: 192.168.1.1
[16:24:10][C][wifi:427]: DNS1: 0.0.0.0
[16:24:10][C][wifi:428]: DNS2: 0.0.0.0
[16:24:10][C][logger:447]: Logger:
[16:24:10][C][logger:448]: Level: DEBUG
[16:24:10][C][logger:449]: Log Baud Rate: 115200
[16:24:10][C][logger:451]: Hardware UART: UART0
[16:24:10][C][template.switch:068]: Template Switch ‘Use wake word’
[16:24:10][C][template.switch:091]: Restore Mode: restore defaults to ON
[16:24:10][C][template.switch:057]: Optimistic: YES
[16:24:10][C][captive_portal:088]: Captive Portal:
[16:24:10][C][web_server:168]: Web Server:
[16:24:10][C][web_server:169]: Address: 192.168.1.192:80
[16:24:10][C][mdns:115]: mDNS:
[16:24:10][C][mdns:116]: Hostname: test
[16:24:10][C][ota:096]: Over-The-Air Updates:
[16:24:10][C][ota:097]: Address: 192.168.1.192:3232
[16:24:10][C][ota:100]: Using Password.
[16:24:10][C][ota:103]: OTA version: 2.
[16:24:10][W][ota:107]: Last Boot was an unhandled reset, will proceed to safe mode in 8 restarts
[16:24:10][C][api:139]: API Server:
[16:24:10][C][api:140]: Address: 192.168.1.192:6053
[16:24:10][C][api:142]: Using noise encryption: YES
[16:24:10][C][debug:064]: Debug component:
[16:24:10][D][debug:076]: ESPHome version 2024.2.1
[16:24:10][D][debug:080]: Free Heap Size: 161116 bytes
[16:24:10][D][debug:159]: Chip: Model=ESP32-C3, Features=WIFI_BGN,BLE, Cores=1, Revision=4
[16:24:10][D][debug:167]: ESP-IDF Version: 4.4.6
[16:24:10][D][debug:172]: EFuse MAC: 54:32:04:47:D3:E4
[16:24:10][D][debug:272]: Reset Reason: Power On Reset
[16:24:10][D][debug:317]: Wakeup Reason: Unknown
[16:24:17][D][switch:012]: ‘Use wake word’ Turning ON.
[16:24:17][D][switch:055]: ‘Use wake word’: Sending state ON
[16:24:17][D][voice_assistant:414]: State changed from IDLE to START_PIPELINE
[16:24:17][D][voice_assistant:420]: Desired state set to START_MICROPHONE
[16:24:17][D][voice_assistant:118]: microphone not running
[16:24:17][D][voice_assistant:202]: Requesting start…
[16:24:17][D][voice_assistant:414]: State changed from START_PIPELINE to STARTING_PIPELINE
[16:24:17][D][voice_assistant:435]: Client started, streaming microphone
[16:24:17][D][voice_assistant:414]: State changed from STARTING_PIPELINE to START_MICROPHONE
[16:24:17][D][voice_assistant:420]: Desired state set to STREAMING_MICROPHONE
[16:24:17][D][voice_assistant:155]: Starting Microphone
[16:24:17][D][voice_assistant:414]: State changed from START_MICROPHONE to STARTING_MICROPHONE
[16:24:17][D][esp-idf:000]: I (28922) I2S: DMA Malloc info, datalen=blocksize=1024, dma_buf_count=4

[16:24:17][D][voice_assistant:521]: Event Type: 1
[16:24:17][D][voice_assistant:524]: Assist Pipeline running
[16:24:17][D][voice_assistant:414]: State changed from STARTING_MICROPHONE to STREAMING_MICROPHONE
[16:24:17][D][voice_assistant:521]: Event Type: 9
[16:24:23][D][switch:016]: ‘Use wake word’ Turning OFF.
[16:24:23][D][switch:055]: ‘Use wake word’: Sending state OFF
[16:24:23][D][voice_assistant:514]: Signaling stop…
[16:24:23][D][voice_assistant:414]: State changed from STREAMING_MICROPHONE to STOP_MICROPHONE
[16:24:23][D][voice_assistant:420]: Desired state set to IDLE
[16:24:23][D][voice_assistant:521]: Event Type: 0
[16:24:23][E][voice_assistant:651]: Error: no_wake_word - No wake word detected
[16:24:23][D][voice_assistant:514]: Signaling stop…
[16:24:23][D][voice_assistant:414]: State changed from STOP_MICROPHONE to STOP_MICROPHONE
[16:24:23][D][voice_assistant:420]: Desired state set to IDLE
[16:24:23][D][voice_assistant:414]: State changed from STOP_MICROPHONE to STOPPING_MICROPHONE
[16:24:23][D][esp-idf:000]: I (34931) I2S: DMA queue destroyed

[16:24:23][D][voice_assistant:521]: Event Type: 2
[16:24:23][D][voice_assistant:611]: Assist Pipeline ended
[16:24:23][D][voice_assistant:414]: State changed from STOPPING_MICROPHONE to IDLE

I have activate the logging for Assistant in configuration.yaml

assist_pipeline:
  debug_recording_dir: /config/stt/assist_pipeline

But the file always ends up 0.0KiB
image

When i use the Browser/app and use voice assistant there this does not happen!
So i assume the mic just doesn’t work.

I tried different things, i even have a Lolin D32 Board here which has a normal ESP32 chip. didn’t get that to work either.

I see people using esp_adf from pull #5230 but that’s S3 only right?

I don’t have speakers as i want to use my media player for that, so i only need the microphone.

L/R is linked to Ground.

BTW the mic itself is working, as tested with arduino IDE.

21

this can be caused by many things, if the mic is working in arduino, then everything hardware related will be working, and your yaml is correct for the mic i2s setup so no problem there.
first thing to check is that HA can stream to the device and the device can stream to HA. Check the HA Core log to see if there are any indications in there as to what the issue is Settings>>system>>logs . If the esp and HA are on different VLAN’s then have them both on the same VLAN. In order to stream audio HA needs to have all UDP Ports open, so check that any firewall is not blocking access through ALL udp ports. If running HA in docker, HA has to have it’s own IP address and network needs to be configured to host. Check those things out and let us know how you get on.