ESPHome voice_assistant not functioning

Hello everyone,

Unfortunately I have a problem with ESPHome and the voice_assistant component. I can’t see any signs that it is active or working.

Background: I have set up a voice pipeline and would expect to be able to interact with it using the voice_assistant component.

Hardware: INMP441, ESP32-S3

Config:

esphome:
  name: jsensor-2
  friendly_name: JSensor 2

  on_boot:
    - priority: -100
      then:
        - voice_assistant.start_continuous:

esp32:
  board: esp32-s3-devkitc-1
  framework:
    type: arduino

# Enable logging
logger:
  baud_rate: 0

# Enable Home Assistant API
api:
  encryption:
    key: "XXX"

ota:
  password: "XXX"

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Fallback Hotspot"
    password: "XXX"

captive_portal:

i2s_audio:
  i2s_lrclk_pin: GPIO12
  i2s_bclk_pin: GPIO13

microphone:
  - platform: i2s_audio
    id: mic
    adc_type: external
    i2s_din_pin: GPIO10
    pdm: False
    on_data:
      - logger.log:
          format: "Received %d bytes"
          args: ['x.size()']


voice_assistant:
  microphone: mic
  use_wake_word: true
  id: assist

Console Output:

[16:28:39][I][app:102]: ESPHome version 2024.2.1 compiled on Feb 28 2024, 16:21:29
[16:28:39][C][wifi:577]: WiFi:
[16:28:39][C][wifi:409]:   Local MAC: C0:4E:30:17:29:58
[16:28:39][C][wifi:414]:   SSID: [redacted]
[16:28:39][C][wifi:415]:   IP Address: 10.4.21.225
[16:28:39][C][wifi:417]:   BSSID: [redacted]
[16:28:39][C][wifi:418]:   Hostname: 'jsensor-2'
[16:28:39][C][wifi:420]:   Signal strength: -55 dB ▂▄▆█
[16:28:39][C][wifi:424]:   Channel: 6
[16:28:39][C][wifi:425]:   Subnet: 255.255.255.0
[16:28:39][C][wifi:426]:   Gateway: 10.4.21.1
[16:28:39][C][wifi:427]:   DNS1: 10.4.21.1
[16:28:39][C][wifi:428]:   DNS2: 0.0.0.0
[16:28:39][C][logger:447]: Logger:
[16:28:39][C][logger:448]:   Level: DEBUG
[16:28:39][C][logger:449]:   Log Baud Rate: 0
[16:28:39][C][logger:451]:   Hardware UART: USB_CDC
[16:28:39][C][captive_portal:088]: Captive Portal:
[16:28:39][C][mdns:115]: mDNS:
[16:28:39][C][mdns:116]:   Hostname: jsensor-2
[16:28:39][C][ota:096]: Over-The-Air Updates:
[16:28:39][C][ota:097]:   Address: jsensor-2.local:3232
[16:28:39][C][ota:100]:   Using Password.
[16:28:39][C][ota:103]:   OTA version: 2.
[16:28:39][C][api:139]: API Server:
[16:28:39][C][api:140]:   Address: jsensor-2.local:6053
[16:28:39][C][api:142]:   Using noise encryption: YES

nothing more and nothing related to voice_assistant

The microphone has already been tested using this script Arduino-DumbDisplay/projects/esp32ddmic/esp32ddmic.ino at 2911832e1661e3e6b53c705a136a8e9b8b0e1da6 · trevorwslee/Arduino-DumbDisplay · GitHub

Within the debug-assistant is also no entry

Does anyone have any ideas as to why this could be, what the error is or how I could debug further?

This is the code I use.

improv_serial:

i2s_audio:
  i2s_lrclk_pin: GPIO27
  i2s_bclk_pin: GPIO26

microphone:
  - platform: i2s_audio
    id: echo_microphone
    i2s_din_pin: GPIO13
    adc_type: external
    pdm: false

speaker:
  - platform: i2s_audio
    id: echo_speaker
    i2s_dout_pin: GPIO25
    dac_type: external
    mode: mono

voice_assistant:
  id: va
  microphone: echo_microphone
  speaker: echo_speaker
  noise_suppression_level: 2
  #auto_gain: 31dBFS
  volume_multiplier: 2.0
  vad_threshold: 3
  on_listening:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 0%
        brightness: 10%
        effect: pulse
  on_tts_start:
    - light.turn_on:
        id: led
        blue: 0%
        red: 0%
        green: 100%
        brightness: 100%
        effect: pulse
  on_end:
    - delay: 100ms
    - wait_until:
        not:
          speaker.is_playing:
    - script.execute: reset_led
  on_error:
    - light.turn_on:
        id: led
        blue: 0%
        red: 100%
        green: 0%
        brightness: 100%
        effect: none
    - delay: 1s
    - script.execute: reset_led
    - script.wait: reset_led
    - lambda: |-
        if (code == "wake-provider-missing" || code == "wake-engine-missing") {
          id(use_wake_word).turn_off();
        }

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO39
      inverted: true
    name: Button
    disabled_by_default: true
    entity_category: diagnostic
    id: echo_button
    on_click:
      - if:
          condition:
            switch.is_off: use_wake_word
          then:
            - if:
                condition: voice_assistant.is_running
                then:
                  - voice_assistant.stop:
                  - script.execute: reset_led
                else:
                  - voice_assistant.start:
          else:
            - voice_assistant.stop
            - delay: 1s
            - script.execute: reset_led
            - script.wait: reset_led
            - voice_assistant.start_continuous:

light:
  - platform: esp32_rmt_led_strip
    id: led
    name: None
    disabled_by_default: true
    entity_category: config
    pin: GPIO14
    default_transition_length: 0s
    chipset: ws2812
    num_leds: 1
    rgb_order: grb
    rmt_channel: 0
    effects:
      - pulse:
          transition_length: 250ms
          update_interval: 250ms

script:
  - id: reset_led
    then:
      - if:
          condition:
            - switch.is_on: use_wake_word
            - switch.is_on: use_listen_light
          then:
            - light.turn_on:
                id: led
                blue: 100%
                red: 100%
                green: 0%
                brightness: 30%
                effect: none
          else:
            - light.turn_off: led

switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(va).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
      - script.execute: reset_led
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(va).set_use_wake_word(false);
      - script.execute: reset_led
  - platform: template
    name: Use Listen Light
    id: use_listen_light
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - script.execute: reset_led
    on_turn_off:
      - script.execute: reset_led

external_components:
  - source: github://pr#5230
    components:
      - esp_adf
    refresh: 0s

esp_adf:

As you can see it is for more involved than your code, but it is mostly driving LEDs and speakers.

I am no expert but i do believe you need this as your framework rather than arduino

  framework:
    type: esp-idf
    version: recommended

Just tried with this in the config:

esp32:
  board: esp32-s3-devkitc-1
  framework:
    type: esp-idf
    version: recommended

unfortunately it does not work and still no output in the logs

if you have the l/r pin on your mic to gnd then you need to define channel: left in your mic config.
remove your on_boot automation and add the following to your api:

 api:
  on_client_connected:
        then:
          - delay: 1s
          - voice_assistant.start_continuous:
          - delay: 1s
          - lambda: id(va).set_use_wake_word(true);
          - delay: 1s