Esp32 voice assistant issues

been running this config on 2 boards just added a 3rd

esphome:
  name: voice-assistant-tardis
  friendly_name: Voice Assistant TARDIS

esp32:
  board: esp32dev
  framework:
    type: arduino

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: "*******************************************"

ota:
  password: "*************************************"

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  manual_ip:
   static_ip: 192.168.1.131
   gateway: 192.168.1.1
   subnet: 255.255.255.0 
  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Voice-Assistant-Tardis"
    password: "C8EfVtX6IiVQ"

captive_portal:

i2s_audio:
  - id: i2s_in
    i2s_lrclk_pin: GPIO26 #WS 
    i2s_bclk_pin: GPIO25 #SCK

microphone:
  - platform: i2s_audio
    adc_type: external
    pdm: false
    id: mic_i2s
    channel: right
    bits_per_sample: 32bit
    i2s_audio_id: i2s_in
    i2s_din_pin: GPIO33  #SD Pin from the INMP441 Microphone


media_player:
  - platform: i2s_audio
    name: "esp_speaker"
    id: media_player_speaker
    i2s_audio_id: i2s_in
    dac_type: external
    i2s_dout_pin: GPIO27   #  DIN Pin of the MAX98357A Audio Amplifier
    mode: mono


voice_assistant:
  microphone: mic_i2s
  id: va
  noise_suppression_level: 2
  auto_gain: 31dBFS
  volume_multiplier: 4.0
  use_wake_word: false
  media_player: media_player_speaker

  on_wake_word_detected: 
    - light.turn_on:
        id: led_light
  on_listening: 
    - light.turn_on:
        id: led_light
        effect: "Slow Pulse"
        red: 0%
        green: 0%
        blue: 100%
        brightness: 100%
  on_stt_end:
    - light.turn_on:
        id: led_light
        effect: "None"
        red: 0%
        green: 100%
        blue: 0%
        brightness: 100%

  on_error: 
    - light.turn_on:
        id: led_light
        effect: "None"
        red: 100%
        green: 0%
        blue: 0%
    - if:
        condition:
          switch.is_on: use_wake_word
        then:

          - switch.turn_off: use_wake_word
          - delay: 1sec 
          - switch.turn_on: use_wake_word      
  


  on_client_connected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.start_continuous:

  on_client_disconnected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.stop:
 
  on_end:
    - light.turn_off:
        id: led_light


binary_sensor:
  - platform: status
    name: API Connection
    id: api_connection
    filters:
      - delayed_on: 1s
    on_press:
      - if:
          condition:
            switch.is_on: use_wake_word
          then:
            - voice_assistant.start_continuous:
    on_release:
      - if:
          condition:
            switch.is_on: use_wake_word
          then:
            - voice_assistant.stop:


switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(va).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
    
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(va).set_use_wake_word(false);

light:
  - platform: neopixelbus
    id: led_light
    type: grb
    pin: GPIO32      # DIN pin of the LED Strip
    num_leds: 6      # change the Number of LEDS according to your LED Strip.
    name: "Light"
    variant: ws2812x
    default_transition_length: 0.5s
      
    effects:
      - pulse:
          name: "Slow Pulse"
          transition_length: 500ms
          update_interval: 500ms
          min_brightness: 50%
          max_brightness: 100%
      - pulse:
          name: "Fast Pulse"
          transition_length: 100ms
          update_interval: 100ms
          min_brightness: 50%
          max_brightness: 100%

now getting this error

[15:43:43][W][component:214]: Component api took a long time for an operation (0.05 s).
[15:43:43][W][component:215]: Components should block for at most 20-30ms.
[15:43:43][D][api.connection:1089]: Home Assistant 2023.12.3 (192.168.1.162): Connected successfully
[15:43:43][E][voice_assistant:375]: Multiple API Clients attempting to connect to Voice Assistant
[15:43:43][E][voice_assistant:376]: Current client: Home Assistant 2023.12.3 (192.168.1.162)
[15:43:43][E][voice_assistant:377]: New client: Home Assistant 2023.12.3 (192.168.1.162)
[15:44:12][W][api.connection:102]: Home Assistant 2023.12.3 (192.168.1.162): Connection reset
[15:44:12][D][voice_assistant:422]: State changed from STREAMING_MICROPHONE to STOP_MICROPHONE
[15:44:12][D][voice_assistant:428]: Desired state set to IDLE
[15:44:12][D][voice_assistant:422]: State changed from STOP_MICROPHONE to STOPPING_MICROPHONE
[15:44:12][D][voice_assistant:422]: State changed from STOPPING_MICROPHONE to IDLE

after this the devices stop working unless I turn on/off the things

You and everyone else. Discord is littered with these same issues from dozens (hundreds?) of frustrated users. It’s simply not ready for prime time. Just think of it as a nice proof of concept. When it works… The YouTube videos are all mostly deceiving and capture the devices when they actually work, which, most of the time, do not.

it seems to have resolved its self and works well now