Matrix Voice and Home Assistant voice (assist)

Configuration
1

Way back when Snips.ai was still open I bought the 7 microphone array Matrix Voice, just in time to NOT get it working before SONOS killed Snips. Rhasspy never quite worked well (or, I couldn’t get it working well enough before my girlfriend promised to kill me if I said “Porcupine” one more time - haven’t touched it since).

But it’s a different era, and with the year of the voice my hope has returned (thanks Home Assistant Devs!)

Does anyone know how to get the Matrix Voice to be the microphone for the new Assist? I haven’t seen any mention of Matrix Voice and Home Assistant since about a year ago, before the year of the voice.

1 Like
2

Just wait until tomorrows release. You’ll get more information. Or join the beta to read up on the changes now.

also read this blog post

3

Tomorrow? That’s like, years from now! How could anyone wait that long!?

Well, thanks, I’ll keep my eyes peeled.

@modernhistorian, maybe there will be some noteworthy news for your unanswered question as well.

4

Have you had any progress with this? I’d also like to use my Matrix Voice, now that wake words are supported!

5

Unfortunately, there was nothing of interest (to this) in the release he mentioned. I haven’t spent more time on this, and doubt that I will in the near future.

But like you, I’m certainly interested if someone figures this out. And I’ll be sure to share if I do.

6

I believe there are 2 ways to try to achieve this:

Try and use the ESP32 on the Matrix Voice (if you have it). There is a project for Rhasspy (not HA Voice Assist) which made this work:

But I believe this is not compatible with HA Voice Assist.

Maybe in the future, there will be a ESPhome firmware/yaml configuration for the Matrix Voice ESP32, but so far not. I guess due to the Matrix Voice using the FPGA it may need quite some additional code for esphome?

The other option, and maybe most promising, would be to try and use Matrix Voice with Rasperry Pi and this:

Now unfortunately all the Matrix Voice Raspbian only works with Buster (not the current Bullseye) so I will try to figure this out when I find some time and post a details tutorial if successful!

1 Like
7

I would love to use the Matrix Voice (ESP32) as a satellite for homeassistant voice, but seems like it is currently not possible.

I have tried to look into the development process for the FPGA of the Matrix Voice, but its simply not my domain and seemed pretty involved, so I gave up on it. I think technically it should be possible to only rely on the ESP32 to control the microphones and LEDs, skipping the FPGA entirely. Although this would obviously not be ideal (since the FPGA could do some very neat things like noise suppression) I think it would be a great start.

8

I wrote an integration for ESPHome which lets you use Matrix Voice with Home Assistant voice control. You can find it here: GitHub - gnumpi/esphome_matrixio: ESPHome Component for MatrixIO Devices

3 Likes
9

Amazing start! I’ll also integrate the Matrix voice in HA. By the way my understanding was that beamforming is implemented in the FPGA and they wrote the firmware already.

10

Mischa:

I tried setting this from the github project. I was able to build and flash the code to my Matrix Voice, and integrate it as a Voice Assistant in HA. But beyond that I cannot get anything else to work.

I see the ‘Start Listening’ is set to off, I tried to turn it on and speak, but the Microphone does not seem to register anything (at least from the logs) and there is no output from the speakers. For some reason, the ‘Start Listening’ turns itself off after a few seconds.

Here are the logs from the device once it boots up:

[11:25:17][I][app:102]: ESPHome version 2023.11.6 compiled on Sep 10 2024, 00:09:56
[11:25:17][C][wifi:559]: WiFi:
[11:25:17][C][wifi:391]:   Local MAC: ****
[11:25:17][C][wifi:396]:   SSID: '*****'
[11:25:17][C][wifi:397]:   IP Address: *.*.*.*
[11:25:17][C][wifi:399]:   BSSID: ****
[11:25:18][C][wifi:400]:   Hostname: 'matrixio-voice'
[11:25:18][C][wifi:402]:   Signal strength: -51 dB ▂▄▆█
[11:25:18][C][wifi:406]:   Channel: 11
[11:25:18][C][wifi:407]:   Subnet: 255.255.255.0
[11:25:18][C][wifi:408]:   Gateway: *.*.*.*
[11:25:18][C][wifi:409]:   DNS1: *.*.*.*
[11:25:18][C][wifi:410]:   DNS2: 0.0.0.0
[11:25:18][C][logger:416]: Logger:
[11:25:18][C][logger:417]:   Level: DEBUG
[11:25:18][C][logger:418]:   Log Baud Rate: 115200
[11:25:18][C][logger:420]:   Hardware UART: UART0
[11:25:18][C][spi:067]: SPI bus:
[11:25:18][C][spi:068]:   CLK Pin: GPIO32
[11:25:18][C][spi:069]:   SDI Pin: GPIO21
[11:25:18][C][spi:070]:   SDO Pin: GPIO33
[11:25:18][C][spi:072]:   Using HW SPI: SPI2_HOST
[11:25:18][C][matrixio_everloop:031]: Matrixio Everloop:
[11:25:18][C][matrixio_everloop:032]:   LEDs: 18
[11:25:18][C][light:103]: Light 'everloop'
[11:25:18][C][light:105]:   Default Transition Length: 1.0s
[11:25:18][C][light:106]:   Gamma Correct: 2.80
[11:25:18][C][template.switch:068]: Template Switch 'Use wake word'
[11:25:18][C][template.switch:091]:   Restore Mode: restore defaults to ON
[11:25:18][C][template.switch:057]:   Optimistic: YES
[11:25:18][C][template.switch:068]: Template Switch 'Use Listen Light'
[11:25:18][C][template.switch:091]:   Restore Mode: restore defaults to ON
[11:25:18][C][template.switch:057]:   Optimistic: YES
[11:25:18][C][template.switch:068]: Template Switch 'Start Listening'
[11:25:18][C][template.switch:091]:   Restore Mode: always OFF
[11:25:18][C][template.switch:057]:   Optimistic: NO
[11:25:18][C][matrixio:026]: Matrixio:
[11:25:18][C][matrixio:027]:   Data rate: 8MHz
[11:25:18][C][matrixio:028]:   CS pin: GPIO23
[11:25:18][C][matrixio:030]:   Found Matrixio Voice with firmware: 65547
[11:25:18][C][matrixio_microphone:041]: Matrixio Microphones:
[11:25:18][C][matrixio_speaker:020]: Matrixio Speaker:
[11:25:18][C][matrixio_speaker:023]:   Sampling frequency: 16000Hz
[11:25:18][C][matrixio_speaker:025]:   Output: Headphone
[11:25:18][C][matrixio_speaker:034]:   Volume: 92%
[11:25:18][C][mdns:115]: mDNS:
[11:25:18][C][mdns:116]:   Hostname: matrixio-voice
[11:25:18][C][ota:097]: Over-The-Air Updates:
[11:25:18][C][ota:098]:   Address: matrixio-voice.local:3232
[11:25:18][C][api:139]: API Server:
[11:25:18][C][api:140]:   Address: matrixio-voice.local:6053
[11:25:18][C][api:144]:   Using noise encryption: NO

Do you have any thoughts on what I could be doing wrong here?

11

I should have mentioned this - the matrix_voice.yaml is unchanged from the repository version other than changes to the wifi connection parameters:

external_components:
  #- source:
  #    type: git
  #    url: https://github.com/gnumpi/esphome_matrixio
  #    ref: main
  - source:
      type: local
      path: esphome/components/

esphome:
  name: matrixio-voice

esp32:
  board: esp32dev
  framework:
    type: esp-idf
    version: recommended

wifi:
  ssid: ****
  password: ****
  fast_connect: true

logger:

ota:

api:

spi:
  clk_pin:  GPIO32
  mosi_pin: GPIO33
  miso_pin: GPIO21

matrixio:
  id: matrixio_dev
  cs_pin: GPIO23

light:
  - platform: matrixio
    name: everloop
    id: everloop

microphone:
  - platform: matrixio
    id: matrixio_mic

speaker:
  - platform: matrixio
    id: matrixio_speaker_hp
    # [headphone,speakers], default: headphone
    audio_out: headphone
    # 0..100, in percent, default: 80
    volume: 90

voice_assistant:
  id: va
  microphone: matrixio_mic
  speaker: matrixio_speaker_hp

  on_listening:
    - light.turn_on:
        id: everloop
        blue: 100%
        red: 0%
        green: 0%
        white: 0%
        brightness: 25%
  on_stt_vad_end:
    - light.turn_on:
        id: everloop
        blue: 50%
        red: 0%
        green: 100%
        white: 0%
        brightness: 25%
  on_tts_start:
    - light.turn_on:
        id: everloop
        blue: 100%
        red: 100%
        green: 0%
        white: 0%
        brightness: 25%
  on_end:
    - delay: 100ms
    - wait_until:
        not:
          speaker.is_playing:
    - script.execute: reset_led
  on_error:
    - light.turn_on:
        id: everloop
        red: 100%
        green: 0%
        blue: 0%
        white: 0%
        brightness: 25%
    - delay: 1s
    - script.execute: reset_led
  on_client_connected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.start_continuous:
          - script.execute: reset_led
  on_client_disconnected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.stop:
          - light.turn_off: everloop

script:
  - id: reset_led
    then:
      - if:
          condition:
            - switch.is_on: use_wake_word
            - switch.is_on: use_listen_light
          then:
            - light.turn_on:
                id: everloop
                red: 100%
                green: 89%
                blue: 71%
                white: 0%
                brightness: 25%
          else:
            - light.turn_off: everloop

switch:
  - platform: template
    name: 'Use wake word'
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(va).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
      - script.execute: reset_led
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(va).set_use_wake_word(false);
      - script.execute: reset_led
  - platform: template
    name: 'Use Listen Light'
    id: use_listen_light
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - script.execute: reset_led
    on_turn_off:
      - script.execute: reset_led

  - platform: template
    name: "Start Listening"
    turn_on_action:
      - voice_assistant.start:
          silence_detection: false

    turn_off_action:
      - voice_assistant.stop:
      - script.execute: reset_led