"ReSpeaker Lite" - new Seeed Studio Voice Assistant Development Kit hardware combine ESP32 with XMOS XU316 DSP chip for advanced audio processing as a ESPHome-based Home Assistant Assist Satellite voice devkit

ESPHome
, ,
8

The pic of the xmos chip on the respeaker board deff is not a 60 pin package. So that leaves the 265 FBGA or 128 TQFP package chips, since it has pins, it is not the FBGA packages. So, from your list, by means of elimination, it should be one of the bottom 2 - 128 TQFP package chips.

If so, there is no external memory support :(. The only difference between the 2 is the quality of components used (industrial vs. commercial).

I may have misunderstood though, it sounds like you are wondering about what chip the hass hardware will have in it?

2 Likes
9

What picture are you looking at? I’m looking at picture on their wiki, an it looks sharper than the others:

image
image1400×1050 82.9 KB

image
image1400×1475 145 KB

As far as I can tell picture show a 60-pin package, or at least I counted 15 pins on each visable side.

As far as I can tell the chip in the picture look to read this:

XMOS
V16A0
G12342P2
TF1148.00

And if do a search for “V16A0 AND XMOS” I only find the datasheet for “XU316-1024-QF60A” SKU in a 60pin package, and from a cost-effectiveness perspective I guess it makes more sense to use “XU316-1024-QF60A-C24” (offering 2400 MIPS) over the faster “XU316-1024-QF60A-C32” (offering 3200 MIPS) even though from developers and end-user perspective we probably want the faster variant:

And I believe that would also make sense from a hardware developer point-of-view to use either XU316-1024-QF60A-C24 (or XU316-1024-QF60A-C32) since XU316-1024-QF60A-C24 is what is used by XMOS’s “XK-VOICE-L71 Voice Reference Design Evaluation Kit” so it is very well documented and tested:

https://www.xmos.com/file/xk-voice-l71-pcb-design-files/?version=latest

1 Like
10

You are correct, I was looking at the picture above on mobile, which is quite blurry.

Edit: check this out, just saw it on YouTube.

1 Like
11

Wow! So apparently FutureProofHomes has also announced his ”Satellite1 PCB Dev Kit” also sound to use same or similar XMOS chip in combination with an ESP32-S3 module but he has designed as a two-board PCBs voice satellite hardware development kit:

GitHub repository:

Website:

gps_generated
gps_generated1024×1024 149 KB

image
image3689×2267 371 KB

image
image1813×1241 292 KB

image
image1813×1241 316 KB

Satellite1 PCB Dev Kit

The Satellite1 PCB Dev Kit contains the two PCBs necessary to build your own completely private voice assistant & multi-sensor with XMOS advanced audio processing & music playback. Add your own speaker and power supplies.

Satellite1 HAT Board:

This board features 4 PDM microphones, 12 NeoPixel LEDs, humidity/temp/lux sensors, 4 buttons (volume up/down, action button & hardware mute), plus the XMOS audio processing chip and a power DAC with for amplified speaker-out connection or 3.5mm headphone connection. All remaining GPIOs are also exposed.

The Satellite1 Hat connects easily to the Sat1 Core Board but can also be paired with a Raspberry Pi or a PC/Mac via USB! Perfect for all your voice assistant and audio projects!

Satellite1 Core Board:

The Satellite1 Core Board contains the ESP32-S3 n16r8, USB-C Power Delivery and 40-pin connection. This board attaches to the companion Sat1 HAT Board.

Looks like he also posted a future roadmap showing that he working on a a nice enclosure (as well as the mentioning of an optional recessed enclosure for in-cealing / in-wall mounting of this smart speaker):

image
image991×411 67.5 KB

image
image799×623 45.9 KB

And yeah, I noticed now that FutureProofHomes had posted a preview video on YouTube showing of an early prototype version devkit of that 4-months ago when he at the time called that project “HomeX” (but at that time he had based the prototype on the wyoming-satellite platform running on a Raspberry Pi instead of using Nabu Casa’s upcoming ESPHome-based voice-kit hardware platform that runs on ESP32-S3 and using an XMOS xCORE chip for audio processing):

PS: The new design reminds me of the “Onju Voice” PCB replacement for the Google Nest Mini (2nd gen), which is a open-source hardware project that I hope someone else will pick up and update now:

12

@FutureProofHomes Can you tell us which exact SKU of XMOS chip your product will be using?

And also wondering if your PBC(s) will be open-source hardware and/or use OSH/OSHW design?

1 Like
13

Here’s the actual XMOS chip we’re using:

I just updated the repo to clarify a bit our open-source strategy. In a nutshell, upon launch all the firmware (ESP & XMOS) will be open source and all our hardware schematics will be published too. The KiCad proj. files will follow a delayed open source model (I’ll publish those dates for us), at which point we will then put out the proj. files too. Open to folks thoughts on this! And again, if you want to work closely with the core-team then please do ping me!

3 Likes
14

@FutureProofHomes Any input on the media playback capabilities for music playback and audio quality via these type of products if using with better speakers?

I read that Nabu Casa’s will have an audio output jack (3.5mm headphone jack) for connecting external speakars.

For reference, I currently have ALL the various Google Nest (and Googgle Home) speakers in different rooms and using them for multi-room music playback.

Nice! Hope that you get more hardware developers onboard!

Cool! So you are using the little faster “XU316-1024-QF60B-C32” (3200 MIPS) SKU and not the “XU316-1024-QF60B-C24” (2400 MIPS) that both the ReSpeaker Lite and XK-VOICE-L71 Voice Reference Design Evaluation Kit are using → Processor Catalogue | XMOS

That sounds awesome! Pun intended :wink:

15

I’ll track the thread over here if that’s okay. :slight_smile:

3 Likes
16

Open source!

Hell yeah.

1 Like
17

FYI @alextrical also wrote that FutureProofHomes Satellite1 will be using a “B” model in the “XU316-1024-QF60B " chip series s native 3.3v IO, as opposed to the “A” models in the “XU316-1024-QF60A " chip series that would need logic level shifters to convert to 1.8v IO for the ESP32. He also mention that they will go with a C32 (3200 MIPS) variant if the economy of scale looks to allow it without adding too much extra cost to the BOM.

18

FYI, FutureProofHomes have posted a new video on their YouTube channel showing off the current design of their ESP32-based hardware prototype upcoming FutureProofHomes Satellite1 voice control development board which looks to now be using such a XU316-1024-QF60A-C24 based XK-VOICE-L71 (XMOS Voice Reference Design Evaluation Kit connected externally. XK-VOICE-L71 (XMOS Voice Reference Design Evaluation Kit features 3,5mm line out jack for audio output to external speakers and @FutureProofHomes mentioned that their final dev-kit product will also feature a 3,5mm jack for audio output. Check it out their latest prototype introduction video here:

https://www.youtube.com/watch?v=Vp5q4RIwCX4

2 Likes
19

I’ve just setup my ReSpeaker.
Is there a way to control the speaker volume?

Additionally, it would be nice to keep the discussion in this thread on topic, it’s confusing to follow with the inclusion of other similar but unrelated developments.

20

Are you using the components from the esphome “voice-kit” repository? If so then maybe best would be for you to post that question as a new issue there to get the attention of the developers workong on that project? See → Issues · esphome/voice-kit · GitHub

Note that even if they are not using the new ReSpeaker Lite kit themselves the question is perhaps better addressed there as theymentioned tjat one goal for that repo is to not be reliant on specific hardware configurations.

Quoting kahrendt from another issue there in that repo:

"We are moving fast and breaking things as we figure out the best way for these components to interact, but we will add all the code to the base ESPHome project once things are stable and working well. One goal for this repo is to not be reliant on specific hardware configurations. These other boards you linked are quite exciting, and I believe they should be compatible (or relatively easily made compatible) with our changes. For example, all the I2S settings and pins are still configurable in yaml, so it should be straightforward to add support for similar boards.

Very little of the code is reliant specifically on the XMOS chip (and the few lines that are should be adaptable or won’t even be there in the final version as we clean up the code), so it should be possible to add support for other DSPs in the future."

2 Likes
21

I used the ESPHome yaml from the Seeed website Voice Assistant System for Home Assitant | Seeed Studio Wiki.

I’ll check out the voice-kit GitHub and see how that compares.

1 Like
22

Tbf, the op can do whatever they want with their thread.

2 Likes
23

I would argue that both ESPHome voice-kit development and other hardware using ESP32 and XMOS xCORE chips combination for audio AI acceleration is on-topic as without those these “ReSpeaker Lite” kits will in practice never be useful in the future for anyone who is not a ESPHome developer themselves.

The “bigger picture” to make all these eventually become useful to the avérage Home Assistant user they not only need to be fully supported in the upstream ESPHome project, they also need to standardize those devices in both ESPHome and the Home Assistant core, and ESHome developers are now working on several new components related to this, including a new entity component as assist_satellite platform for that which will represent a standard VoIP-based voice satellite for Home Assistant Assist voice control. Check out this architecture discussion (which sounds like it has essentially been approved)

Initial entity component for this new assist_satellite platform been merged to Home Assistant core:

Bigger picture:

  • Standardize how voice satellites expose their capabilities (ie, available wake words)
  • Standardize how voice satellites are configured (which wake word to listen to)
  • Automate based on the state of the satellite’s pipeline
  • Control the behavior of a voice satellite from HA during the setup wizard
  • Skip wake word and listen for a command (with or without executing it)
  • Listen for a specific wake word (without running a pipeline)
  • Control a voice satellite from HA using service calls
  • Announce text using the TTS portion of the satellite’s pipeline

Also see related patches with new features submitted to both ESPHome and the Home Assistant core:

2 Likes
24

Hi,
I’ve just received mine and I’m trying to modify the yaml to use micro wake word.
It seems to detect when I say the wake word but it stops streaming the audio just after switching to this state.
Is anyone succeeded on making it work ?

25

“mine” which ? ReSpeaker Lite, or one of the other similar devices mentioned through this thread ?
It would also make it easier for others to assist if you give a bit more information about your configuration (hardware and software). Eg, is the yaml in the ESPHome configuration for the device ?

1 Like
26

Have you tried using the experimental ”voice-kit” fork repo project from ESPHome instead of the main esphome repo?

27

Okay, I’m a bit confused. I already had 2 XIOS S3’s so I just ordered the board. How is the XIAO soldiered to the board? Taking a look at all the pics here and seseds website shows the XIAO sitting on the respeaker lite but I’m not seeing anything soldiered in any pictures.

They also have you flashing 2 different things, one for just the board and another for the i2s. Then on the page with instructions and YAML for ESPHome it says a Sonoff smart switch is required although outside adding it to HA, it’s not referenced anywhere.

It’s using Openeakeword with the default YAML, this is just our laziness by Seeed. I’m pretty sure the DFU driver is for when an ESP32-S3 XIAO isn’t used and using this with a PC. Then the i2s driver is able to be downloaded but the flashing instructions link takes you to a non existent page.

All in all Seeed has done a terrible job with documentaries and broken links on how to flash new firmware is just plain odd.

DFU firmware

1000007177
10000071771080×2173 327 KB

I2S firmware

1000007178
10000071781080×2171 309 KB

The frustrating part is they have an example of using a wake work using tensorflow lite but only.for Arduino…

Anyone who has the pre soldiered version could you post a pic of how it’s actually attached to the board? Below is the yaml from Seeed.

esphome:
  name: esp32s3
  friendly_name: ReSpeakerv3
  platformio_options:
    board_build.flash_mode: dio
    board_build.mcu: esp32s3
  # on_boot:
  #   then:
  #     - if:
  #            condition:
  #              switch.is_on: 
  #            then:
  #              - voice_assistant.start_continuous:


esp32:
  board: esp32-s3-devkitc-1
  variant: esp32s3
  framework:
    type: esp-idf
    version: recommended

logger:
  level: VERY_VERBOSE

api:


wifi:
  ssid: "Wi-Fi name"
  password: "Wi-Fi password"

captive_portal:

external_components:
  - source: github://QingWind6/ESPHome_XIAO-ESP32S3

i2s_audio_xiao:
  i2s_lrclk_pin: GPIO7
  i2s_bclk_pin: GPIO8
  i2s_mclk_pin: GPIO9


microphone:
  - platform: i2s_audio_xiao
    id: xiao_mic
    adc_type: external
    i2s_din_pin: GPIO44
    pdm: false
    bits_per_sample: 32bit
    channel: left

speaker:
  - platform: i2s_audio_xiao
    id: xiao_speaker
    dac_type: external
    i2s_dout_pin: GPIO43
    mode: stereo

voice_assistant:
  microphone: xiao_mic
  use_wake_word: true
  noise_suppression_level: 0
  auto_gain: 0dBFS
  volume_multiplier: 1
  speaker: xiao_speaker
  id: assist
  on_listening:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 0%
        effect: "Slow Pulse"
  on_stt_vad_end:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 0%
        effect: "Fast Pulse"
  on_tts_start:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 0%
        brightness: 100%
        effect: none
  on_end:
    - delay: 100ms
    - wait_until:
        not:
          speaker.is_playing:
    - script.execute: reset_led
  on_error:
    - light.turn_on:
        id: led
        red: 100%
        green: 0%
        blue: 0%
        brightness: 100%
        effect: none
    - delay: 1s
    - script.execute: reset_led
  on_client_connected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.start_continuous:
          - script.execute: reset_led
  on_client_disconnected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.stop:
          - light.turn_off: led

light:
  - platform: esp32_rmt_led_strip
    id: led
    name: None
    disabled_by_default: true
    entity_category: config
    pin: GPIO1
    default_transition_length: 0s
    chipset: ws2812
    num_leds: 1
    rgb_order: grb
    rmt_channel: 0
    effects:
      - pulse:
          name: "Slow Pulse"
          transition_length: 250ms
          update_interval: 250ms
          min_brightness: 50%
          max_brightness: 100%
      - pulse:
          name: "Fast Pulse"
          transition_length: 100ms
          update_interval: 100ms
          min_brightness: 50%
          max_brightness: 100%

output:
  - platform: ledc
    id: light_output
    pin: GPIO21
    inverted: true


script:
  - id: reset_led
    then:
      - if:
          condition:
            - switch.is_on: use_wake_word
            - switch.is_on: use_listen_light
          then:
            - light.turn_on:
                id: led
                effect: none

          else:
            - light.turn_off: 
               id: led


switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(assist).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(assist).set_use_wake_word(false);

  - platform: template
    name: Use Listen Light
    id: use_listen_light
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - script.execute: reset_led
    on_turn_off:
      - script.execute: reset_led