LEGO R2D2 Voice Assistant

Share your Projects!
,
1

Because I don’t have a 3D printer (yet), and was thinking of a cool and convenient way to deploy a voice assistant. LEGO is a cool alternative to 3D printing to build a casing, but definitely not cheap! The servo movement sound gives you confirmation that the wake word is detected and is very helpful when you are not able to see the lights of the ATOM echo. But I also made sure you can see the lights of the ATOM echo through the front, so you can see when VAD starts and stops.

It looks really nice on the shelf! However it’s just a fun project. The ATOM echo is not a premium product, so this voice assistant (in terms of microphone/speaker hardware) is far off the standards of Google and Alexa.

I am combining this with local AI, which works a lot better when the command is misheard from a distance. (The demo video above is using the default HA Conversation agent).

IMG20240909195544
IMG202409091955441920×2560 198 KB
IMG20240909195702
IMG202409091957021920×2560 220 KB
IMG20240909195713
IMG202409091957131920×2560 179 KB
IMG20240909195753
IMG202409091957531920×2560 227 KB
IMG20240909195919
IMG202409091959191920×2560 241 KB
IMG20240909195935
IMG202409091959351920×2560 165 KB

The R2D2 Voice Assistant is based on this demo video.

7 Likes
2

This is awesome, are you using the m5stack for controlling the servo or using a second ESP modul?

3

Thanks! The servo is connected to the Atom Echo, so no extra ESP used.

4

Could you please share the ESP config, I couldn’t handled it.
either the sound or the servo works, but not at the same time

5

Will do when I find the time! (Hopefully tomorrow)

6

thank you :slight_smile:

7 
esphome:
  name: m5stack-atom-echo-226060-226060
  name_add_mac_suffix: false
  friendly_name: M5Stack Atom Echo 226060
  project:
    name: m5stack.atom-echo-voice-assistant
    version: "1.0"
  min_version: 2023.11.1
############# Tried to add a R2D2 reaction after wakeword detected, but speaker cant be used ##########
#  includes:
#    - r2d2.h

esp32:
  board: m5stack-atom
  framework:
    type: esp-idf

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: "kcExnLd4TyDcQ3OD9jF/FnsNTNjme4UFxjw96Xx1nuw="

ota:
  password: "e6b40e3f40f1434978f0113da5246078"

dashboard_import:
  package_import_url: github://esphome/firmware/voice-assistant/m5stack-atom-echo.yaml@main

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  on_connect:
    - delay: 5s  # Gives time for improv results to be transmitted
    - ble.disable:
  on_disconnect:
    - ble.enable:
  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "M5Stack-Atom-Echo"
    password: "KXDLC8pEgAFK"

captive_portal:

improv_serial:

esp32_improv:
  authorizer: none

# Using ledc output to control a standard 50Hz RC servo over the normal 1 to 2 mS pulse range
output:
  - platform: ledc
    pin: GPIO21
    id: servo
    frequency: 50 Hz
    min_power: 5.0%    # 5% at 50Hz is 1mS  (20mS cycles)
    max_power: 10.0%   # 10% at 50Hz is 2mS (20mS cycles)

number:
  - platform: template
    name: "Servo Control"
    optimistic: true
    min_value: 0
    max_value: 100
    step: 1
    on_value:
      - then:
        - lambda: !lambda |-
            id(servo).set_level(x / 100);

button:
  - platform: factory_reset
    id: factory_reset_btn
    name: Factory reset

i2s_audio:
  i2s_lrclk_pin: GPIO33
  i2s_bclk_pin: GPIO19

microphone:
  - platform: i2s_audio
    id: echo_microphone
    i2s_din_pin: GPIO23
    adc_type: external
    pdm: true

speaker:
  - platform: i2s_audio
    id: echo_speaker
    i2s_dout_pin: GPIO22
    dac_type: external
    mode: mono

voice_assistant:
  id: va
  microphone: echo_microphone
  speaker: echo_speaker
  noise_suppression_level: 2
  auto_gain: 31dBFS
  volume_multiplier: 2.0
  vad_threshold: 3
  on_wake_word_detected:
    - output.set_level:
        id: servo
        level: 50%
#    - speaker.play:
#        id: echo_speaker
#        data: !lambda return r2d2_excited;
  on_listening:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 100%
        effect: "Slow Pulse"
  on_stt_vad_end:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 100%
        effect: "Fast Pulse"
  on_tts_start:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 100%
        brightness: 100%
        effect: none
    - output.set_level:
        id: servo
        level: 0%
    - delay: 500ms
    - output.set_level:
        id: servo
        level: 100%
  on_end:
    - delay: 100ms
    - wait_until:
        not:
          speaker.is_playing:
    - script.execute: reset_led
  on_error:
    - light.turn_on:
        id: led
        red: 100%
        green: 0%
        blue: 0%
        brightness: 100%
        effect: none
    - delay: 1s
    - script.execute: reset_led
  on_client_connected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.start_continuous:
          - script.execute: reset_led
  on_client_disconnected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.stop:
          - light.turn_off: led

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO39
      inverted: true
    name: Button
    disabled_by_default: true
    entity_category: diagnostic
    id: echo_button
    on_multi_click:
      - timing:
          - ON for at least 250ms
          - OFF for at least 50ms
        then:
          - if:
              condition:
                switch.is_off: use_wake_word
              then:
                - if:
                    condition: voice_assistant.is_running
                    then:
                      - voice_assistant.stop:
                      - script.execute: reset_led
                    else:
                      - voice_assistant.start:
              else:
                - voice_assistant.stop
                - delay: 1s
                - script.execute: reset_led
                - script.wait: reset_led
                - voice_assistant.start_continuous:
      - timing:
          - ON for at least 10s
        then:
          - button.press: factory_reset_btn

light:
  - platform: esp32_rmt_led_strip
    id: led
    name: None
    disabled_by_default: true
    entity_category: config
    pin: GPIO27
    default_transition_length: 0s
    chipset: SK6812
    num_leds: 1
    rgb_order: grb
    rmt_channel: 0
    effects:
      - pulse:
          name: "Slow Pulse"
          transition_length: 250ms
          update_interval: 250ms
          min_brightness: 50%
          max_brightness: 100%
      - pulse:
          name: "Fast Pulse"
          transition_length: 100ms
          update_interval: 100ms
          min_brightness: 50%
          max_brightness: 100%

script:
  - id: reset_led
    then:
      - if:
          condition:
            - switch.is_on: use_wake_word
            - switch.is_on: use_listen_light
          then:
            - light.turn_on:
                id: led
                red: 100%
                green: 89%
                blue: 71%
                brightness: 60%
                effect: none
          else:
            - light.turn_off: led

switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(va).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
      - script.execute: reset_led
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(va).set_use_wake_word(false);
      - script.execute: reset_led
  - platform: template
    name: Use listen light
    id: use_listen_light
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - script.execute: reset_led
    on_turn_off:
      - script.execute: reset_led

external_components:
  - source: github://pr#5230
    components:
      - esp_adf
    refresh: 0s

esp_adf:

interval:
  - interval: 1s
    then:
      - if:
          condition:
            api.connected:
          then:
            - if:
                condition:
                  and:
                    - switch.is_on: use_wake_word
                    - not:
                      - voice_assistant.is_running
                then:
                  - voice_assistant.start_continuous:
8

Thank you very much :slight_smile:

1 Like
9

How are you doing the r2d2 noises is that a seperate speaker?

10

Nope just the Atom Echo in there. I am using this TTS integration.

11

Thank you so much, @BramNH . Im trying to build myself, your code will be very useful!

1 Like
12

Hi @BramNH, awesome, I love it !

I buy the same R2-D2 Lego to do it with my son, dou you have some picture of the implementation and link of what you have bought to do it ? (Witch servo ti plug with the atom) ?

Thank in advance !

13

Thanks! I will add some pictures to this post this afternoon. I used a SG90 mini servo on which I screwed a flat Lego piece, that could be attached to the inside of the R2 head.

1 Like
14

I updated the post with pictures of the build

1 Like
15

Thank a lot, for sharing all that think ! R2-D2 building in progress :slightly_smiling_face:
Last question about link between sg90 and atom, based on your esphome config you plug the servo on gpio21 like un that picture?

1000019802
10000198021079×1110 85.2 KB

16

Yes indeed, servo is directly plugged into powered by the 5V pin and controlled by GPIO21 of the Atom Echo. Have fun! :slight_smile:

1 Like
17

@BramNH hey this is great, I have been trying to integrate a servo with the atom echo for awhile now with no luck. I 3D printed a droid to use as a voice assistant. I gave your code a try, but it is giving me an error message…

INFO ESPHome 2024.10.3
INFO Reading configuration /config/esphome/esphome-web-23e8a4.yaml...
INFO Updating https://github.com/esphome/esphome.git@pull/5230/head
Failed config

ota.unknown: [source /config/esphome/esphome-web-23e8a4.yaml:27]
  
  'ota' requires a 'platform' key but it was not specified.
  password: e6b40e3f40f1434978f0113da5246078
18

You need to modify some things to make it work for your microcontroller. OTA password is a unique password for Over The Air updates (this one for my device). Follow the esphome tutorial to get to know more about this.

I also built this on a much older version of ESPHome (maybe 2024.6). So I am not sure if it is directly compatible with your 2024.10

19

Thanks, I got it up and running! I just removed those few lines and was able to compile and install. Now just have to install it in my droid. Here is the code in case anyone else runs into the issue. Thanks again!

esphome:
  name: atom-echo-r5-astromech
  name_add_mac_suffix: false
  friendly_name: Atom Echo - R5 Astromech
  project:
    name: m5stack.atom-echo-r5-voice-assistant
    version: "1.0"
  min_version: 2023.11.1
############# Tried to add a R2D2 reaction after wakeword detected, but speaker cant be used ##########
#  includes:
#    - r2d2.h

esp32:
  board: m5stack-atom
  framework:
    type: esp-idf

# Enable logging
logger:

# Enable Home Assistant API
api:

ota:
  - platform: esphome
    id: ota_esphome

dashboard_import:
  package_import_url: github://esphome/firmware/voice-assistant/m5stack-atom-echo.yaml@main

wifi:
  ssid: 
  password: 
  on_connect:
    - delay: 5s  # Gives time for improv results to be transmitted
    - ble.disable:
  on_disconnect:
    - ble.enable:
  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "M5Stack-Atom-Echo"
    password: "KXDLC8pEgAFK"

captive_portal:

improv_serial:

esp32_improv:
  authorizer: none

# Using ledc output to control a standard 50Hz RC servo over the normal 1 to 2 mS pulse range
output:
  - platform: ledc
    pin: GPIO21
    id: servo
    frequency: 50 Hz
    min_power: 5.0%    # 5% at 50Hz is 1mS  (20mS cycles)
    max_power: 10.0%   # 10% at 50Hz is 2mS (20mS cycles)

number:
  - platform: template
    name: "Servo Control"
    optimistic: true
    min_value: 0
    max_value: 100
    step: 1
    on_value:
      - then:
        - lambda: !lambda |-
            id(servo).set_level(x / 100);

button:
  - platform: factory_reset
    id: factory_reset_btn
    name: Factory reset

i2s_audio:
  i2s_lrclk_pin: GPIO33
  i2s_bclk_pin: GPIO19

microphone:
  - platform: i2s_audio
    id: echo_microphone
    i2s_din_pin: GPIO23
    adc_type: external
    pdm: true

speaker:
  - platform: i2s_audio
    id: echo_speaker
    i2s_dout_pin: GPIO22
    dac_type: external
    channel: mono

voice_assistant:
  id: va
  microphone: echo_microphone
  speaker: echo_speaker
  noise_suppression_level: 2
  auto_gain: 31dBFS
  volume_multiplier: 2.0
  vad_threshold: 3
  on_wake_word_detected:
    - output.set_level:
        id: servo
        level: 50%
#    - speaker.play:
#        id: echo_speaker
#        data: !lambda return r2d2_excited;
  on_listening:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 100%
        effect: "Slow Pulse"
  on_stt_vad_end:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 100%
        effect: "Fast Pulse"
  on_tts_start:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 100%
        brightness: 100%
        effect: none
    - output.set_level:
        id: servo
        level: 0%
    - delay: 500ms
    - output.set_level:
        id: servo
        level: 100%
  on_end:
    - delay: 100ms
    - wait_until:
        not:
          speaker.is_playing:
    - script.execute: reset_led
  on_error:
    - light.turn_on:
        id: led
        red: 100%
        green: 0%
        blue: 0%
        brightness: 100%
        effect: none
    - delay: 1s
    - script.execute: reset_led
  on_client_connected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.start_continuous:
          - script.execute: reset_led
  on_client_disconnected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.stop:
          - light.turn_off: led

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO39
      inverted: true
    name: Button
    disabled_by_default: true
    entity_category: diagnostic
    id: echo_button
    on_multi_click:
      - timing:
          - ON for at least 250ms
          - OFF for at least 50ms
        then:
          - if:
              condition:
                switch.is_off: use_wake_word
              then:
                - if:
                    condition: voice_assistant.is_running
                    then:
                      - voice_assistant.stop:
                      - script.execute: reset_led
                    else:
                      - voice_assistant.start:
              else:
                - voice_assistant.stop
                - delay: 1s
                - script.execute: reset_led
                - script.wait: reset_led
                - voice_assistant.start_continuous:
      - timing:
          - ON for at least 10s
        then:
          - button.press: factory_reset_btn

light:
  - platform: esp32_rmt_led_strip
    id: led
    name: None
    disabled_by_default: true
    entity_category: config
    pin: GPIO27
    default_transition_length: 0s
    chipset: SK6812
    num_leds: 1
    rgb_order: grb
    rmt_channel: 0
    effects:
      - pulse:
          name: "Slow Pulse"
          transition_length: 250ms
          update_interval: 250ms
          min_brightness: 50%
          max_brightness: 100%
      - pulse:
          name: "Fast Pulse"
          transition_length: 100ms
          update_interval: 100ms
          min_brightness: 50%
          max_brightness: 100%

script:
  - id: reset_led
    then:
      - if:
          condition:
            - switch.is_on: use_wake_word
            - switch.is_on: use_listen_light
          then:
            - light.turn_on:
                id: led
                red: 100%
                green: 89%
                blue: 71%
                brightness: 60%
                effect: none
          else:
            - light.turn_off: led

switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(va).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
      - script.execute: reset_led
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(va).set_use_wake_word(false);
      - script.execute: reset_led
  - platform: template
    name: Use listen light
    id: use_listen_light
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - script.execute: reset_led
    on_turn_off:
      - script.execute: reset_led

external_components:
  - source: github://pr#5230
    components:
      - esp_adf
    refresh: 0s

esp_adf:

interval:
  - interval: 1s
    then:
      - if:
          condition:
            api.connected:
          then:
            - if:
                condition:
                  and:
                    - switch.is_on: use_wake_word
                    - not:
                      - voice_assistant.is_running
                then:
                  - voice_assistant.start_continuous: