Death Star Voice Assistant

Voice Assistant Contest
1

So my first inspiration for this project was this Death Star speaker:

I definitely wanted to do something with it, but I don’t think I had a definite plan in mind. (It’s been long enough and I’ve been working on the voice version long enough that I can’t say for certain any more.) Maybe a speaker that hung from our vaulted ceilings.

But then I came across this article:

and I knew that I had to combine the two. I also had hopes that I could make it a voice assistant. At the time we had just gotten the Amazon Alexa device, and I thought it would be great to have my own and not have to worry about Amazon spying on us.

So I resized the Death Star model to fit these 60 LED rings for a clock:

https://www.aliexpress.us/item/2251832846472906.html

(Though now I’m thinking of trying one of these: https://www.aliexpress.us/item/3256805361023510.html Just less soldering and already comes in a perfect circle. Seems to be slightly larger, though. Would probably have to redo the stl file for that. I might also have to add a ring to the Death Star stl to hold the clock ring in place easier. TBD)

And it needed some other modifications to get only half of it so I could wall-mount it, as well as make some mounts for the boards inside.

Not too long after that I came across Snips.ai and was super excited, as it did what I wanted, and all I needed to make satellites was a Raspberry Pi Zero. I got a bunch of those and some of the Playstation Eye cameras https://dn3sfj3czhskp.cloudfront.net/offers/534090/1510742357.jpg

which I figured looked cool, cause I could turn them on their side and 3d print a little piece, and it would look like a tie fighter to go with the Death Star. So I was playing around getting Snips to where I liked it, as well as getting the Death Star 3d model ready with mounts inside. I was going to mount a micro arduino board to handle the LED clock as well as the hit-sound sensor, and then some USB-ethernet adapter all inside the Death Star along with the Raspberry Pi Zero to make it all work.

So I was 95% to what my dream was, though still a ways off on getting the coding in place, and all of a sudden the Snips dashboard stopped working. Snips had been sold. I was devastated, and a year or two of work down the drain. (It takes me a long time to do these things, between my day job and kids and wife. I just don’t have a lot of free time where I also have the brain power to do some of the more difficult things.)

So back to the drawing board I went. I started to teach myself machine learning. I knew that some portion of the Snips stuff was open source, but to even know how to work with it I had to do some remedial learning. It was very slow going. I’m an EE, with some programming background, but I had basically no ML background at all. I would work on it for a while, get either frustrated by my lack of progress, or get other requests of things to get done from my wife or kids, so progress was extremely slow. Plus I always seem to have at least 5-10 projects in the works, so this one would very often get pushed to the end of the line, with no reasonable path to getting it done very soon.

Then a little over a year ago I was very excited to read that Home Assistant was doing the Year of Voice! I was sure that this meant I could finally have my Death Star, if I only waited for the back end work to get taken care of. I already had been using it for years in the house, gradually adding devices here and there, so I was all set once the support was there.

I used the notebook to create wake words for both “Death Star” and “Hey Death Star” which should also work for “Okay Death Star.” I wasn’t sure if just having two syllables was enough, which is why I have been using the three syllable version. It still gets a false trigger maybe once a day or so. I also added in a wake word for “Darth Vader” though I’m not using it yet. If/when home assistant allows for listening for multiple wake words at a time, I will enable it, and then can play around with doing other fun things when it is triggered.

The model is currently sized for these speakers:

https://www.aliexpress.us/item/3256803985928127.html

It can be adjusted as needed with changing some of the constants in the OpenSCAD file. I wanted it to be a 4 Ohm speaker so that I could get the max power out of my 5V power input. Since it’s by far the easiest for me to just use a USB plug to get power in, I worked around that as a constraint.

Of course some of the audio/voice files are too large for GitHub, so I put everything there in the Google folder.
There are multiple options for the wake word. Any of the files that end in .tflite, and the name is the actual wake word.
There are also multiple options for the voice output. The ‘high’ quality voices were trained to 30,000 steps with clips that I grabbed from various Star Wars movies and cleaned up in Audacity. I started with the lessac high quality voice using the Google Colab notebook provided by Nabu Casa people.
The exact same method was done for the ‘medium’ voice output, but it only went up to 10k steps, if I remember correctly. I wanted both outputs so that it would run on lesser hardware if I wanted to.

To get the external server running with GPU acceleration, I used code found here: GitHub - baudneo/wyoming-addons-gpu at gpu

Future / To-Do:

  • I have yet to get the clock ring to light up, which is why it’s not in the video. I suspect some kind of hardware issue at this point, but haven’t yet had the time to troubleshoot.
  • I would love to turn down the speaker volume in code, especially at night when people are going to bed. Haven’t figured out yet if that can be done in the satellite or if it has to be in Home Assistant itself.
  • Working on another revision of the board to use the S3 chips to do wakeword on the device. I have a working one, but I messed up on understanding the datasheet on the USB port, so I have to solder wires to do programming. Not ideal. :slight_smile:
  • Which would lead to needing to create wakeword files for that system.
  • Might also convert the board to just using the direct chips instead of the speaker amp and microphone modules. Not in a great hurry for this.
  • Still want to code up a system to also detect impacts from using the same microphone so it could be a Nerf dart target.
  • Working on a ring PCB so that I don’t have to hot-glue in the laser lights.
  • Putting in randomness, so sometimes Vader might say something like “I have turned off the lights. Do not make me do it again.” Or whatever. Going to teach my sons to be able to hide all kinds of fun ‘Easter Eggs’ in there.

If anyone wants to play with making their own, all of my source files are on the Google Drive, I think. If you’re unsure about anything, let me know and I’ll do what I can. If I missed some critical info, let me know and I’ll get an update on there.

https://drive.google.com/drive/folders/1hDpHxRE_dsLG1RAh9epF2R9A2iLYw3wT?usp=sharing

You can see the current hot-glue version here.
Google Photos

I’ve gotta thank Nabu Casa for doing the bulk of the work to get me there with this project! Like I said, it’s been many years, and I’ve had many setbacks. I was maybe 75% of the way there when the contest was announced, so it was good to have a deadline to get this done. My wife and kids love showing this off to people, and give me praise for it. It gives me a very inordinate amount of childish joy every time Darth Vader tells me that he “turned off the light.” :rofl: The contest honestly doesn’t matter at all, and I already have a Home Assistant blue, as well as the server with an Nvidia video card. I just absolutely LOVE this project! I’m hoping that it might bring a few other people their own joy to put together and have on their wall.

2

Gods that’s incredible and honestly hilariously ominous, especially when combined with the Darth Vader voice responses.

The model and lights look incredible btw! Honestly, I think any fan would already love having this just as a night light, let alone a voice assistant :grin:

3

Thank you, that made my day! And thank you about the compliment on the looks! I should have mentioned, but I dragged my kids into painting them for me. I have another in the living room, and then a few more in process of getting put together. Probably will end up with 8-10 throughout the house by the time I’m done.

I had it in my bedroom in the video, but then removed it because every day or two it gets a false trigger randomly. Sometimes no one is even in the room with it when it happens. I would have a very unhappy wife if she was woken from a deep sleep from a raspy Darth Vader! :laughing: Not even to mention how much I would not enjoy having that voice bring me out of a dream.

But your post made me think a bit more about it, and I realized that I could simply disable the wake word on that device on a timed automation. Or maybe if a light gets turned on by something else when the timer normally says it should be off? I’m sure there are other factors here, but it doesn’t even need anything more than what I already have for me to make it work. So thank you for that!

I mean, I still want to have control of the volume, but I was stuck on that without realizing that I already have enough for a good start.

1 Like
4

Do you have the ESPHome config, I may have missed it :wink:
Thx for your entry!

5

Oops, yeah, you’re right, I never did get that uploaded. I just put it on my Google drive folder, but I’ll add it here as well.

This of course does not include my attempts at integrating the LED ring for the clock, since I have yet to make that work. :rofl:

esphome:
  name: living-room-deathstar
  friendly_name: Living Room Deathstar

esp32:
  board: esp32dev
  framework:
    type: arduino

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: !secret living_room_deathstar_encryption_key

ota:
  password: !secret living_room_deathstar_ota_password

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  # Optional manual IP
  manual_ip:
    static_ip: 192.168.0.46
    gateway: 192.168.0.1
    subnet: 255.255.255.0

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Deathstar-Living-Room"
    password: !secret living_room_deathstar_portal_password

captive_portal:
        

i2s_audio:
  - id: i2s_in
    i2s_lrclk_pin: GPIO26   #WS / LRC
    i2s_bclk_pin: GPIO25    #SCK /BCLK

microphone:
  - platform: i2s_audio
    adc_type: external
    pdm: false
    id: mic_i2s
    channel: right
    bits_per_sample: 32bit
    i2s_audio_id: i2s_in
    i2s_din_pin: GPIO33    #SD

speaker:
  - platform: i2s_audio
    id: my_speaker
    dac_type: external
    i2s_dout_pin: GPIO27   #DIN 
    mode: mono
    i2s_audio_id: i2s_in



light:
  - platform: fastled_clockless
    id: led
    name: None
    disabled_by_default: true
    entity_category: config
    pin: GPIO4
    default_transition_length: 0s
    chipset: WS2812B
    num_leds: 8
    rgb_order: rgb
    effects:
      - pulse:
          name: "Slow Pulse"
          transition_length: 250ms
          update_interval: 250ms
          min_brightness: 50%
          max_brightness: 100%
      - pulse:
          name: "Fast Pulse"
          transition_length: 100ms
          update_interval: 100ms
          min_brightness: 50%
          max_brightness: 100%


script:
  - id: reset_led
    then:
      - if:
          condition:
            - switch.is_on: use_wake_word
            - switch.is_on: use_listen_light
          then:
            - light.turn_on:
                id: led
                red: 100%
                green: 89%
                blue: 71%
                brightness: 60%
                effect: none
          else:
            - light.turn_off: led

voice_assistant:
  microphone: mic_i2s
  id: va
  noise_suppression_level: 4
  auto_gain: 31dBFS
  volume_multiplier: 6.0
  use_wake_word: true
  speaker: my_speaker
  
  on_error: 
    - light.turn_on:
        id: led
        red: 100%
        green: 0%
        blue: 0%
        brightness: 100%
        effect: none
    - delay: 1s
    - script.execute: reset_led  
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - switch.turn_off: use_wake_word
          - switch.turn_on: use_wake_word

  on_client_connected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.start_continuous:
          - script.execute: reset_led

  on_client_disconnected:
    - if:
        condition:
          switch.is_on: use_wake_word
        then:
          - voice_assistant.stop:
          - light.turn_off: led

  on_listening:
    - light.turn_on:
        id: led
        blue: 0%
        red: 0%
        green: 100%
        effect: "Slow Pulse"
  on_stt_vad_end:
    - light.turn_on:
        id: led
        blue: 0%
        red: 0%
        green: 100%
        effect: "Fast Pulse"
  on_tts_start:
    - light.turn_on:
        id: led
        blue: 100%
        red: 0%
        green: 0%
        brightness: 100%
        effect: none
  on_end:
    - delay: 100ms
    - wait_until:
        not:
          speaker.is_playing:
    - script.execute: reset_led
    - light.turn_off: led

  

binary_sensor:
  - platform: status
    name: API Connection
    id: api_connection
    filters:
      - delayed_on: 1s
    on_press:
      - if:
          condition:
            switch.is_on: use_wake_word
          then:
            - voice_assistant.start_continuous:
    on_release:
      - if:
          condition:
            switch.is_on: use_wake_word
          then:
            - voice_assistant.stop:


switch:
  - platform: template
    name: Use wake word
    id: use_wake_word
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - lambda: id(va).set_use_wake_word(true);
      - if:
          condition:
            not:
              - voice_assistant.is_running
          then:
            - voice_assistant.start_continuous
      - script.execute: reset_led
    
    on_turn_off:
      - voice_assistant.stop
      - lambda: id(va).set_use_wake_word(false);
      - script.execute: reset_led

  - platform: template
    name: Use Listen Light
    id: use_listen_light
    optimistic: true
    restore_mode: RESTORE_DEFAULT_ON
    entity_category: config
    on_turn_on:
      - script.execute: reset_led
    on_turn_off:
      - script.execute: reset_led
6

The results of the contest are out!
They may be of interest to you :wink:
Have a look!

1 Like
7

Thank you for letting me know!

And thanks for holding the contest - there was a lot of amazing work done by so many people!

No need to send me the SkyConnect. I already have one. :slight_smile: Like I said, my reason for adding my entry was just a hope that it might add joy to someone else’s life.

1 Like