LOLA59 Voice Assistant + Multisensor + Apple Airplay

[journeywithlogan]

Sorry for the late submission!!
https://youtu.be/xVTrE-Fjf_Y?si=gwutcR-wTda-YdCO
Guys I’m proud to announce my LOLA-59 droid voice assistant. It took me months to get everything to this point and its not even finished but i just wanted to get it out there for you guys to see it.

I wanted a multi-sensor that did everything I needed it to do for daily use without needing multiple sensor so I decided to build one.

Hardware:
BM280 - Temp, Altitude, Humidity Sensor
Light Sensor -
Two sg90 servo motor to control the wings
Two PIR Motion Sensors
MMWave sensor
Airplay
GPIO Button
DFPlayer Mini
ESP32 S3
ESP32 WROVER-E mini running Squeezelite

I had a power draw/code issue unfortunately but there are two relays in there that control the wings and make the vibrate like a birds wings. I promise everything worked separately but when I put it all together I had issues lol. The problem that I noticed is that with all these sensors going a million miles an hour sending data the voice assistant seems to struggle but I could be wrong. Im new to this stuff and if your in the esphome forums I’m always asking a million questions lol

[jenova70 (JLo)]

looks great! Would you mind sharing the ESPHome config, please?
And any other material you want to share with the community?
(The reference of the droid for example)

[journeywithlogan]

Yea no problem, sorry I was so in a rush to upload for the contest I didnt really break it down.

The droid is a star wars droid from the Obi-Wan series on Disney + she accompined Princess Leia throughout the series. They sell the toy here Lola_59 on Amazon. I purchased the small version in Ross for $7.99 and you may still find them there. Ive bought several like 6 or 7 lol because ill be doing one of these in all my rooms.

There is all the bigger version you can buy at Walmart for like $70 but they also had them on sale for $20 the other day. I digress lol

The wings are moved by two sg90 servo motors and are supposed to move and activate on wake word detection. You can seem them move in the link below. Wing MovementI had them try to mimic insect like movement. However I think between power draw and all the other sensors running at the same time the system just shutdown. I probably have a short somewhere. You can see the guts of this thing below.

The sound you heard in the previous video is the DFPlayer going off with wakeword detection. I gutted the droid and replaced everything on it and removed parts to make it work. It came with a small board on the inside and leds which I gutted and replaced with ones that could be used by esphome/Home assistant

Theres a mmwave sensor in there and two PIR motion sensors. One where the eye would be and then one in the back. There were pros and cons for both so I decided I would put both in this thing. The goal was to have automations fire when I walk in the office.

Second, I had a temp sensor outside of the droid on the bottom. and a light sensor to the top.

Lastly I had thrown inside there an esp32 wrover-e mini to run squeezelite so I could have airplay. The speakers are mounted in the wings and will flap open for listen mode eventually whenever I finish the thing lol

Hindsight, I’ve already purchased and will run the voice assistant on its own board Seed Studio S3 mini board in the head of LOLA-59 with an INMP441 like damn near on top of it like ive seen one of the mods do in the ESPHOME forums. The voice processing run constantly but with other sensors sending all the date constantly updating it just needs its own or I slow down how often the other sensors send data. Or someone can help me lol!!!

Here is the code for esp32 its a mess lol but it all works. There are some lambdas written in there to randomize the wing movements for a script I wrote. Im no coder and it ran but how random it is God knows lol

ESPHOME CODE

esphome:
name: lola-v22
friendly_name: LOLA V22
platformio_options:
board_build.flash_mode: dio
board_build.arduino.memory_type: qio_opi
board_build.f_flash: 80000000L
on_boot:
- priority: -100
then:
- wait_until: api.connected
- delay: 5s
- if:
condition:
switch.is_on: use_wake_word
then:
- voice_assistant.start_continuous:
esp32:
board: esp32-s3-devkitc-1
variant: esp32s3
flash_size: 16MB
framework:
type: arduino
version: recommended
psram:
mode: octal
speed: 120MHz

Enable logging

logger:

Enable Home Assistant API

api:
encryption:
key: “rMK9fdHHxuBClb3UupL2Dffc2TOWMLnrYkRzzGvFLcU=”
services:

• service: dfplayer_next
  then:

  • dfplayer.play_next:

• service: dfplayer_previous
  then:

  • dfplayer.play_previous:

• service: dfplayer_play
  variables:
  file: int
  then:

  • dfplayer.play: !lambda ‘return file;’

• service: dfplayer_play_loop
  variables:
  file: int
  loop_: bool
  then:

  • dfplayer.play:
    file: !lambda ‘return file;’
    loop: !lambda ‘return loop_;’

• service: dfplayer_play_folder
  variables:
  folder: int
  file: int
  then:

  • dfplayer.play_folder:
    folder: !lambda ‘return folder;’
    file: !lambda ‘return file;’

• service: dfplayer_play_loop_folder
  variables:
  folder: int
  then:

  • dfplayer.play_folder:
    folder: !lambda ‘return folder;’
    loop: true

• service: dfplayer_set_device_tf
  then:

  • dfplayer.set_device: TF_CARD

• service: dfplayer_set_device_usb
  then:

  • dfplayer.set_device: USB

• service: dfplayer_set_volume
  variables:
  volume: int
  then:

  • dfplayer.set_volume: !lambda ‘return volume;’

• service: dfplayer_set_eq
  variables:
  preset: int
  then:

  • dfplayer.set_eq: !lambda ‘return static_castdfplayer::EqPreset(preset);’

• service: dfplayer_sleep
  then:

  • dfplayer.sleep

• service: dfplayer_reset
  then:

  • dfplayer.reset

• service: dfplayer_start
  then:

  • dfplayer.start

• service: dfplayer_pause
  then:

  • dfplayer.pause

• service: dfplayer_stop
  then:

  • dfplayer.stop

• service: dfplayer_random
  then:

  • dfplayer.random

• service: dfplayer_volume_up
  then:

  • dfplayer.volume_up

• service: dfplayer_volume_down
  then:

  • dfplayer.volume_down
    ota:
    password: “b58e0914ff20e38638aa53b9c0137f7d”

wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password

Enable fallback hotspot (captive portal) in case wifi connection fails

ap:
ssid: “Lola-V22 Fallback Hotspot”
password: “a6YnBWUrpTHY”
ld2410:
id: ld2410_radar
uart_id: uart_2

dfplayer:
uart_id: uart_bus1

uart:

• id: uart_bus1
  tx_pin: GPIO20
  rx_pin: GPIO19
  baud_rate: 9600
  parity: NONE
  stop_bits: 1

• id: uart_2
  tx_pin: GPIO14
  rx_pin: GPIO13
  baud_rate: 256000
  parity: NONE
  stop_bits: 1
  #########################################################################################
  #Global Integers
  globals:

• id: randomDelay
  type: int
  restore_value: no
  initial_value: ‘0’

i2c:

• id: bus_a
  sda: 44
  scl: 43
  scan: true
• id: bus_b
  sda: 38
  scl: 39
  scan: True

i2s_audio:

• id: i2s_out
  i2s_lrclk_pin: GPIO04 #WS
  i2s_bclk_pin: GPIO05 #SCK

microphone:

• platform: i2s_audio
  id: va_mic
  adc_type: external
  i2s_din_pin: GPIO01
  channel: left
  pdm: false
  bits_per_sample: 32bit
  #media_player:

- platform: i2s_audio

dac_type: external

mode: mono

name: Lola Speaker2

id: Lola_speaker2

i2s_audio_id: i2s_out

i2s_dout_pin: GPIO39

voice_assistant:
id: va
microphone: va_mic
auto_gain: 31dBFS
noise_suppression_level: 2
volume_multiplier: 4.0
use_wake_word: true

on_wake_word_detected:
then:
- dfplayer.play: 9

switch:

• platform: template
  name: Use wake word
  id: use_wake_word
  optimistic: true
  restore_mode: RESTORE_DEFAULT_ON
  entity_category: config
  on_turn_on:
  • lambda: id(va).set_use_wake_word(true);
  • if:
    condition:
    not:
    - voice_assistant.is_running
    then:
    - voice_assistant.start_continuous
    on_turn_off:
  • voice_assistant.stop
  • lambda: id(va).set_use_wake_word(false);

##########################################################################################
#Outputs
output:

• platform: ledc
  id: bottom_red
  pin: GPIO42
  frequency: 50Hz
• platform: ledc
  id: bottom_green
  pin: GPIO41
  frequency: 50Hz
• platform: ledc
  id: bottom_blue
  pin: GPIO40
• platform: ledc
  id: top_red
  pin: GPIO07
  frequency: 50Hz
• platform: ledc
  id: top_green
  pin: GPIO015
  frequency: 50Hz
• platform: ledc
  id: top_blue
  pin: GPIO016
  frequency: 50Hz
• platform: ledc
  id: pwm_output
  pin: GPIO11
  frequency: 50 Hz
• platform: ledc
  id: pwm_output2
  pin: GPIO12
  frequency: 50 Hz
  ############################################################################################
  #Servo Code
  servo:
• id: right_wing
  output: pwm_output
• id: left_wing
  output: pwm_output2

number:

• platform: template
  id: servo_value
  name: Right Wing
  min_value: -100
  initial_value: 0
  max_value: 100
  step: 1
  optimistic: true
  set_action:
  then:
  - servo.write:
  id: right_wing
  level: !lambda ‘return x / 100.0;’

• platform: template
  id: servo_value1
  name: Left Wing
  min_value: -100
  initial_value: 0
  max_value: 100
  step: 1
  optimistic: true
  set_action:
  then:
  - servo.write:
  id: left_wing
  level: !lambda ‘return x / 100.0;’

• platform: ld2410
  timeout:
  name: timeout
  light_threshold:
  name: light threshold
  max_move_distance_gate:
  name: max move distance gate
  max_still_distance_gate:
  name: max still distance gate
  g0:
  move_threshold:
  name: g0 move threshold
  still_threshold:
  name: g0 still threshold
  g1:
  move_threshold:
  name: g1 move threshold
  still_threshold:
  name: g1 still threshold
  g2:
  move_threshold:
  name: g2 move threshold
  still_threshold:
  name: g2 still threshold
  g3:
  move_threshold:
  name: g3 move threshold
  still_threshold:
  name: g3 still threshold
  g4:
  move_threshold:
  name: g4 move threshold
  still_threshold:
  name: g4 still threshold
  g5:
  move_threshold:
  name: g5 move threshold
  still_threshold:
  name: g5 still threshold
  g6:
  move_threshold:
  name: g6 move threshold
  still_threshold:
  name: g6 still threshold
  g7:
  move_threshold:
  name: g7 move threshold
  still_threshold:
  name: g7 still threshold
  g8:
  move_threshold:
  name: g8 move threshold
  still_threshold:
  name: g8 still threshold
  ##############################################################################################
  switch (Button)
  binary_sensor:

• platform: gpio
  pin:
  number: 2
  mode: INPUT_PULLUP
  inverted: true
  name: Lola Button"
  id: lola_button
  on_press:
  then:
  - script.execute: wings_animation
  on_double_click:
  then:
  script.stop: wings_animation

• platform: gpio
  pin: 9
  name: “Rear Motion Sensor”
  device_class: motion

• platform: ld2410
  has_target:
  name: Radar Target
  id: radar_has_target
  has_moving_target:
  name: Radar Moving Target
  has_still_target:
  name: Radar Still Target
  ###############################################################################################
  #Light
  light:

• platform: rgb
  name: “Lola Bottom Lights”
  red: bottom_red
  blue: bottom_blue
  green: bottom_green

• platform: rgb
  name: “Lola Lights”
  red: top_red
  blue: top_blue
  green: top_green
  ##############################################################################################
  #Media Player
  sensor:

• platform: bh1750
  i2c_id: bus_a
  name: “BH1750 Illuminance”
  address: 0x23
  update_interval: 10s

• platform: ld2410
  light:
  name: light
  moving_distance:
  name : Moving Distance
  still_distance:
  name: Still Distance
  moving_energy:
  name: Move Energy
  still_energy:
  name: Still Energy
  detection_distance:
  name: Detection Distance
  g0:
  move_energy:
  name: g0 move energy
  still_energy:
  name: g0 still energy
  g1:
  move_energy:
  name: g1 move energy
  still_energy:
  name: g1 still energy
  g2:
  move_energy:
  name: g2 move energy
  still_energy:
  name: g2 still energy
  g3:
  move_energy:
  name: g3 move energy
  still_energy:
  name: g3 still energy
  g4:
  move_energy:
  name: g4 move energy
  still_energy:
  name: g4 still energy
  g5:
  move_energy:
  name: g5 move energy
  still_energy:
  name: g5 still energy
  g6:
  move_energy:
  name: g6 move energy
  still_energy:
  name: g6 still energy
  g7:
  move_energy:
  name: g7 move energy
  still_energy:
  name: g7 still energy
  g8:
  move_energy:
  name: g8 move energy
  still_energy:
  name: g8 still energy

• platform: bme280_i2c
  i2c_id: bus_b
  temperature:
  name: “BME280 Temperature”
  id: bme280_temperature
  pressure:
  name: “BME280 Pressure”
  id: bme280_pressure
  humidity:
  name: “BME280 Relative Humidity”
  id: bme280_humidity
  address: 0x76
  update_interval: 15s

• platform: template
  name: “Altitude”
  lambda: |-
  const float STANDARD_SEA_LEVEL_PRESSURE = 1013.25; //in hPa, see note
  return ((id(bme280_temperature).state + 273.15) / 0.0065) *
  (powf((STANDARD_SEA_LEVEL_PRESSURE / id(bme280_pressure).state), 0.190234) - 1); // in meter
  update_interval: 15s
  icon: ‘mdi:signal’
  unit_of_measurement: ‘m’

• platform: absolute_humidity
  name: “Absolute Humidity”
  temperature: bme280_temperature
  humidity: bme280_humidity

• platform: template
  name: “Dew Point”
  lambda: |-
  return (243.5*(log(id(bme280_humidity).state/100)+((17.67id(bme280_temperature).state)/
  (243.5+id(bme280_temperature).state)))/(17.67-log(id(bme280_humidity).state/100)-
  ((17.67id(bme280_temperature).state)/(243.5+id(bme280_temperature).state))));
  unit_of_measurement: °C
  icon: ‘mdi:thermometer-alert’
  ##############################################################################################

##############################################################################################
#Scripts#
script:

• id: wing_shuttering
  then:

  • lambda: id(right_wing).write(39); #.48%
  • delay: .004s
  • lambda: id(right_wing).write(-.28);
  • delay: .01s
  • lambda: id(right_wing).write(39); #.48%
  • delay: .004s
  • lambda: id(right_wing).write(-.28);
  • delay: .01s
  • lambda: id(right_wing).write(39); #.48%
  • delay: .004s
  • lambda: id(right_wing).write(-.28);
  • delay: .01s

• id: wings_open
  then:

  • lambda: id(right_wing).write(-0.23);
  • lambda: id(left_wing).write(.058);

• id: wings_closed
  then:

  • lambda: id(right_wing).write(1.0);
  • lambda: id(left_wing).write(-.78);

• id: wings_animation
  then:

  • lambda: id(right_wing).write(-0.23);
  • lambda: id(left_wing).write(-.78);
  • lambda: ‘id(randomDelay) = rand() % 1800;’
  • logger.log:
    format: “rand returned value %i”
    args: [‘id(randomDelay)’]
  • delay: !lambda ‘return(id(randomDelay));’
  • lambda: id(left_wing).write(.058);
  • lambda: id(right_wing).write(1.0);
  • lambda: ‘id(randomDelay) = rand() % 1800;’
  • logger.log:
    format: “rand returned value %i”
    args: [‘id(randomDelay)’]
  • delay: !lambda ‘return(id(randomDelay));’
  • lambda: id(right_wing).write(-0.23);
  • lambda: id(left_wing).write(-.78);
  • lambda: ‘id(randomDelay) = rand() % 1800;’
  • logger.log:
    format: “rand returned value %i”
    args: [‘id(randomDelay)’]
  • delay: !lambda ‘return(id(randomDelay));’
  • lambda: id(left_wing).write(.058);
  • lambda: id(right_wing).write(1.0);
  • lambda: ‘id(randomDelay) = rand() % 1800;’
  • logger.log:
    format: “rand returned value %i”
    args: [‘id(randomDelay)’]
  • delay: !lambda ‘return(id(randomDelay));’
  • lambda: id(right_wing).write(-0.23);
  • lambda: id(left_wing).write(-.78);
  • lambda: ‘id(randomDelay) = rand() % 1800;’
  • logger.log:
    format: “rand returned value %i”
    args: [‘id(randomDelay)’]
  • delay: !lambda ‘return(id(randomDelay));’
  • lambda: id(left_wing).write(.058);
  • lambda: id(right_wing).write(1.0);
  • lambda: ‘id(randomDelay) = rand() % 1800;’
  • logger.log:
    format: “rand returned value %i”
    args: [‘id(randomDelay)’]
  • delay: !lambda ‘return(id(randomDelay));’
  • lambda: id(right_wing).write(-0.23);
  • lambda: id(left_wing).write(-.78);
  • lambda: ‘id(randomDelay) = rand() % 1800;’
  • logger.log:
    format: “rand returned value %i”
    args: [‘id(randomDelay)’]
  • delay: !lambda ‘return(id(randomDelay));’
  • lambda: id(left_wing).write(.058);
  • lambda: id(right_wing).write(1.0);
  • lambda: ‘id(randomDelay) = rand() % 1800;’
  • logger.log:
    format: “rand returned value %i”
    args: [‘id(randomDelay)’]
  • delay: !lambda ‘return(id(randomDelay));’
  • lambda: id(right_wing).write(-0.23);
  • lambda: id(left_wing).write(-.78);
  • lambda: ‘id(randomDelay) = rand() % 1800;’
  • logger.log:
    format: “rand returned value %i”
    args: [‘id(randomDelay)’]
  • delay: !lambda ‘return(id(randomDelay));’
  • lambda: id(left_wing).write(.058);
  • lambda: id(right_wing).write(1.0);
  • lambda: ‘id(randomDelay) = rand() % 1800;’
  • logger.log:
    format: “rand returned value %i”
    args: [‘id(randomDelay)’]
  • delay: !lambda ‘return(id(randomDelay));’
  • lambda: id(right_wing).write(-0.23);
  • lambda: id(left_wing).write(-.78);
  • lambda: ‘id(randomDelay) = rand() % 1800;’
  • logger.log:
    format: “rand returned value %i”
    args: [‘id(randomDelay)’]
  • delay: !lambda ‘return(id(randomDelay));’
  • lambda: id(left_wing).write(.058);
  • lambda: id(right_wing).write(1.0);
  • lambda: ‘id(randomDelay) = rand() % 1800;’
  • logger.log:
    format: “rand returned value %i”
    args: [‘id(randomDelay)’]
  • delay: !lambda ‘return(id(randomDelay));’

captive_portal:

[PhilippeP]

I do have one of those , even two : a small one and the big animatronic one …
Great job !

[journeywithlogan]

Thank you!!! Appreciate the love!!

[jenova70 (JLo)]

The results of the contest are out!
They may be of interest to you
Have a look!

[journeywithlogan]

Extremely appreciative!! Thank you guys for considering my project worthy!! Home Assistant is simply the best product with the best community!