Home Assistant Voice Preview Edition with Display

Hi everyone,

I want to share my project with my VPE.
The display shows the current time and the outside temperature, the temperature of the hot water and the production of the PV.

This is my first post here so I will try to upload the STL files of my case. If you can give me a hint how to do this please :slight_smile:
edit: https://www.thingiverse.com/thing:6926739
The internal speaker is in the case and the case is not completely closed to give the speaker a chance :slight_smile:

The yaml code for the display and the sensors:
I also added 2 sensors to show me the stt text and the tts text of the VPE to fine tune my custom sentences to get better results:

substitutions:
  name: home-assistant-voice-096a1b
  friendly_name: Voice1
packages:
  Nabu Casa.Home Assistant Voice PE: github://esphome/home-assistant-voice-pe/home-assistant-voice.yaml
esphome:
  name: ${name}
  name_add_mac_suffix: false
  friendly_name: ${friendly_name}
api:
  encryption:
    key: xxxxxxxx=


wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

text_sensor:
  - platform: template
    name: "text-to-speech"
    id: tts
  - platform: template
    name: "speech-to-text"
    id: stt

voice_assistant:
  id: va
  on_stt_end:
    - text_sensor.template.publish:
        id: tts
        state: !lambda 'return x;'
  on_tts_start:
    - text_sensor.template.publish:
        id: stt
        state: !lambda 'return x;'

time:
  - platform: homeassistant
    id: esptime

sensor:
  - platform: homeassistant
    id: inside_temperature
    entity_id: sensor.lumi_lumi_weather_temperature_2
    internal: true
  - platform: homeassistant
    id: outside_temperature
    entity_id: sensor.wetter_temperature_average
    internal: true
  - platform: homeassistant
    id: warmwasser_temperature
    entity_id: sensor.warmwasser_temperatur_2
    internal: true
  - platform: homeassistant
    id: pv_produktion
    entity_id: sensor.pv_produktion_gesamt
    internal: true

font:
  - file: 'digital-7.ttf'
    id: font_clock80
    size: 80
  - file: 'digital-7.ttf'
    id: font_clock60
    size: 60
  - file: 'digital-7.ttf'
    id: font_clock40
    size: 40
  - file: 'digital-7.ttf'
    id: font_clock32
    size: 32
  - file: 'BebasNeue-Regular.ttf'
    id: font80
    size: 80
  - file: 'BebasNeue-Regular.ttf'
    id: font60
    size: 60
  - file: 'BebasNeue-Regular.ttf'
    id: font50
    size: 50
  - file: 'BebasNeue-Regular.ttf'
    id: font32
    size: 32

  - file: 'digital-7.ttf'
    id: font3
    size: 80  

  - file: 'BebasNeue-Regular.ttf'
    id: font4
    size: 60
    
  - file: 'BebasNeue-Regular.ttf'
    id: font5
    size: 20
    
  - file: 'BebasNeue-Regular.ttf'
    id: font6
    size: 32

image:
  - file: mdi:water-thermometer-outline
    id: warmwasser_icon
    resize: 30x30
  - file: mdi:solar-power-variant
    id: pv_icon
    resize: 30x30
  - file: mdi:sun-thermometer-outline
    id: outtemp_icon
    resize: 30x30

spi:
  clk_pin: GPIO40 # (SCK/CLK)
  mosi_pin: GPIO41 # (SDA)

display:
  - platform: ili9xxx
    model: ST7735
    id: my_display
    dc_pin: GPIO01
    cs_pin: GPIO48
    reset_pin: GPIO42
    invert_colors: false
    rotation: 90
    dimensions:
      height: 160
      width: 128
      offset_width: 0
      offset_height: 0
    pages:
      - id: page1
        lambda: |-
          // Print time in HH:MM format
          it.strftime(80, 0, id(font80), TextAlign::TOP_CENTER, "%H:%M", id(esptime).now());
          it.image(0, 88, id(outtemp_icon));          
          // Print outside temperature (from homeassistant sensor)
          if (id(outside_temperature).has_state()) {
            it.printf(160, 70, id(font60), TextAlign::TOP_RIGHT , "%.1f°", id(outside_temperature).state);  // Celsius als Text
          }
      - id: page2
        lambda: |-
          // Print time in HH:MM format
          it.strftime(80, 0, id(font80), TextAlign::TOP_CENTER, "%H:%M", id(esptime).now());
          it.image(0, 88, id(warmwasser_icon));
          // Print Warmwasser-Temperatur (from homeassistant sensor)
          if (id(warmwasser_temperature).has_state()) {
            it.printf(160, 70, id(font60), TextAlign::TOP_RIGHT , "%.1f°", id(warmwasser_temperature).state);  // Celsius als Text
          }

      - id: page3
        lambda: |-
          // Print time in HH:MM format
          it.strftime(80, 0, id(font80), TextAlign::TOP_CENTER, "%H:%M", id(esptime).now());
          it.image(0, 88, id(pv_icon));
          // Print pv_produktion (from homeassistant sensor)
          if (id(pv_produktion).has_state()) {
            it.printf(160, 75, id(font50), TextAlign::TOP_RIGHT , "%.0f W", id(pv_produktion).state);  // Celsius als Text
          }


# Define the GPIO pin for controlling the display backlight
output:
  - platform: gpio
    id: display_backlight_output   # Ă„ndere den id-Namen fĂĽr das output
    pin: GPIO02  # Ă„ndere den Pin auf den richtigen fĂĽr dein Display

# Define a switch to control the backlight
switch:
  - platform: output
    name: "Display Backlight"
    id: display_backlight_switch   # Ă„ndere den id-Namen fĂĽr den switch
    output: display_backlight_output
    restore_mode: ALWAYS_ON
    
# For example cycle through pages on a timer
interval:
  - interval: 5s
    then:
      - display.page.show_next: my_display
      - component.update: my_display
    
9 Likes

Looking really good. How is the sound from the internal speaker?

The internal speaker is OK, but only for speech, not for music. I use an external speaker. In my case it is at the bottom near the opening and you can hear it, but a little quieter than in the original case

This looks really nice. What display are you using? I have a little ESP32 based LED clock on my spouse’s nightstand, but if we ever wanted to do voice commands, this would be nice, as it would still be one device on her nightstand.

i bought this one: AZDelivery 3 x 1,77 Zoll SPI TFT Display 128x160 Pixel ST7735 on amazon.
Do you have also an alarm on your led clock, or only the clock?
My VPE is on the living room, but i have a second one which will be in the bed room an for this one i hope to implement an alarm clock and i want to show the alarm time in the second row of the display.
So if you have an alarm clock on your esp32 it would be nice to see your yaml to get some ideas :slight_smile:

Nope. Just a clock. My spouse had a pretty old LED clock, and that battery connector on it broke, so when the power goes out the clock resets. So I built one with this LED and an ESP32:

and used ESPHome to link it to HA to get the time. As an added bonus, it now updates during the daily savings switch in the US.

This looks cool, is it possible to show how you hooked things up inside. Such as how you mounted the speaker, what pins from the LCD to the board, etc…

This is cool! Can it do “Close captioning” for the voice assistant responses?

This script will show you how to have the speaker show TTS (text-2-speech).

substitutions:
  name: home-assistant-voice-#####
  friendly_name: Home Assistant Voice - Guest Room
packages:
  Nabu Casa.Home Assistant Voice PE: github://esphome/home-assistant-voice-pe/home-assistant-voice.yaml
esphome:
  name: ${name}
  name_add_mac_suffix: false
  friendly_name: ${friendly_name}
api:
  encryption:
    key: ########


wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

text_sensor:
  - platform: template
    name: "User Speech Input"
    id: stt_display
    update_interval: never  
  - platform: template
    name: "Assistant Response"
    id: tts_display
    update_interval: never  

# https://github.com/esphome/home-assistant-voice-pe/blob/5288dde504325130d50f9465b92a6d712203f49a/home-assistant-voice.yaml#L999
voice_assistant:
  id: va
  on_stt_end:
    - text_sensor.template.publish:
        id: stt_display
        state: !lambda 'return x;'
    - display.page.show: page_tts
    - component.update: my_display

  on_tts_start:
    - text_sensor.template.publish:
        id: tts_display
        state: !lambda 'return x;'
    - display.page.show: page_tts
    - component.update: my_display    

  on_listening:
    - display.page.show: page_listen  
    - component.update: my_display        

time:
  - platform: homeassistant
    id: esptime

sensor:
  - platform: homeassistant
    id: inside_temperature
    entity_id: sensor.gw1000b_v1_7_5_indoor_temperature
    internal: true
  - platform: homeassistant
    id: outside_temperature
    entity_id: sensor.gw1000b_v1_7_5_temperature_1
    internal: true
  - platform: homeassistant
    id: pv_produktion
    entity_id: sensor.pv_produktion_gesamt
    internal: true

font:
  - file: 'digital-7.ttf'
    id: font_clock80
    size: 80
  - file: 'digital-7.ttf'
    id: font_clock60
    size: 60
  - file: 'digital-7.ttf'
    id: font_clock40
    size: 40
  - file: 'digital-7.ttf'
    id: font_clock32
    size: 32
  - file: 'BebasNeue-Regular.ttf'
    id: font70
    size: 70
  - file: 'BebasNeue-Regular.ttf'
    id: font80
    size: 80
  - file: 'BebasNeue-Regular.ttf'
    id: font60
    size: 60
  - file: 'BebasNeue-Regular.ttf'
    id: font55
    size: 55
  - file: 'BebasNeue-Regular.ttf'
    id: font50
    size: 50
  - file: 'BebasNeue-Regular.ttf'
    id: font32
    size: 32
  - file: 'BebasNeue-Regular.ttf'
    id: font20
    size: 20


  - file: 'digital-7.ttf'
    id: font3
    size: 80  

  - file: 'BebasNeue-Regular.ttf'
    id: font4
    size: 60
    
  - file: 'BebasNeue-Regular.ttf'
    id: font5
    size: 20
    
  - file: 'BebasNeue-Regular.ttf'
    id: font6
    size: 32

image:
  - file: mdi:water-thermometer-outline
    id: warmwasser_icon
    resize: 30x30
  - file: mdi:solar-power-variant
    id: pv_icon
    resize: 30x30
  - file: mdi:sun-thermometer-outline
    id: outtemp_icon
    resize: 30x30
  - file: mdi:home-thermometer
    id: intemp_icon
    resize: 30x30

spi:
  clk_pin: GPIO40 # (SCK/CLK)
  mosi_pin: GPIO41 # (SDA)

display:
  - platform: ili9xxx
    model: ST7735
    id: my_display
    dc_pin: GPIO01
    cs_pin: GPIO48
    reset_pin: GPIO42
    invert_colors: false
    rotation: 90
    dimensions:
      height: 160
      width: 128
      offset_width: 0
      offset_height: 0
    pages:
      # Boot Page
      - id: page_boot
        lambda: |-
          it.printf(64, 80, id(font32), TextAlign::CENTER, "One Moment");

      # Listen Page
      - id: page_listen
        lambda: |-
          it.printf(64, 80, id(font32), TextAlign::CENTER, "Listening...");

      # TTS Show Info
      - id: page_tts
        lambda: |-
          // Function to break text into multiple lines
          auto wrap_text = [](std::string text, int max_chars_per_line) -> std::vector<std::string> {
              std::vector<std::string> lines;
              while (text.length() > max_chars_per_line) {
                  size_t space_pos = text.find_last_of(" ", max_chars_per_line);
                  if (space_pos == std::string::npos) space_pos = max_chars_per_line;
                  lines.push_back(text.substr(0, space_pos));
                  text = text.substr(space_pos + 1);
              }
              lines.push_back(text);
              return lines;
          };

          // Get speech input (STT) and response (TTS)
          std::string user_text = id(stt_display).state;
          std::string bot_text = id(tts_display).state;

          // Wrap text to fit display
          int max_chars = 16; // Adjust based on font size and display width
          auto user_lines = wrap_text(user_text, max_chars);
          auto bot_lines = wrap_text(bot_text, max_chars);

          // Print User Speech (STT)
          int y_offset = 10; // Start position for text
          for (const auto& line : user_lines) {
              it.printf(5, y_offset, id(font20), TextAlign::TOP_LEFT, "%s", line.c_str());
              y_offset += 15; // Move down for the next line
          }

          // Print Assistant Response (TTS)
          y_offset += 10; // Add some space between sections
          for (const auto& line : bot_lines) {
              it.printf(5, y_offset, id(font20), TextAlign::TOP_LEFT, "%s", line.c_str());
              y_offset += 15;
          }

      # Misc Pages
      - id: page1
        lambda: |-
          // Print time in HH:MM p format                              
          int hour = id(esptime).now().hour;
          int hour_12 = (hour % 12 == 0) ? 12 : hour % 12;  // Convert to 12-hour format
          std::string am_pm = id(esptime).now().strftime("%p");  // Get "AM" or "PM"
          std::transform(am_pm.begin(), am_pm.end(), am_pm.begin(), ::tolower); // Convert string to lowercase
          it.printf(65, 0, id(font70), TextAlign::TOP_CENTER, "%d:%02d ", hour_12, id(esptime).now().minute);
          it.printf(130, 0, id(font70), TextAlign::TOP_LEFT, "%c", am_pm[0]);  // Print 'a' or 'p'

          // Outside Temp
          it.image(0, 88, id(outtemp_icon));          
          // Print outside temperature (from homeassistant sensor)
          if (id(outside_temperature).has_state()) {
            it.printf(150, 72, id(font50), TextAlign::TOP_RIGHT, "%.1f°F", id(outside_temperature).state);
          }          
      - id: page2
        lambda: |-
          // Print time in HH:MM p format                              
          int hour = id(esptime).now().hour;
          int hour_12 = (hour % 12 == 0) ? 12 : hour % 12;  // Convert to 12-hour format
          std::string am_pm = id(esptime).now().strftime("%p");  // Get "AM" or "PM"
          char short_am_pm = am_pm[0] + 32;  // Convert 'A'/'P' to lowercase manually
          it.printf(65, 0, id(font70), TextAlign::TOP_CENTER, "%d:%02d ", hour_12, id(esptime).now().minute);
          it.printf(130, 0, id(font70), TextAlign::TOP_LEFT, "%c", short_am_pm);

          // Inside Temp
          it.image(0, 88, id(intemp_icon));          
          // Print outside temperature (from homeassistant sensor)
          if (id(inside_temperature).has_state()) {
            it.printf(150, 72, id(font50), TextAlign::TOP_RIGHT, "%.1f°F", id(inside_temperature).state);
          }          

# Define the GPIO pin for controlling the display backlight
output:
  - platform: gpio
    id: display_backlight_output   # Ă„ndere den id-Namen fĂĽr das output
    pin: GPIO02  # Ă„ndere den Pin auf den richtigen fĂĽr dein Display

# Define a switch to control the backlight
switch:
  - platform: output
    name: "Display Backlight"
    id: display_backlight_switch   # Ă„ndere den id-Namen fĂĽr den switch
    output: display_backlight_output
    restore_mode: ALWAYS_ON
    
# Screen Flow
# Make sure to below define the DisplayPage vector for the pages: above you want
# toggled whent here is no TTS happening.
interval:
  - interval: 5s
    then:
      # Show page_tts when there's speech detected (STT or TTS is not empty)
      - if:
          condition:
            lambda: "return !id(stt_display).state.empty() || !id(tts_display).state.empty();"
          then:
            - display.page.show: page_tts  
            - component.update: my_display
            - delay: 15s  # Keep page_tts visible for 15 seconds
            - text_sensor.template.publish:
                id: stt_display
                state: ""  # Clear STT display
            - text_sensor.template.publish:
                id: tts_display
                state: ""  # Clear TTS display

      # If there is no speech, cycle through the display pages in a specific order
      - if:
          condition:
            lambda: "return id(stt_display).state.empty() && id(tts_display).state.empty();"
          then:
            - lambda: |-
                // Get the current active page
                auto current_page = id(my_display).get_active_page();
                
                // List of pages to cycle through, in order
                std::vector<esphome::display::DisplayPage *> pages = {id(page1), id(page2)};
                
                // Default to the first page (page1) if current_page is not found in the list
                esphome::display::DisplayPage *next_page = pages[0];

                // Find the next page in the sequence
                for (size_t i = 0; i < pages.size(); i++) {
                    if (pages[i] == current_page) {
                        next_page = (i + 1 < pages.size()) ? pages[i + 1] : pages[0];  // Loop back to page1 if at last page
                        break;
                    }
                }

                // Set the display to the next page in the cycle
                id(my_display).show_page(next_page);
                id(my_display).update();

1 Like

I also modified the STL files to make holes for the speaker (bottom and case), as well as a way to hold the screen cover without glue.

I used this screen from Amazon US

With this pin config:

+-------------+-----------------+----------------------+----------------------------+
| ST7735 Pin  | ESP32 Pin       | ESP32 Board Label   | Description                |
+-------------+-----------------+----------------------+----------------------------+
| GND         | GND (IO47)      | IO47 (Top-left )    | Ground                     |
| VCC         | 3.3V            | 3.3V                 | Power                     |
| SCL (SCK)   | GPIO40          | IO40                 | SPI Clock (SCK)           |
| SDA (MOSI)  | GPIO41          | IO41                 | SPI Data (MOSI)           |
| RES         | GPIO42          | IO42                 | Reset                     |
| DC          | GPIO01          | IO01                 | Data/Command              |
| CS          | GPIO48          | IO48                 | Chip Select               |
| BL          | GPIO02          | IO02                 | Backlight Control (On/Off)|
+-------------+-----------------+----------------------+----------------------------+