ESPHome API vs MQTT for Deep Sleep in 2022

Goal: write a function to measure connection time for API and MQTT.

Hi,

I was following advice given to me on a post I made a while ago when I was stuck making MQTT work.
ESPHome Can’t Prevent Deep Sleep
The advice was to use API instead of MQTT because API has had some recent improvements that make MQTT less necessary. I’ve seen posts about MQTT being a security issue, and I did find it to be a bit of a mess around. I’m pretty keen to get the API solution working, but when I converted my code to use API, I still found that heaps of data was missed. My config may have been bad, but I also don’t have a way to measure the difference in awake time between API and MQTT.

How would I go about writing a lambda function (or YAML if that’s possible) to measure the time between waking up from deep sleep and connecting to the Home Assistant API or MQTT broker?

4 Likes

I’ve gotten as far as this.

esphome:
  ....
  on_loop:
    - lambda: |-
        Somehow check time taken to connect to API or MQTT.
        id(connection_time).publish_state( WHATEVER WAS MEASURED );

sensor:
  - platform: template
    name: "Connection Time"
    id: connection_time
    update_interval: never

  - platform: all the other sensors
    ....

Thanks anan,

I managed to get something to work which I’ll post in a sec.

But I couldn’t get the MQTT function on_connect to work since I kept getting the [on_connect] is an invalid option for [mqtt] error. Instead, I targetted the MQTT id within a lambda.

I couldn’t get the api.connected command to work inside a lambda function so I did something similar to what I did for WIFI and MQTT.

I got impossible results. Can somebody please help me fix this?

MQTT connects just behind WIFI, somewhere around 4s. API connects somewhere between 100ms and 200ms which can’t be right. MQTT and API should connect after WIFI, but somehow API has a significantly faster connection time than WIFI. This doesn’t make any sense.
I’ve also only got run_duration set to 2s so how are the WIFI and MQTT connection times near 4s?

My code:
EDIT: I found that if (id(api_client).is_connected() && !id(last_api_state)) { was missing the .is_connected() part, but now that this is fixed, MQTT and API aren’t reporting any connection time, but WIFI is still around 4s. I’ve adjusted run_duration to 10s, but still getting nothing.

substitutions:
  # Device Properties
  devicename: dev-air-quality
  friendly_name: Dev-Air-Quality
  device_topic: AirQuality
  device_sleephelper_topic: admin/esphome_disable_deep_sleep
  
  # Sleep Properties
  sleep_time: 10s #2min
  run_time: 2s #1s
  global_update_interval: $sleep_time
  
  # Sensor Properties
  #temperature_offset: -3.0

  # Pinout
  # Wake        D0  # GPIO16
  pin_i2c_scl:  D1  # GPIO5
  pin_i2c_sda:  D2  # GPIO4
  pin_rx:       D5  # GPIO14
  pin_tx:       D6  # GPIO12
  pin_dht:      D7  # GPIO13

globals:
  - id: last_wifi_state
    type: bool
    restore_value: no
    initial_value: 'false'
  - id: last_mqtt_state
    type: bool
    restore_value: no
    initial_value: 'false'
  - id: last_api_state
    type: bool
    restore_value: no
    initial_value: 'false'

esphome:
  name: $devicename
  on_loop:
    # Measure WIFI connection time.
    # Measure MQTT connection time. Attempted to use the on_connect option in the MQTT section, but couldn't get it to work.
    # Measure API connection time. Attempted to use the api.connected option but couldn't get it to work within a lambda.
    - lambda: |-
        unsigned long t = millis();
        
        if (id(wifi_client).is_connected() && !id(last_wifi_state)) {
          id(wifi_connection_time).publish_state(t);
          id(last_wifi_state) = true;
        }
        
        if (id(mqtt_client).is_connected() && !id(last_mqtt_state)) {
          id(mqtt_connection_time).publish_state(t);
          id(last_mqtt_state) = true;
        }
        
        if (id(api_client).is_connected() && !id(last_api_state)) {
          id(api_connection_time).publish_state(t);
          id(last_api_state) = true;
        }

esp8266:
  board: d1_mini

# Enable logging
logger:

# Enable Over-The-Air flashing
ota:
  #safe_mode: True
  password: !secret ota_password

deep_sleep:
  id: deep_sleep_control
  sleep_duration: $sleep_time
  run_duration: $run_time

wifi:
  id: wifi_client
  power_save_mode: LIGHT #There are other options. Just checking if this helps with self heating.
  fast_connect: true # Skip wifi scan to save time.
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  # Set a static IP address.
  manual_ip:
    static_ip: 192.168.86.200
    gateway: 192.168.86.1
    subnet: 255.255.255.0
  use_address: "192.168.86.200"

# Enable Home Assistant API
api:
  id: api_client
  password: !secret api_password

# Enable MQTT
mqtt:
  id: mqtt_client
  broker: !secret mqtt_broker
  port: !secret mqtt_port
  username: !secret mqtt_user
  password: !secret mqtt_password
  discovery: true
  discovery_retain: true
  # Set empty birth and will messages to avoid showing unavailable during sleep
  birth_message:
  will_message:
  topic_prefix: $device_topic
  on_message:
    - topic: $device_sleephelper_topic
      qos: 0                                          #1
      payload: 'on'
      then:
        - logger.log:
            format: 'MQTT Admin: disable deep sleep'
            level: ERROR
        - deep_sleep.prevent: deep_sleep_control
    - topic: $device_sleephelper_topic
      qos: 0                                          #1
      payload: 'off'
      then:
        - logger.log:
            format: 'MQTT Admin: entering deep sleep'
            level: ERROR
        - deep_sleep.enter: deep_sleep_control


i2c:
  sda: $pin_i2c_sda
  scl: $pin_i2c_scl
  id: bus_i2c
  scan: False       # Save time by skipping the I2C scan. All addresses are known.
  frequency: 50kHz

# Air Quality, Temperature and Humidity.
sensor:
  # WIFI Connection Time
  - platform: template
    name: "$friendly_name WIFI Connection Time"
    id: wifi_connection_time
    update_interval: never
    unit_of_measurement: ms
  # MQTT Connection Time
  - platform: template
    name: "$friendly_name MQTT Connection Time"
    id: mqtt_connection_time
    update_interval: never
    unit_of_measurement: ms
  # API Connection Time
  - platform: template
    name: "$friendly_name API Connection Time"
    id: api_connection_time
    update_interval: never
    unit_of_measurement: ms

  # Sensor uptime
  # The uptime platform doesn't do what I thought it did. Always reports 0.
  #- platform: uptime
  #  name: "$friendly_name Uptime"
  #  id: device_uptime
    
  # DHT11 Temperature and Humidity Sensor
  - platform: dht
    model: DHT22 #DHT11
    pin: $pin_dht
    temperature:
      name: "$friendly_name Temperature"
      id: temperature
      #filters: # Only needed because the DHT temperature sensor in the bedroom is too close to the ESP8266.
      #  - offset: $temperature_offset
    humidity:
      name: "$friendly_name Humidity"
      id: humidity
    update_interval: $global_update_interval
    
  # CO2, Temperature and Humidity
  - platform: scd30
    i2c_id: bus_i2c
    address: 0x61
    automatic_self_calibration: true
    update_interval: $global_update_interval
    co2:
      name: "$friendly_name CO2"
      id: co2
      accuracy_decimals: 1
    temperature:
      name: "$friendly_name Temperature SCD"
      id: temperature_scd
      accuracy_decimals: 2
      filters:
        - offset: 0
    humidity:
      name: "$friendly_name Humidity SCD"
      id: humidity_scd
      accuracy_decimals: 1

I think your run_time doesn’t start until priority zero is reached - which is after the network is connected as well as API or MQTT.

Thanks Daryl

After fixing my code (I edited the code in the comment I’m replying to), I couldn’t get any data reported so I decided to move to a fresh ESP8266 that only checks connection time.

This should work better than my original code which was trying to mess around with other sensors.
This code is meant to achieve three things.

  • It records the time in milliseconds when each connection is made (WIFI, MQTT and API).
    Then if all three times are obtained, it publishes the data and enters deep sleep.
  • If 30,000ms is reached, then it publishes the times it has obtained, logs which times were recorded, and then still enters deep sleep.
  • Publish the number of connections (WIFI, MQTT and API) that didn’t record a value and publish that.

The problem is that most of the time nothing gets published. I can see in the Serial logs that times for WIFI and MQTT are being recorded, but API is mostly getting nothing. API did get a value one time which resulted in data being published.

  • If WIFI, MQTT and API record a connection time, all three times are published and timeout_count publishes with a count of 0.
  • If WIFI and MQTT record a connection time but API does not, nothing is published. My code should reach 30s and then publish WIFI and MQTT times, a timeout_count of 1, and a log message saying “API failed to connect”. But nothing happens and it doesn’t seem to be waiting 30s to enter deep sleep.
  • If nothing connects, my code should reach 30s, publish no times, publish a timeout_count of 3, and 3 log messages saying each of the connections failed to connect. But nothing happens and it doesn’t seem to be waiting 30s to enter deep sleep.
    Can somebody please help me fix this?

Edit: Solved. See my next post for updated code and results.

My Code

substitutions:
  # Device Properties
  devicename: dev-connection-time
  friendly_name: Dev-Connection-Time
  device_topic: ConnectionTime
  device_sleephelper_topic: admin/esphome_disable_deep_sleep
  
  # Logger Properties
  log_tag: yaml
  
  # Sleep Properties
  sleep_time: 10s #1min
  # In this case, run_duration will act as a time out. Other conditions should start deep sleep sooner.
  #run_time: 1min


globals:
  - id: wifi_last_state
    type: bool
    restore_value: no
    initial_value: 'false'
  - id: wifi_time
    type: unsigned long
    restore_value: no
    initial_value: '0'
  - id: mqtt_last_state
    type: bool
    restore_value: no
    initial_value: 'false'
  - id: mqtt_time
    type: unsigned long
    restore_value: no
    initial_value: '0'
  - id: api_last_state
    type: bool
    restore_value: no
    initial_value: 'false'
  - id: api_time
    type: unsigned long
    restore_value: no
    initial_value: '0'
  - id: timeout_count_value
    type: int
    restore_value: no
    initial_value: '0'


esphome:
  name: $devicename
  
  on_boot:
    then:
      - logger.log:
          format: "Booted"
          level: ERROR

  on_loop:
    - lambda: |-
        unsigned long t = millis();
        
        
        if (id(wifi_client).is_connected() && !id(wifi_last_state)) {
          id(wifi_time) = t;
          id(wifi_last_state) = true;
          ESP_LOGE("${log_tag}", "WIFI connection time of %lu ms recorded", t);
        }
        if (id(mqtt_client).is_connected() && !id(mqtt_last_state)) {
          id(mqtt_time) = t;
          id(mqtt_last_state) = true;
          ESP_LOGE("${log_tag}", "MQTT connection time of %lu ms recorded", t);
        }
        if (id(api_client).is_connected() && !id(api_last_state)) {
          id(api_time) = t;
          id(mqtt_last_state) = true;
          ESP_LOGE("${log_tag}", "API connection time of %lu ms recorded", t);
        }
        
        
        if (millis() > 30000) {
          if (id(wifi_time) != 0) {
            id(wifi_connection_time).publish_state(id(wifi_time));
          }
          else {
            id(timeout_count_value)++;
            ESP_LOGE("${log_tag}", "WIFI failed to connect");
          }
          if (id(mqtt_time) != 0) {
            id(mqtt_connection_time).publish_state(id(mqtt_time));
          }
          else {
            id(timeout_count_value)++;
            ESP_LOGE("${log_tag}", "MQTT failed to connect");
          }
          if (id(api_time) != 0) {
            id(api_connection_time).publish_state(id(api_time));
          }
          else {
            id(timeout_count_value)++;
            ESP_LOGE("${log_tag}", "API failed to connect");
          }
        id(timeout_count).publish_state(id(timeout_count_value));
        ESP_LOGE("${log_tag}", "Entering deep sleep because of time out", t);
        delay(1000);
        id(deep_sleep_control).begin_sleep();
        }
        
        
        if (id(wifi_time) != 0) {
          if (id(mqtt_time) != 0) {
            if (id(api_time) != 0) {
              id(wifi_connection_time).publish_state(id(wifi_time));
              id(mqtt_connection_time).publish_state(id(mqtt_time));
              id(api_connection_time).publish_state(id(api_time));
              id(timeout_count).publish_state(id(timeout_count_value));
              ESP_LOGE("${log_tag}", "All data published", t);
              delay(1000);
              id(deep_sleep_control).begin_sleep();
            }
          }
        }


esp8266:
  board: d1_mini

# Enable logging
logger:
  level: INFO #WARN
  # Publish logs to MQTT
  on_message:
    level: WARN
    then:
      - mqtt.publish:
          topic: $device_topic/logger
          payload: !lambda |-
            return "Level: " + to_string(level) + " ||| Tag: " + tag + " ||| Message: " + message;

# Enable Over-The-Air flashing
ota:
  #safe_mode: True
  password: !secret ota_password
  on_end:
    - logger.log:
        format: "OTA Update"
        level: ERROR

deep_sleep:
  id: deep_sleep_control
  sleep_duration: $sleep_time
  #run_duration: $run_time

wifi:
  id: wifi_client
  power_save_mode: LIGHT
  fast_connect: true # Skip wifi scan to save time.
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  
  # Set a static IP address.
  manual_ip:
    static_ip: 192.168.86.201
    gateway: 192.168.86.1
    subnet: 255.255.255.0
  # If changing the name or the IP address of the device, this provides the old address so that ESPHome knows which device to overwrite.
  use_address: "192.168.86.201"

# Enable Home Assistant API
api:
  id: api_client
  password: !secret api_password

# Enable MQTT
mqtt:
  id: mqtt_client
  broker: !secret mqtt_broker
  port: !secret mqtt_port
  username: !secret mqtt_user
  password: !secret mqtt_password
  discovery: true
  discovery_retain: true
  # Set empty birth and will messages to avoid showing unavailable during sleep
  birth_message:
  will_message:
  topic_prefix: $device_topic
  on_message:
    - topic: $device_sleephelper_topic
      qos: 0        #1
      payload: 'on'
      then:
        - deep_sleep.prevent: deep_sleep_control
    - topic: $device_sleephelper_topic
      qos: 0        #1
      payload: 'off'
      then:
        - deep_sleep.enter: deep_sleep_control

sensor:
  # WIFI Connection Time
  - platform: template
    name: "$friendly_name WIFI Connection Time"
    id: wifi_connection_time
    update_interval: never
    unit_of_measurement: ms
    
  # MQTT Connection Time
  - platform: template
    name: "$friendly_name MQTT Connection Time"
    id: mqtt_connection_time
    update_interval: never
    unit_of_measurement: ms
    
  # API Connection Time
  - platform: template
    name: "$friendly_name API Connection Time"
    id: api_connection_time
    update_interval: never
    unit_of_measurement: ms
    
  # Error Count
  - platform: template
    name: "$friendly_name Time Out Error Count"
    id: timeout_count
    update_interval: never
    unit_of_measurement: timeouts

Serial logs

[22:45:53][I][deep_sleep:103]: Beginning Deep Sleep
[22:45:53][W][wifi_esp8266:483]: Event: Disconnected ssid=[redacted] bssid=[redacted] reason='Association Leave'
[22:46:03]rl 1[***gibberish***][I][logger:214]: Log initialized
[22:46:03][I][app:029]: Running through setup()...
[22:46:03][E][main:155]: Booted
[22:46:03][I][wifi:245]: WiFi Connecting to 'Ghostwater'...
[22:46:07][I][wifi:502]: WiFi Connected!
[22:46:07][E][yaml:071]: WIFI connection time of 3852 ms recorded
[22:46:07][I][mqtt:175]: Connecting to MQTT...
[22:46:07][I][mqtt:215]: MQTT Connected!
[22:46:07][I][app:062]: setup() finished successfully!
[22:46:07][E][yaml:076]: MQTT connection time of 4014 ms recorded
[22:46:07][I][app:102]: ESPHome version 2022.1.3 compiled on Jul 17 2022, 22:41:51
[22:46:07][I][deep_sleep:103]: Beginning Deep Sleep
[22:46:07][W][wifi_esp8266:483]: Event: Disconnected ssid=[redacted] bssid=[redacted] reason='Association Leave'
[22:46:17]rl 1[***gibberish***][I][logger:214]: Log initialized
[22:46:17][I][app:029]: Running through setup()...
[22:46:17][E][main:155]: Booted
[22:46:17][I][wifi:245]: WiFi Connecting to 'Ghostwater'...
[22:46:21][I][wifi:502]: WiFi Connected!
[22:46:21][E][yaml:071]: WIFI connection time of 3860 ms recorded
[22:46:21][I][mqtt:175]: Connecting to MQTT...
[22:46:21][I][mqtt:215]: MQTT Connected!
[22:46:21][I][app:062]: setup() finished successfully!
[22:46:21][E][yaml:076]: MQTT connection time of 4008 ms recorded
[22:46:21][I][app:102]: ESPHome version 2022.1.3 compiled on Jul 17 2022, 22:41:51
[22:46:21][I][deep_sleep:103]: Beginning Deep Sleep
[22:46:21][W][wifi_esp8266:483]: Event: Disconnected ssid=[redacted] bssid=[redacted] reason='Association Leave'
[22:46:31]rl l[***gibberish***][I][logger:214]: Log initialized
[22:46:31][I][app:029]: Running through setup()...
[22:46:31][E][main:155]: Booted
[22:46:31][I][wifi:245]: WiFi Connecting to 'Ghostwater'...
[22:46:35][I][wifi:502]: WiFi Connected!
[22:46:35][E][yaml:071]: WIFI connection time of 3854 ms recorded
[22:46:35][I][mqtt:175]: Connecting to MQTT...
[22:46:35][I][mqtt:215]: MQTT Connected!
[22:46:35][I][app:062]: setup() finished successfully!
[22:46:35][E][yaml:076]: MQTT connection time of 4006 ms recorded
[22:46:35][I][app:102]: ESPHome version 2022.1.3 compiled on Jul 17 2022, 22:41:51
[22:46:35][E][yaml:081]: API connection time of 4027 ms recorded
[22:46:35][E][yaml:120]: All data published
[22:46:36][I][deep_sleep:103]: Beginning Deep Sleep
[22:46:36][W][wifi_esp8266:483]: Event: Disconnected ssid=[redacted] bssid=[redacted] reason='Association Leave'
[22:46:46]rl 1[***gibberish***][I][logger:214]: Log initialized
[22:46:46][I][app:029]: Running through setup()...
[22:46:46][E][main:155]: Booted
[22:46:46][I][wifi:245]: WiFi Connecting to 'Ghostwater'...
[22:46:49][I][wifi:502]: WiFi Connected!
[22:46:49][E][yaml:071]: WIFI connection time of 3858 ms recorded
[22:46:49][I][mqtt:175]: Connecting to MQTT...
[22:46:49][I][mqtt:215]: MQTT Connected!
[22:46:50][I][app:062]: setup() finished successfully!
[22:46:50][E][yaml:076]: MQTT connection time of 4013 ms recorded
[22:46:50][I][app:102]: ESPHome version 2022.1.3 compiled on Jul 17 2022, 22:41:51
[22:46:50][I][deep_sleep:103]: Beginning Deep Sleep
[22:46:50][W][wifi_esp8266:483]: Event: Disconnected ssid=[redacted] bssid=[redacted] reason='Association Leave'
[22:47:00]rl 1[***gibberish***][I][logger:214]: Log initialized
[22:47:00][I][app:029]: Running through setup()...
[22:47:00][E][main:155]: Booted
[22:47:00][I][wifi:245]: WiFi Connecting to 'Ghostwater'...
[22:47:03][I][wifi:502]: WiFi Connected!
[22:47:03][E][yaml:071]: WIFI connection time of 3862 ms recorded
[22:47:03][I][mqtt:175]: Connecting to MQTT...
[22:47:03][I][mqtt:215]: MQTT Connected!
[22:47:04][I][app:062]: setup() finished successfully!
[22:47:04][E][yaml:076]: MQTT connection time of 4019 ms recorded
[22:47:04][I][app:102]: ESPHome version 2022.1.3 compiled on Jul 17 2022, 22:41:51
[22:47:04][I][deep_sleep:103]: Beginning Deep Sleep
[22:47:04][W][wifi_esp8266:483]: Event: Disconnected ssid=[redacted] bssid=[redacted] reason='Association Leave'
[22:47:13]rl 1[***gibberish***][I][logger:214]: Log initialized
[22:47:13][I][app:029]: Running through setup()...
[22:47:13][E][main:155]: Booted
[22:47:13][I][wifi:245]: WiFi Connecting to 'Ghostwater'...
[22:47:17][I][wifi:502]: WiFi Connected!
[22:47:17][E][yaml:071]: WIFI connection time of 3851 ms recorded
[22:47:17][I][mqtt:175]: Connecting to MQTT...
[22:47:17][I][mqtt:215]: MQTT Connected!
[22:47:17][I][app:062]: setup() finished successfully!
[22:47:17][E][yaml:076]: MQTT connection time of 4009 ms recorded
[22:47:17][I][app:102]: ESPHome version 2022.1.3 compiled on Jul 17 2022, 22:41:51
[22:47:17][I][deep_sleep:103]: Beginning Deep Sleep

I’ve finished my testing (probably) and I have reliable results (also probably).

Note that error count resets to 0 when the ESP boots, and increments for each connection that fails. If error count stays at 0 the entire time, then WIFI, MQTT and API all connected. If error count returns a 1, then one of the 3 failed at time out, and if it returns a 2, then 2 of the 3 failed at time out.

Results
MQTT consistently connects 100ms-300ms behind WIFI.
However, API takes significantly longer and sits around the 57s mark.
It’s also possible that my test isn’t very good and I’m not giving API a fair chance.
I hope that one day I will be able to remove MQTT, but for now, that’s what I’m going to use for deep sleep applications.

Test Settings

  • Timeout: 90s
  • Sleep Time: 10s

Versions

  • Home Assistant Core 2022.7.5
  • Home Asssistant OS 8.2
  • ESPHome 2022.1.3

Notes for further testing
I’m curious to see if changing sleep time has an effect on connection time. When I had timeout set to 30s, half the time API would connect at around 18s, and the other half of the time, API would fail to connect and this test would time out.
In the test with a 90s time out, the error count stays at 0. During the earlier tests with the 30s time out, API often failed to connect and error count would change to 1.

A connection time of 0ms is not possible if WIFI connects at roughly 4s. If MQTT or API have a connection time of 0ms, they timed out. I just couldn’t figure out how to publish an empty data point and break up the line in the plot.

My code

substitutions:
  # Device Properties
  devicename: dev-connection-time
  friendly_name: Dev-Connection-Time
  device_topic: ConnectionTime
  device_sleephelper_topic: admin/esphome_disable_deep_sleep
  
  # Logger Properties
  log_tag: yaml
  
  # Sleep Properties
  sleep_time: 10s #1min
  #run_time: 1min #run_duration has been replaced by a script.


globals:
  # WIFI connection time and state
  - id: wifi_last_state
    type: bool
    restore_value: no
    initial_value: 'false'
  - id: wifi_time
    type: unsigned long
    restore_value: no
    initial_value: '0'
    
  # MQTT connection time and state
  - id: mqtt_last_state
    type: bool
    restore_value: no
    initial_value: 'false'
  - id: mqtt_time
    type: unsigned long
    restore_value: no
    initial_value: '0'
    
  # API connection time and state
  - id: api_last_state
    type: bool
    restore_value: no
    initial_value: 'false'
  - id: api_time
    type: unsigned long
    restore_value: no
    initial_value: '0'
    
  # Error counter
  - id: timeout_count_value
    type: int
    restore_value: no
    initial_value: '0'
    
  # Time out in ms
  - id: timeout
    type: int
    restore_value: no
    initial_value: '90000'
    
  # reason for shutting down.
  - id: shutdown_reason
    type: std::string
    restore_value: no
    initial_value: '"run_duration expired"'


esphome:
  name: $devicename
  
  on_boot:
    then:
      - logger.log:
          format: "Booted"
          level: ERROR

  on_loop:
    - if:
        condition:
          # If timeout hasn't been reached.
          - lambda: "return millis() < id(timeout);"
        then:
          # Check and log the time each connection is made.
          - lambda: |-
              unsigned long t = millis();
              
              if (id(wifi_client).is_connected() && !id(wifi_last_state)) {
                id(wifi_time) = t;
                id(wifi_last_state) = true;
                ESP_LOGE("${log_tag}", "WIFI connection time of %lu ms recorded", t);
              }
              
              if (id(mqtt_client).is_connected() && !id(mqtt_last_state)) {
                id(mqtt_time) = t;
                id(mqtt_last_state) = true;
                ESP_LOGE("${log_tag}", "MQTT connection time of %lu ms recorded", t);
              }
              
              if (id(api_client).is_connected() && !id(api_last_state)) {
                id(api_time) = t;
                id(api_last_state) = true;
                ESP_LOGE("${log_tag}", "API connection time of %lu ms recorded", t);
              }
          # If all connections are successful before timing out, publish data and enter deep sleep.
          - lambda: |-
              if (id(wifi_last_state) && id(mqtt_last_state) && id(api_last_state)) {
                id(shutdown_reason) = "All Connections Successful";
                id(start_deep_sleep).execute();
              }
        else:
          # If the timeout has been reached, increment the error counter by 1 for each failed connection.
          - lambda: |-
              if (id(wifi_time) == 0) {
                id(timeout_count_value)++;
                ESP_LOGE("${log_tag}", "WIFI failed to connect");
              }
              
              if (id(mqtt_time) == 0) {
                id(timeout_count_value)++;
                ESP_LOGE("${log_tag}", "MQTT failed to connect");
              }
              
              if (id(api_time) == 0) {
                id(timeout_count_value)++;
                ESP_LOGE("${log_tag}", "API failed to connect");
              }
              
              id(shutdown_reason) = "Timeout";
          # Publish data and enter deep sleep.
          - script.execute: start_deep_sleep

  on_shutdown:
    - lambda: |-
        ESP_LOGE("${log_tag}", "Entering Deep Sleep at %lu ms, Reason: %s", millis(), id(shutdown_reason).c_str());


esp8266:
  board: d1_mini

# Enable logging
logger:
  level: INFO #WARN
  # Publish logs to MQTT
  on_message:
    level: WARN
    then:
      - mqtt.publish:
          topic: $device_topic/logger
          payload: !lambda |-
            return "Level: " + to_string(level) + " ||| Tag: " + tag + " ||| Message: " + message;

# Enable Over-The-Air flashing
ota:
  #safe_mode: True
  password: !secret ota_password
  on_end:
    - logger.log:
        format: "OTA Update"
        level: ERROR

deep_sleep:
  id: deep_sleep_control
  sleep_duration: $sleep_time
  #run_duration: $run_time

wifi:
  id: wifi_client
  power_save_mode: LIGHT #There are other options. Just checking if this helps with self heating.
  fast_connect: true # Skip wifi scan to save time.
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  
  # Set a static IP address.
  manual_ip:
    static_ip: 192.168.86.201
    gateway: 192.168.86.1
    subnet: 255.255.255.0
  # If changing the name or the IP address of the device, this provides the old address so that ESPHome knows which device to overwrite.
  use_address: "192.168.86.201"

# Enable Home Assistant API
api:
  id: api_client
  password: !secret api_password

# Enable MQTT
mqtt:
  id: mqtt_client
  broker: !secret mqtt_broker
  port: !secret mqtt_port
  username: !secret mqtt_user
  password: !secret mqtt_password
  discovery: true
  discovery_retain: true
  # Set empty birth and will messages to avoid showing unavailable during sleep
  birth_message:
  will_message:
  topic_prefix: $device_topic
  
# It is not ok to enter deep sleep as soon as this device realises the HA helper for OTA mode is set to False.
# I really need to get deep_sleep.allow to work so that it can replace deep_sleep.enter in the on_message: 'off' section.
# Disable for now, cry for help later.
#  on_message:
#    - topic: $device_sleephelper_topic
#      qos: 0                                          #1
#      payload: 'on'
#      then:
#        - logger.log:
#            format: 'MQTT Admin: disable deep sleep'
#            level: ERROR
#        - deep_sleep.prevent: deep_sleep_control
#    - topic: $device_sleephelper_topic
#      qos: 0                                          #1
#      payload: 'off'
#      then:
#        - logger.log:
#            format: 'MQTT Admin: entering deep sleep'
#            level: ERROR
#        - deep_sleep.enter: deep_sleep_control

sensor:
  # WIFI Connection Time
  - platform: template
    name: "$friendly_name WIFI Connection Time"
    id: wifi_connection_time
    update_interval: never
    unit_of_measurement: ms
    
  # MQTT Connection Time
  - platform: template
    name: "$friendly_name MQTT Connection Time"
    id: mqtt_connection_time
    update_interval: never
    unit_of_measurement: ms
    
  # API Connection Time
  - platform: template
    name: "$friendly_name API Connection Time"
    id: api_connection_time
    update_interval: never
    unit_of_measurement: ms
    
  # Time Out Error Count
  - platform: template
    name: "$friendly_name Time Out Error Count"
    id: timeout_count
    update_interval: never
    unit_of_measurement: timeouts

script:
  # Script to publish data and enter deep sleep.
  - id: start_deep_sleep
    then:
      - lambda: |-
          id(wifi_connection_time).publish_state(id(wifi_time));
          id(mqtt_connection_time).publish_state(id(mqtt_time));
          id(api_connection_time).publish_state(id(api_time));
          id(timeout_count).publish_state(id(timeout_count_value));
          delay(3000);
          id(deep_sleep_control).begin_sleep();
4 Likes

Thanks burner66,

I didn’t realise I still had the community repo and I was trying to figure out why updates weren’t detected.

I updated to ESPHome 2022.6.2 and rerun the test overnight. See below.

My code is the same as my previous post, but I did update the firmware on my device after updating ESPHome.

Results
MQTT consistently connects 100ms-300ms behind WIFI, with a small handful of exceptions.
API still took significantly longer but did have some minor improvements, connecting between 47s and 57s. API also had 1 failed attempt to connect since it hadn’t connected when the time out of 90s was reached. I don’t think this is because of the update because this test ran a lot longer than the previous test and had a higher chance of timing out. MQTT also took a lot longer during this specific cycle and connect at around 74s when WIFI was still sitting at 3.8s.
I hope that one day I will be able to remove MQTT, but for now, that’s what I’m going to use for deep sleep applications.

Test Settings

  • Timeout: 90s
  • Sleep Time: 10s

Versions

  • Home Assistant Core 2022.7.5
  • Home Asssistant OS 8.2
  • ESPHome 2022.6.2

Hardware

  • Raspberry Pi 4

I’m still curious to see how sleep time affects the API connection time, but I’ll have to run that test later. And also how wifi power saving affects it.

Sorry that my x-axes don’t line up. Hopefully it’s obvious which connection times that error lines up with.

Sleep time does make a difference.

These results show sleep time changing from 10s to 2min, and an unintentional power outage (sorry).
API definitely improved, but not enough to catch up to MQTT.
I have no idea why MQTT became less reliable, but I’ll keep it running and see what happens.

1 Like

For deep sleep applications native API is discouraged in favour of MQTT. The details are in this GitHub issue: https://github.com/esphome/feature-requests/issues/46.

Thanks Luca. That’s a good resource and I appreciate it. But did you read my original post? This topic is about testing the difference and seeing how much API has improved.

1 Like

Thanks anon. I’d definitely be keen to add that to the list of tests. I will need to tweak my code to measure the time between leaving deep sleep and when data is sent to HA instead of when a connection is established. I probably won’t be able to start that this week, but hopefully soon.

1 Like

I found this really interesting - I am trying to speed up connection time for a temperature sensor outside and based on your initial results I switched to MQTT.

I have not tried to replicate your test but just when I look at the speed of data coming in, I find very little difference between the reporting frequency of the same data when looking at the “MQTT sensor” and the “API sensor”. On reporting battery life, I am getting a few less odd reports with the API. The MQTT seems to throw up 0% more often (the battery is actually say at 70%). I have tried many variations of filters to get rid of the 0% data points, without success.

For a battery powered device which wakes up you could do this with a little script, something like

  • set update interval to :zero: for adc (to don’t update itself)
  • Have a script which updates the adc component and then sends the value (when not :zero:) via mqtt (or webhook for the ones who turned of their broker already :muscle:)

Definitely, very nice work @Mortalitas :+1:

Would really also like to see how webhooks perform in this area. Nice thing with them is that it’s even possible to trigger something on a esphome node (from another esphome node) without ha (not that this is a use case for this experiments - but it’s anyways nice to know :stuck_out_tongue:)

I need to get into webhooks - but haven’t got my head around that concept yet.

On the zero data I have just tried multiple variations of the filter and maybe have one that works for me.

      - name: "FB01 Battery Percentage"
        accuracy_decimals: 1
        unit_of_measurement: '%'
        filters:
          - max:
             window_size: 2
             send_every: 2
             send_first_at: 2

So far after 30 minutes of “wake for 5 seconds every 2 minutes” I am getting a much more stable report. It will still vary by a couple of percent, but much better than a 60-80% variation. If it remains stable for an hour, I will throw a capacitor on the circuit next to see if that helps.

I am using a DFRobot Fuel Gauge and assuming I have wired it correctly. (Their diagrams don’t really give great guidance)

This is somewhat the “poor mans” version for deep sleep. Meaning it can (and should work) but typically wasting cycles ( = :battery: ) because the esp probably could go to sleep earlier (when “finished” it’s work).

The docs should have you covered, specially the deep_sleep.enter action could be useful to send the esphome node to sleep ones the job is done.

I was using short hand. It is deep sleep - and the two minutes is only for testing. When I am using it for real it will be deep-sleep for 20 minutes, wake for 5 seconds.

That’s why I was interested in the OP’s test. Would the API connect quick enough to make more battery savings or is the MQTT set-up faster overall? I use MQTT to put it to sleep and wake it, but there is an API version of that too.

Well if you look at the results from @Mortalitas tests it looks like that the native api (still) isn’t able to compete with mqtt (and probably webhooks) in terms of “first connection”. :satellite:

The reason is probably due to the design of the native api. The esphome node is the servers and needs to wait for ha (or any other client) connect via the native api to it. The latest improvements should speed up that process drastically (when the esphome nodes announces itself in the network via a mDNS broadcast ha should pick that up and immediately connect to it). :wavy_dash:

In general a encrypted WPAx wifi (and on top of it DHCP) is just not the right technology for battery operated devices. Most of the wake time will just be scanning for a AP, connecting to it (4-way handshake) followed maybe by DHCP (3-way handshake) till finally being able to communicate via TCP connection (3-way handshake to establish a connection). :handshake::handshake::handshake::handshake: :handshake::handshake::handshake: :handshake::handshake::handshake:

All this overhead (to be able to communicate over TCP) is probably magnitudes higher than the intended work or payload most battery operated devices probably should do. For example the time to be connected could be 2500ms while the task (reading sensor values and sending it) could be only 150ms :timer_clock:

Probably a more sophisticated way is to use something like esp-now instead and avoid the WPAx wifi completely. The “downside” of this solution that it needs a intermediate/gateway (spare esp should do) to bridge payloads into HA. :signal_strength:

https://yewtu.be/watch?v=iEkGRI8_txE

At the time of writing there (still) has no official esp-now component landed in esphome but more than one custom component is available, for example: :flying_saucer:

2 Likes

Thanks for that. It looks very interesting.

ESP-Now is something I have just started reading up on - and as I have a couple of spare ESP32s here, it could be a pathway. The new Zigbee boards might also be worth checking out because I have a couple of ZigBee sensors and they are very light on the battery.

I would like to find a way to make my current board (Firebeetle ESP32-E) stay in deep sleep from say 10pm to 8am and then go into the wake/sleep cycle of 5-seconds/20-minutes. That could save a lot of battery, making the API/MQTT debate redundant.

On battery life, I also have an ESP8266 in the Cricket package, using MQTT. It measures my pool temperature and runs off 2xAA batteries. So far it has lasted several months, reporting every 20 minutes.