Reporting sensors on a different interval than the local control loop on ESP32

ESPHome
1

I am having a lot of fun building custom monitoring and control devices using ESP32 and ESPhome. At first the basic approach was to set the update_interval to be what was needed for the control loop to be fast enough, or for the data to update fast enough for display purposes. However this had a negative side effect which is flooding HA with a lot of unnecessary data. In some cases it bogs down my HA instance even though it is running on a dedicated i7 with tons of RAM and SSD space. I started looking into ways to have my sensors and control loop run as fast as possible on the ESP but only report back what makes sense to HA in order to reduce the load and the data bloat.

Below is one YAML where my work is still in progress. One issue that I am running into is that the Room Temperature and Humidity shows up immediately in the logs but it takes at least 20 seconds (based on a few tests where I counted the time from everything else loading to the time those two sensors loaded) to go from 0 to the updated value. It is a minor issue as it only happens upon boot but I would like to fix it if possible.

In order to avoid reporting back all the sensor data to HA, I pump their values to global variables (not restored so no issue on flash) and use those to publish the data to HA on the interval that makes most sense. Also, the control part is not attached to the sensor value output (on_value) but in an interval so I can set the speed I want for both. It doesnā€™t really matter to me if the sensor updates faster or slower than the control loop (data will be not used or it will use the last value) as this process changes relatively slowly. My goal is just to run the control loop at any speed I want without that resulting in a massive amount of data being sent back to HA.

In this YAML, the conversion is not completeā€¦ meaning not all the values passed back to HA are flowing through variables and rate limited. Before I convert it all, I am trying to figure out why the values are not updating immediately in HA given the sensor spits the values out immediately. In order to eliminate the dashboard as a cause, I am looking at the integration page where you see the raw controls and sensors of the device.

At the very bottom of the YAML, where I have the code that publishes the sensor value to HA, I tried publishing the variables and the raw sensor output, but both options have the same result (delayed display).

Summary of what this YAML does: It controls the speed of a blower fan that sucks air away from the computer exhaust under my desk so that the hot air doesnā€™t roast me alive. I can override the speed of the blower, or let it do its thing. Another thing that I am implementing in this revision is that all my controls and sensors in HA are created when adding the ESPHome device. My first iteration required me to create helpers (I think I shared the older iteration on a thread I posted on this device I made).

substitutions:
  devicename: computer-exhaust-fan
  devicename_no_dashes: computer_exhaust_fan
  friendly_devicename: "Computer Exhaust Fan"
  device_description: "Computer Exhaust Fan"
  update_interval_s: "1s"
  update_interval_wifi: "120s"
  #temp_calibration: "-0.7" #The DS18B20 measures about 0.7C too high
  
esphome:
  name: ${devicename}
  comment: ${device_description}
  platform: ESP32
  board: esp32doit-devkit-v1
  on_boot:
     then:
      - lambda: |-
          id(setup_controls).execute();
          id(setup_sensors).execute();   

# Enable logging
logger:

# Enable Home Assistant API
api: 
  password: !secret api_pwd

ota:
  password: !secret ota_pwd

wifi:
  ssid: !secret iot_wifi_ssid
  password: !secret iot_wifi_password

#Faster than DHCP. Also use if can't reach because of name change
#  manual_ip:
#    static_ip: 192.168.3.195
#    gateway: 192.168.3.1
#    subnet: 255.255.255.0
#    dns1: 192.168.1.25
#    dns2: 192.168.1.26

#Manually override what address to use to connect to the ESP.
#Defaults to auto-generated value. Example, if you have changed your
#static IP and want to flash OTA to the previously configured IP address.
  use_address: 192.168.3.195
  
  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "${devicename}"
    password: !secret iot_wifi_password

web_server:
  port: 80
  include_internal: true

captive_portal:

# Sync time with Home Assistant
time:
  - platform: homeassistant
    id: ha_time

i2c:
  - id: bus_a
    sda: 32
    scl: 33
    scan: true

#Configuration entry for 18B20 sensor
dallas:
  - pin: 27
    update_interval: ${update_interval_s}

text_sensor:
  - platform: wifi_info
    ip_address:
      name: "${friendly_devicename}: IP"
      icon: "mdi:ip-outline"
      update_interval: ${update_interval_wifi}
    ssid:
      name: "${friendly_devicename}: SSID"
      icon: "mdi:wifi-settings"
      update_interval: ${update_interval_wifi}
    bssid:
      name: "${friendly_devicename}: BSSID"
      icon: "mdi:wifi-settings"
      update_interval: ${update_interval_wifi}
    mac_address:
      name: "${friendly_devicename}: MAC"
      icon: "mdi:network-outline"
    scan_results:
      name: "${friendly_devicename}: Wifi Scan"
      icon: "mdi:wifi-refresh"
      disabled_by_default: true

#https://esphome.io/guides/automations.html?highlight=restore_value#bonus-2-global-variables
globals: ##to set default reboot behavior
  # Variables to recall (saves them after 1 minute by default, can be changed)
  - id: computer_exhaust_fan
    type: bool
    restore_value: yes
    initial_value: "false"
  - id: override_fan_speed
    type: bool
    restore_value: yes
    initial_value: "false"
  - id: fan_speed_override
    type: float
    restore_value: yes
    initial_value: '0.0'
  - id: max_fan_speed_temperature
    type: float
    restore_value: yes
    initial_value: '35.0'

  - id: fan_speed
    type: float
    restore_value: no
    initial_value: '0'
  - id: computer_exhaust_temperature
    type: float
    restore_value: no
    initial_value: '0.0'
  - id: room_temperature
    type: float
    restore_value: no
    initial_value: '0.0'
  - id: room_humidity
    type: float
    restore_value: no
    initial_value: '0.0'

switch:
  - platform: restart
    name: "${friendly_devicename}: Restart"

  - platform: template
    name: "${friendly_devicename}"
    id: computer_exhaust_fan_ha
    icon: "mdi:fan"
    optimistic: true
    turn_on_action:
      - globals.set:
          id: computer_exhaust_fan
          value: 'true'
    turn_off_action:
      - globals.set:
          id: computer_exhaust_fan
          value: 'false'

  - platform: template
    name: "${friendly_devicename}: Override Fan Speed"
    id: override_fan_speed_ha
    icon: "mdi:fan"
    optimistic: true
    turn_on_action:
      - globals.set:
          id: override_fan_speed
          value: 'true'
    turn_off_action:
      - globals.set:
          id: override_fan_speed
          value: 'false'

button:
  - platform: safe_mode
    name: "${friendly_devicename}: Restart (Safe Mode)"

output:
  - platform: ledc
    pin: 23
    frequency: 19531 Hz
    id: fan_pwm

number:
  - platform: template
    name: "${friendly_devicename}: Max Fan Speed Temperature"
    id: max_fan_speed_temperature_ha
    unit_of_measurement: "Ā°C"
    optimistic: true #Not sure what this does, research. leave? remove?
    min_value: 25
    max_value: 40
    step: 1
    on_value:
      then:
        - lambda: |-
            id(max_fan_speed_temperature) = id(max_fan_speed_temperature_ha).state;

  - platform: template
    name: "${friendly_devicename}: Fan Speed Override"
    icon: "mdi:speedometer"
    id: fan_speed_override_ha
    unit_of_measurement: "%"
    optimistic: true #Not sure what this does, research. leave? remove?
    min_value: 0
    max_value: 100
    step: 1
    on_value:
      then:
        - lambda: |-
            id(fan_speed_override) = id(fan_speed_override_ha).state;

sensor:
  - platform: wifi_signal
    name: "${friendly_devicename}: WiFi Signal"
    update_interval: ${update_interval_wifi}
    device_class: signal_strength

  - platform: template
    id: report_fan_speed
    name: "${friendly_devicename}: Fan Speed"
    icon: "mdi:fan"
    lambda: return id(fan_speed) * 100;
    unit_of_measurement: "%"
    accuracy_decimals: 0
    update_interval: ${update_interval_s}


#  - platform: qmp6988
#    i2c_id: bus_a
#    temperature:
#      name: "${friendly_devicename}: Temperature 2"
#      oversampling: 16x
#    pressure:
#      name: "${friendly_devicename}: Pressure"
#      oversampling: 16x
#    address: 0x70
#    update_interval: 5s
#    iir_filter: 2x

  - platform: sht3xd
    i2c_id: bus_a
    address: 0x44
    update_interval: 1s
    humidity:
      id: room_humidity_sensor
      internal: true
      force_update: true
      device_class: "humidity"
      state_class: "measurement"
      on_value:
        - globals.set:
            id: room_humidity
            value: !lambda 'return id(room_humidity_sensor).state;'
    temperature:
      id: room_temperature_sensor
      internal: true
      force_update: true
      device_class: "temperature"
      state_class: "measurement"
      on_value:
        - globals.set:
            id: room_temperature
            value: !lambda 'return id(room_temperature_sensor).state;'

  - platform: template
    name: "${friendly_devicename}: Room Temperature"
    id: report_room_temperature
    device_class: "temperature"
    state_class: "measurement"
    unit_of_measurement: "Ā°C"
    update_interval: never

  - platform: template
    name: "${friendly_devicename}: Room Humidity"
    id: report_room_humidity
    device_class: "humidity"
    state_class: "measurement"
    unit_of_measurement: "%"
    update_interval: never

  - platform: dallas
    address: 0xf2020292454de528
    name: "${friendly_devicename}: Temperature"
    device_class: "temperature"
    state_class: "measurement"
    id: computer_temp
    filters:
      - sliding_window_moving_average:
          window_size: 10
          send_every: 10
      # Map MEASURED -> TRUTH
      - calibrate_linear:
        - 0.0 -> 0.0
        - 25.0 -> 24.3
        - 40.0 -> 39.3
      - lambda: return x;
    on_value:
      - globals.set:
          id: computer_exhaust_temperature
          value: !lambda 'return id(computer_temp).state;'

interval:
  - interval: 1s
    then:
      lambda: |-
        if (id(computer_exhaust_fan)) {      
            if (!id(override_fan_speed)) {
                if (id(computer_exhaust_temperature) > id(max_fan_speed_temperature)) {
                  id(fan_speed) = 1.0;
                  ESP_LOGD("ALERT", "OVER int(id(max_fan_speed_temperature))C! Setting fan to 100");
                }
                else if (id(computer_exhaust_temperature) <= id(room_temperature_sensor).state) {
                  id(fan_speed) = 0.00;
                  ESP_LOGD("PWM", "Minimum fan speed as exhaust is <= ambient.");
                }
                else {
                  id(fan_speed) = (1.0 - 0.00) * ((id(computer_exhaust_temperature) - id(room_temperature_sensor).state)/(id(max_fan_speed_temperature) - id(room_temperature_sensor).state));
                }
                ESP_LOGD("PWM", "TEMPERATURE CONTROL - Max Speed @ %dC - CURRENT PWM: %d%%" , int(id(max_fan_speed_temperature)), int(id(fan_speed) * 100));
            } else {
                id(fan_speed) = id(fan_speed_override) / 100;
                ESP_LOGD("ALERT", "PWM OVERRIDE: %d%% - CURRENT PWM: %d%%", int(id(fan_speed_override)), int(id(fan_speed) * 100));
            }
        } else {
            id(fan_speed) = 0.0;
            ESP_LOGD("ALERT", "FANS TURNED OFF");
        }
        id(fan_pwm).set_level(id(fan_speed));

  - interval: 60s
    then:
      lambda: |-
        id(report_room_temperature).publish_state(id(room_temperature));  // Report Room Temperature Sensor
        id(report_room_humidity).publish_state(id(room_humidity));        // Report Room Humidity Sensor



script:
  - id: setup_controls
    then:
      - lambda: |-
          auto call = id(max_fan_speed_temperature_ha).make_call();
          call.set_value(id(max_fan_speed_temperature));
          call.perform();
      - lambda: |-
          auto call = id(fan_speed_override_ha).make_call();
          call.set_value(id(fan_speed_override));
          call.perform();
      - lambda: |-
          id(override_fan_speed_ha).publish_state(id(override_fan_speed));
          id(computer_exhaust_fan_ha).publish_state(id(computer_exhaust_fan));

  - id: setup_sensors
    then:
      - lambda: |-
          id(report_room_temperature).publish_state(id(room_temperature_sensor).state);  // Report Room Temperature Sensor
          id(report_room_humidity).publish_state(id(room_humidity_sensor).state);        // Report Room Humidity Sensor

image
image1398Ɨ1807 215 KB

image
image1920Ɨ1196 108 KB

image
image1920Ɨ1141 171 KB

Edit: In this last picture I had not yet added the external sensor that reports room temperature. The idea is that it is pointless to have the blower on if the computer Exhaust is the same temperature as the room regardless of what the temperature is. This way the fan turns off when the two temperatures are close.

2

Not sure I fully understand your use case and goals, but are you aware of this approach to limiting traffic to HA?

3

@Mahko_Mahko Say for example I am running a control loop that evaluates inputs 1/s then I need data 1/s too (or more but unused data will simply be dropped). However, for display purposes I only need data way less frequently as it does not change much or I donā€™t care to see it. For example the room temperature which changes slowly or the fan speed which changes frequently by a little which makes no appreciable difference for display. While the traditional approach is to slow everything down which would be fine for such a low speed and criticality system (referring to my project) I prefer to run the control as fast as possible with inputs coming in as fast as they can and then just rate limit the data for display. In one case this allowed me to publish multiple correlated values at the same time which eliminated an issue where the values shown were not adding up properly if I let ESPHome decide what value to show.

I have tried throttling the data as you suggest but if I recall correctly it fatr limits the inputs for the control loop too. What I have not tried is to leave the raw data unthrottled, use that for my loop and throttle the template in esphome used for displayā€¦ if that is what you were suggesting, I have not tried it yetā€¦ as always thanks for the feedback and suggestions!

4

picking up that discussion ā€¦ I currently do it like this

sensor:
# hardware
  - <<: !include
      file: sensor/bme280.yaml
      vars:
        vaddr: 0x76
        vprefix: ${gname}/bme280/
        vsendevery: 8
        vupdate: 30s # (8*30s = 4min)
# wifi
  - <<: !include
      file: sensor/wifisignal.yaml
      vars:
        vname: ${gname}/rssi
        vsendevery: 8
        vupdate: 30s # (8*30s = 4min)
# calc
  - <<: !include
      file: sensor/absfeuchte.yaml
      vars:
        vname: ${gname}/calc/huma
        vupdate: 10min

with the folloing advantages:
since passing the address I could use 0x76 aswell as 0x77 BME sensors without the need to modiify the included bme280.yaml
I do make use of 1 global variable(substitution) since thatā€™s needed in the esphome: section besides multiple places downwards. All other variables bound to the corresponding include files.
In case a sensor reports multiple values I pass a prefix for the first part of the name, of 1 value the name.
strategy does work for the packages: aswell as for the sensor: section.
Means my main yaml files for each device is if ever 100 lines total, since itā€™s more or less a collection of includes which either are correct or not, and thus similar across 1 - n devices which require them.

Thereā€™s 1 thing left Iā€™m not yet found a solution.
Some of the includes do require their update interval given within a lamba.
See the following example: (I would need to pass the ā€˜3750ā€™)

# MLX90614
- platform: custom
  lambda: |-
    auto mlx = new MLX90614(3750); // 30000ms/8 = 30s
    App.register_component(mlx);
    return {mlx->ambient_temperature_sensor, mlx->object_temperature_sensor, mlx->emissivity_value_sensor};
  sensors:
  - name: ${vprefix}ambi
    state_topic: ${vprefix}ambi
    id: a
    filters:
      - sliding_window_moving_average:
          window_size: 16

and I havenā€™t found a way to get that addressed, means passing a variable which ends up in the lamba. Anyone with an idea?

5

@justone While I need to review your code more I think what you are doing is limiting data from sensors and thus the control loop (if any) is working with less data. I am only trying to limit what gets published back to HA without limiting the sensors.

Regarding your question, I would create a global variable and then just reference that as your parameter.

globals:
  id: myvar
  type: int
  restore_value: no
  initial_value: '3750'

and the use it like this:

lambda: |-
    auto mlx = new MLX90614(id(myvar));

I am not a professional programmer but I believe that this will work.

Edit:

or you can use a substitution too. If I remember correctly the syntax is id(${nameofsubstitution}) if used in a lambda.

6

upsā€¦ when reading your lines yes ā€¦ I simply must have used ā€œid(ā€ stupid me ā€¦ asking.

to the initial one ā€¦ when you say you want to limit data but not the sensors?
What are all these sensor readings good for?

if you would build a sliding moving average or something like that, ok, you would benfit from reducing the explicit error of a single reading, but just picking each nā€™th reading to reduce updating HA ā€¦ I donā€™t get the benefit from that.

7

There is none, and I am not advocating having sensors update faster than the control loop (although I see no harm in doing so as long as the data isnā€™t sent back to HA) but the general method I see used is to rate limit the sensor data and to act on it when a new value is available. If I need the control loop to run fast, that typically results in lots of data sent back to HA using the common approach. My goal is to get as much data as needed to run the loop as fast as I need and just publish the data back to HA at a slower rate. This also allows me to run a faster control without bogging down HA to the point it is unusable (Iā€™ve had this happen many times). Iā€™ll post a better code example once I get closer to what I am trying to achieve.

8

Perhaps you could make use of on_raw_value for your control loop actions? Then apply throttle filter?

You could also look into using copy sensors. The original sensor could be marked as internal and drive internal calculations and the copy sensor could be throttled and be available in HA.

1 Like
9

@Mahko_Mahko Awesome suggestion! Iā€™ll look into it and try it out on one of my few systems I needs to make less chatty!

1 Like