A Wemo D1 Mini along with an ultrasonic distance measurement sensor (HC-SR04) are measuring the salt level in my water softener. The salt level changes very slowly and only every few days therefore I want to curb the data sent back to HA. I have tested many options but can’t seem to achieve my goal.
Goal:
Measure distance (ideally performing an average to smooth results but may be overkill)
Calculate salt height in tank (total height - sensor to salt distance)
Map salt height to a percentage of how full the tank is
Update all values asap upon power up / reboot / etc.
Refresh values every few hours
In my many attempts mostly playing around with update_interval and throttle I end up having to wait a long time to get the templated sensors to show up. The code below has short delays so it may seem fine but once I set the refresh to hours, I would have to wait for hours before a salt height and tank fill would show.
What is the correct way to throttle values? Ideally I can read values as often/fast as I want within the ESP and only pass back values periodically. This way I can see a live value on the Wemo’s web page and get periodic updated values in HA.
In the code below you will also see a throttle_average as at some point the measurement was unstable and was causing the tank percentage to constantly change. This will be less noticeable when there is a long delay but I still don’t like the idea of the value ever increasing, as it should only increase if I am adding salt. I figured that averaging the numbers might help mitigate this. The sliding average alternative seemed overkill but please correct me if I am wrong.
substitutions:
devicename: water-softener
devicename_no_dashes: water_softener
friendly_devicename: "Water Softener"
device_description: "Water Softener"
#Distance between bottom of tank and sensor in cm as sensor value gets converted to cm right away
distance_sensor_to_bottom: "90"
update_interval_s: "5s"
throttle_interval_s: "120s"
update_interval_wifi: "120s"
esphome:
name: ${devicename}
comment: ${device_description}
esp8266:
board: d1_mini
wifi:
ssid: !secret iot_wifi_ssid
password: !secret iot_wifi_password
# Enable fallback hotspot (captive portal) in case wifi connection fails
ap:
ssid: "${devicename} Hotspot"
password: !secret iot_wifi_password
logger:
baud_rate: 0 #disabled
api:
ota:
web_server:
port: 80
include_internal: true
text_sensor:
- platform: wifi_info
ip_address:
name: "${friendly_devicename}: IP"
icon: "mdi:ip-outline"
update_interval: ${update_interval_wifi}
ssid:
name: "${friendly_devicename}: SSID"
icon: "mdi:wifi-settings"
update_interval: ${update_interval_wifi}
bssid:
name: "${friendly_devicename}: BSSID"
icon: "mdi:wifi-settings"
update_interval: ${update_interval_wifi}
mac_address:
name: "${friendly_devicename}: MAC"
icon: "mdi:network-outline"
scan_results:
name: "${friendly_devicename}: Wifi Scan"
icon: "mdi:wifi-refresh"
disabled_by_default: true
sensor:
- platform: wifi_signal
name: "${friendly_devicename}: WiFi Signal"
update_interval: ${update_interval_wifi}
device_class: signal_strength
#Sensor to salt measurement from sensor
- platform: ultrasonic
trigger_pin: D1
echo_pin: D2
timeout: 4.0m #This value should be close to actual limit of sensor
id: sensor_to_salt_cm
name: "${friendly_devicename}: Sensor to salt"
icon: "mdi:format-vertical-align-bottom"
update_interval: ${update_interval_s}
state_class: measurement
unit_of_measurement : "cm"
accuracy_decimals: 1
filters:
- filter_out: nan
- throttle_average: ${throttle_interval_s}
- lambda: return (x*100); #convert from m to cm
#Salt Level
- platform: template
id: saltLevel
name: "${friendly_devicename}: Salt Level"
icon: "mdi:arrow-expand-vertical"
update_interval: ${update_interval_s}
lambda: return ${distance_sensor_to_bottom} - id(sensor_to_salt_cm).state ;
unit_of_measurement: "cm"
accuracy_decimals: 1
filters:
- throttle: ${throttle_interval_s}
#Salt Tank Fill Percentage
- platform: template
id: saltTankLevel
name: "${friendly_devicename}: Salt Tank Level"
icon: "mdi:arrow-expand-vertical"
update_interval: ${update_interval_s}
lambda: return id(sensor_to_salt_cm).state ;
unit_of_measurement: "%"
accuracy_decimals: 0
filters:
- calibrate_linear:
# Map 0.0 (from sensor) to 0.0 (true value)
- 20.0 -> 100.0
- 55.0 -> 50.0
- 90.0 -> 0.0
- throttle: ${throttle_interval_s}
switch:
- platform: restart
name: "${friendly_devicename}: Restart"
button:
- platform: safe_mode
name: "${friendly_devicename}: Restart (Safe Mode)"
#deep_sleep:
# id: ${devicename_no_dashes}_deep_sleep
# sleep_duration: 30min
time:
- platform: sntp
id: ${devicename_no_dashes}_time
status_led:
pin:
number: D4
inverted: true
My understanding is it work with your update interval. So lets say your update interval is “10s” your first reading on boot will be 10s x 6 = 60s or 1 minute. I do this because normally your first reading is wrong. Then we have a window size of 60. Now you have the send every 60, so 60 x 10s = 10 minutes. This gives a smooth average reading every 10 minutes.
or maybe you just want a heartbeat. I haven’t use a heartbeat so don’t know if this is correct.
Maybe just put “1h” in for your heartbeat first and see if in your logs it sends every hour. Then change it to what you would like. if your update interval is 10s I think this will just give you a reading on boot in 60s and a raw heartbeat reading every 24h or every day.
You got 3 sensors to play around with for testing anyway so maybe try both.
@Blacky Thank you for the suggestion! I tried option#1 and all the values updated within a few seconds so that was great. Looking at the logs, I see the sensor updating every 10s and up until now no new values have been sent to HA… awesome! Hopefully the log won’t stop logging if I leave it open in the background for a few hours to confirm the values get updated in HA after an hour… I guess I can use Grafana to see when data points come in if the logs aren’t there.
Thanks again!
EDIT: Sharing latest iteration of the code for the water softener salt level measurement using a Wemo D1 and an ultrasonic sensor in case others are interested:
substitutions:
devicename: water-softener
devicename_no_dashes: water_softener
friendly_devicename: "Water Softener"
device_description: "Water Softener"
#Distance between bottom of tank and sensor in cm as sensor value gets converted to cm right away
distance_sensor_to_bottom: "90"
update_interval_s: "10s"
throttle_interval_s: "120s"
update_interval_wifi: "120s"
esphome:
name: ${devicename}
comment: ${device_description}
esp8266:
board: d1_mini
wifi:
ssid: !secret iot_wifi_ssid
password: !secret iot_wifi_password
# Enable fallback hotspot (captive portal) in case wifi connection fails
ap:
ssid: "${devicename} Hotspot"
password: !secret iot_wifi_password
logger:
baud_rate: 0 #disabled
api:
ota:
web_server:
port: 80
include_internal: true
text_sensor:
- platform: wifi_info
ip_address:
name: "${friendly_devicename}: IP"
icon: "mdi:ip-outline"
update_interval: ${update_interval_wifi}
ssid:
name: "${friendly_devicename}: SSID"
icon: "mdi:wifi-settings"
update_interval: ${update_interval_wifi}
bssid:
name: "${friendly_devicename}: BSSID"
icon: "mdi:wifi-settings"
update_interval: ${update_interval_wifi}
mac_address:
name: "${friendly_devicename}: MAC"
icon: "mdi:network-outline"
scan_results:
name: "${friendly_devicename}: Wifi Scan"
icon: "mdi:wifi-refresh"
disabled_by_default: true
sensor:
- platform: wifi_signal
name: "${friendly_devicename}: WiFi Signal"
update_interval: ${update_interval_wifi}
device_class: signal_strength
#Sensor to salt measurement from sensor
- platform: ultrasonic
trigger_pin: D1
echo_pin: D2
timeout: 4.0m #This value should be close to actual limit of sensor
id: sensor_to_salt_cm
name: "${friendly_devicename}: Sensor to salt"
icon: "mdi:format-vertical-align-bottom"
update_interval: ${update_interval_s}
state_class: measurement
unit_of_measurement : "cm"
accuracy_decimals: 1
filters:
- filter_out: nan
#- throttle_average: ${throttle_interval_s}
- lambda: return (x*100); #convert from m to cm
- median:
window_size: 60
send_every: 60
send_first_at: 6
#Salt Level
- platform: template
id: saltLevel
name: "${friendly_devicename}: Salt Level"
icon: "mdi:arrow-expand-vertical"
update_interval: ${update_interval_s}
lambda: return ${distance_sensor_to_bottom} - id(sensor_to_salt_cm).state ;
unit_of_measurement: "cm"
accuracy_decimals: 1
filters:
#- throttle: ${throttle_interval_s}
- median:
window_size: 60
send_every: 60
send_first_at: 6
#Salt Tank Fill Percentage
- platform: template
id: saltTankLevel
name: "${friendly_devicename}: Salt Tank Level"
icon: "mdi:arrow-expand-vertical"
update_interval: ${update_interval_s}
lambda: return id(sensor_to_salt_cm).state ;
unit_of_measurement: "%"
accuracy_decimals: 0
filters:
- calibrate_linear:
# Map 0.0 (from sensor) to 0.0 (true value)
- 20.0 -> 100.0
- 55.0 -> 50.0
- 90.0 -> 0.0
#- throttle: ${throttle_interval_s}
- median:
window_size: 60
send_every: 60
send_first_at: 6
switch:
- platform: restart
name: "${friendly_devicename}: Restart"
button:
- platform: safe_mode
name: "${friendly_devicename}: Restart (Safe Mode)"
#deep_sleep:
# id: ${devicename_no_dashes}_deep_sleep
# sleep_duration: 30min
time:
- platform: sntp
id: ${devicename_no_dashes}_time
status_led:
pin:
number: D4
inverted: true
EDIT: It is sending updates every 10 minutes. I now changed send_every: 60 to send_every: 120 to see if the 60 was referring to measurements (1 every 10s) rather minutes. If that is the case, I would expect it to update every 20 minutes now. If so, changing it to 600 should bump it up to 1hr which is higher than needed but totally fine.
If you go to 120 maybe change your window size to 120 or the same if you go to 600 make your window 600. It will take all the readings in window and give you a average so it will be stable and as you said your water softener is slow in change. Try it and see if you like it
@Blacky - I ended up having some issues with using median but I found an alternative which works which is similar. I adopted it only on the main sensor the one that actually reads from hardware. The issue I kept having in all my attempts was that each sensor did its own thing so while all the numbers are related, the time delta would sometimes result in numbers that did not match. It wasn’t a big issue other than it annoyed me that the measurements did not sum up to the exact distance between the sensor and the bottom of the tank (90cm). What I wanted, and I think I got it, was to trigger the template sensors to only report when the main one updates. So throttling the main one, resulted in the other ones only reporting when the the main one does.
I did not figure out how the timing works though… just that things work better when window_size and send_every have the same value, so even though I don’t need an average on so many readings, I kept it at 120 since setting send_every to 120 results in a reading every 10 minutes (not sure why… math doesn’t work out… the sensor updates every 10s, so 60 readings at 10s each is 600s which is 10 minutes, but instead I need 120 to get to 10 minutes. 60 only gets me to… 5 minutes.).
Code below is latest interaction… WIP.
substitutions:
devicename: water-softener
devicename_no_dashes: water_softener
friendly_devicename: "Water Softener"
device_description: "Water Softener"
#Distance between bottom of tank and sensor in cm as sensor value gets converted to cm right away
distance_sensor_to_bottom: "90"
#Set to 4hrs. Needs to be very long otherwise it overrides the frequency set in the main sensor
update_interval_s: "14400s"
#Only reason not to set it very long it for wifi troubleshooting
update_interval_wifi: "120s"
#Whirlpool WHES48 has 8 markers, 10cm apart (middle to middle), but 8th can/should be ignored
cm_per_salt_level_marker: "10"
esphome:
name: ${devicename}
comment: ${device_description}
esp8266:
board: d1_mini
wifi:
ssid: !secret iot_wifi_ssid
password: !secret iot_wifi_password
# Enable fallback hotspot (captive portal) in case wifi connection fails
ap:
ssid: "${devicename} Hotspot"
password: !secret iot_wifi_password
logger:
baud_rate: 0 #disabled
api:
ota:
web_server:
port: 80
include_internal: true
text_sensor:
- platform: wifi_info
ip_address:
name: "${friendly_devicename}: IP"
icon: "mdi:ip-outline"
update_interval: ${update_interval_wifi}
ssid:
name: "${friendly_devicename}: SSID"
icon: "mdi:wifi-settings"
update_interval: ${update_interval_wifi}
bssid:
name: "${friendly_devicename}: BSSID"
icon: "mdi:wifi-settings"
update_interval: ${update_interval_wifi}
mac_address:
name: "${friendly_devicename}: MAC"
icon: "mdi:network-outline"
scan_results:
name: "${friendly_devicename}: Wifi Scan"
icon: "mdi:wifi-refresh"
disabled_by_default: true
sensor:
- platform: wifi_signal
name: "${friendly_devicename}: WiFi Signal"
update_interval: ${update_interval_wifi}
device_class: signal_strength
#Sensor to salt measurement from sensor
- platform: ultrasonic
trigger_pin: D1
echo_pin: D2
timeout: 4.0m #This value should be close to actual limit of sensor
id: sensor_to_salt_cm
name: "${friendly_devicename}: Sensor to salt"
icon: "mdi:format-vertical-align-bottom"
update_interval: 10s #Controls how often the sensor reports internally, not to HA, so 10s is fine even if overkill
state_class: measurement
unit_of_measurement : "cm"
accuracy_decimals: 1
filters:
- filter_out: nan
- sliding_window_moving_average:
window_size: 120 #Both 60 and 120 seem to not affect the cadence, however I would use a smaller number to get rid of outliers sooner
send_every: 120 #Works out to 10 minutes but not sure why
send_first_at: 1
- lambda: return (x*100); #convert from m to cm
on_value:
- sensor.template.publish:
id: saltLevel
state: !lambda 'return ${distance_sensor_to_bottom} - id(sensor_to_salt_cm).state ;'
- sensor.template.publish:
id: saltTankLevel
state: !lambda 'return id(sensor_to_salt_cm).state ;'
- sensor.template.publish:
id: saltTankLevelNumber
state: !lambda 'return (${distance_sensor_to_bottom} - id(sensor_to_salt_cm).state) / ${cm_per_salt_level_marker};'
#Salt Level
- platform: template
id: saltLevel
name: "${friendly_devicename}: Salt Level"
icon: "mdi:arrow-expand-vertical"
#lambda: return ${distance_sensor_to_bottom} - id(sensor_to_salt_cm).state ;
unit_of_measurement: "cm"
accuracy_decimals: 1
update_interval: ${update_interval_s}
#Salt Tank Fill Percentage
- platform: template
id: saltTankLevel
name: "${friendly_devicename}: Salt Tank Level"
icon: "mdi:arrow-expand-vertical"
#lambda: return id(sensor_to_salt_cm).state ;
unit_of_measurement: "%"
accuracy_decimals: 0
update_interval: ${update_interval_s}
filters:
- calibrate_linear:
# Map 0.0 (from sensor) to 0.0 (true value)
- 20.0 -> 100.0
- 55.0 -> 50.0
- 90.0 -> 0.0
#Manufacturer specific Tank Fill Level
- platform: template
id: saltTankLevelNumber
name: "${friendly_devicename}: Salt Tank Level Number"
icon: "mdi:arrow-expand-vertical"
#lambda: return (${distance_sensor_to_bottom} - id(sensor_to_salt_cm).state) / ${cm_per_salt_level_marker};
unit_of_measurement: "Lvl"
accuracy_decimals: 0
update_interval: ${update_interval_s}
filters:
- filter_out: nan
switch:
- platform: restart
name: "${friendly_devicename}: Restart"
button:
- platform: safe_mode
name: "${friendly_devicename}: Restart (Safe Mode)"
#deep_sleep:
# id: ${devicename_no_dashes}_deep_sleep
# sleep_duration: 30min
time:
- platform: sntp
id: ${devicename_no_dashes}_time
status_led:
pin:
number: D4
inverted: true
And this is my dashboard (still working on it & trying options). After the large spike, the values are smoother thanks to median and then the one I adopted. I am not sure what was making the reading go up/down but I suspect it has to do with the garage temperature as I’ve read that accuracy of this sensor varies with temperature (sound speed through air changes with temp… but is that enough for such a short distance?):
I haven’t tested a 4 hour interval as I normally make it 10s but if it work for you then that’s all good
I found, through a ESP chip the voltage changes on temp, air pressure, humidity and your readings can change because of this. I think the code below will show you your voltage on your pin 17 as on the ESP8266 only pin A0 (GPIO17) supports this. If you see the voltage change even though levels don’t change (check your pin reference) that is probably why your readings are floating. If you find a fix for this I would be interested to hear from you.
# ESP Voltage
sensor:
- platform: adc
pin: GPIO17 #check your pin I don't have the drawing for reference
name: "${friendly_devicename}: Voltage"
update_interval: 10s
icon: "mdi:current-dc"
accuracy_decimals: 2
attenuation: 11db
filters:
- median:
window_size: 60
send_every: 60
send_first_at: 6
@Blacky The 14400s interval is there in place of disabling the update_interval altogether which I could not find how to do… setting it to 0s just got me in trouble with the ESP updating sensors as fast as it could. That setting is only for the template sensors that have no input of their own as they all pull their input from the main sensor. Their value is calculated in the lambdas of the main sensor.
The ultrasonic sensor I am using is digital so no adc involved. However I agree that temperature fluctuations on the ESP are likely to increase error. Anyhow, if you look at the graph above, I was trying to get rid of the large ramps and drops rather that the smaller ones. The large ones indicate some drift (likely temperature related) while the small ones are likely within the poor accuracy of all the hardware components involved. In terms of distance, the fluctuations are around 0.2cm… noise.
With regards to your issue, since improving hardware would be on the top of my list, but not feasible with ESPs, have you tried sliding average rather than median? Oversampling and averaging should help improve your measurement.
Edit:
This document seems to address your problem and I saw while scanning on my phone that it offered some solutions :.
