I have a steadily increasing sensor (water consumption). Now I want a second sensor showing the consumption of the last 24 hours. So the formula would be: s2 = s1[current] - s1[24h ago].
With the utility meter I get only the consumption of the current day. This is not what I want. I need a sliding window. How can I do this? Maybe there is already a custom filter making a delay?
- platform: statistics
name: "Water Used in the Last 24 Hrs"
entity_id: sensor.your_sensor_here
state_characteristic: change
max_age:
hours: 24
sampling_size: 3500 # See below
You need to work out a sampling size with enough samples to cover 24 hours.
Sampling Size = 24 * 60 * 60 / <sensor update interval in seconds>
If your sensor updates irregularly make a guess at the average. Then increase the result of the calculation by 20%.
Watch the buffer_usage_ratio attribute of the sensor over a few days. You want this to be nearly full but never get completely full. i.e. close to 1 but never actually reaching 1.
2 Likes
canât he use a utility meter for this as setup time as 24hours?
1 Like
The statistics sensor has now gone to âunknownâ. The reason is that there was no change in the source sensor and so there is no new sensor value published. The statistics integration will not find values which are 24h old. What to do now?
Wait until it gets more states?
Or create a template sensor that sets the state to 0 if it is unknown.
Alternative solution using Node-Red
May I suggest an alternative solution that fetches a sensor value now, the sensor value from 24 hours ago, calculates x = s[now] - s[24hr ago] and populates a new sensor with the result, updated every how-often you like?
Added bonus - start and end readings (for period) as attributes
Node red code follows (this is the first time I have posted code like this so fingers crossed it is OK)
[{"id":"7a1df2413cd1c7ab","type":"api-get-history","z":"4a7ee910a33fb633","name":"solar energy history","server":"44b2605f.5d41","version":0,"startdate":"","enddate":"","entityid":"sensor.solar_energy","entityidtype":"is","useRelativeTime":false,"relativeTime":"24 hours","flatten":true,"output_type":"array","output_location_type":"msg","output_location":"payload","x":490,"y":1020,"wires":[["688d2cd4e62d8e5f"]]},{"id":"efbbacff28899875","type":"change","z":"4a7ee910a33fb633","name":"value now","rules":[{"t":"set","p":"topic","pt":"msg","to":"ending","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":660,"y":960,"wires":[["7739434efee0e422"]]},{"id":"ec0aae998ff88568","type":"moment","z":"4a7ee910a33fb633","name":"-24:00 hr","topic":"","input":"","inputType":"date","inTz":"ETC/UTC","adjAmount":"1440","adjType":"minutes","adjDir":"subtract","format":"","locale":"en-GB","output":"payload.startdate","outputType":"msg","outTz":"Europe/London","x":180,"y":1020,"wires":[["48f74368712ac9bb"]]},{"id":"8f55ae6fb8316574","type":"inject","z":"4a7ee910a33fb633","name":"Every 10 minutes - clean object","props":[{"p":"payload"}],"repeat":"600","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{}","payloadType":"json","x":230,"y":960,"wires":[["ec0aae998ff88568","d37837e57c12c80b"]]},{"id":"48f74368712ac9bb","type":"moment","z":"4a7ee910a33fb633","name":"-23:59 hr","topic":"","input":"","inputType":"date","inTz":"ETC/UTC","adjAmount":"1439","adjType":"minutes","adjDir":"subtract","format":"","locale":"en-GB","output":"payload.enddate","outputType":"msg","outTz":"Europe/London","x":320,"y":1020,"wires":[["7a1df2413cd1c7ab"]]},{"id":"688d2cd4e62d8e5f","type":"change","z":"4a7ee910a33fb633","name":"value then","rules":[{"t":"set","p":"payload","pt":"msg","to":"$number(payload[0].state)","tot":"jsonata"},{"t":"set","p":"topic","pt":"msg","to":"starting","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":670,"y":1020,"wires":[["7739434efee0e422"]]},{"id":"d37837e57c12c80b","type":"api-current-state","z":"4a7ee910a33fb633","name":"solar energy now","server":"44b2605f.5d41","version":3,"outputs":1,"halt_if":"","halt_if_type":"str","halt_if_compare":"is","entity_id":"sensor.solar_energy","state_type":"num","blockInputOverrides":false,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entityState"}],"for":"0","forType":"num","forUnits":"minutes","override_topic":false,"state_location":"payload","override_payload":"msg","entity_location":"data","override_data":"msg","x":490,"y":960,"wires":[["efbbacff28899875"]]},{"id":"7739434efee0e422","type":"join","z":"4a7ee910a33fb633","name":"","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":830,"y":980,"wires":[["354d5492bd7b58d8"]]},{"id":"354d5492bd7b58d8","type":"change","z":"4a7ee910a33fb633","name":"Difference","rules":[{"t":"set","p":"payload.difference","pt":"msg","to":"$round(msg.payload.ending-msg.payload.starting,2)\t","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":970,"y":980,"wires":[["483da723e30a2bb2"]]},{"id":"483da723e30a2bb2","type":"ha-entity","z":"4a7ee910a33fb633","name":"solar rolling 24 hours","server":"44b2605f.5d41","version":2,"debugenabled":false,"outputs":1,"entityType":"sensor","config":[{"property":"name","value":"Solar energy rolling 24 hours"},{"property":"device_class","value":"measurement"},{"property":"icon","value":"mdi:counter"},{"property":"unit_of_measurement","value":"kWh"},{"property":"state_class","value":"energy"},{"property":"last_reset","value":""}],"state":"payload.difference","stateType":"msg","attributes":[{"property":"StartReading","value":"payload.starting","valueType":"msg"},{"property":"EndReading","value":"payload.ending","valueType":"msg"}],"resend":true,"outputLocation":"payload","outputLocationType":"none","inputOverride":"allow","outputOnStateChange":false,"outputPayload":"","outputPayloadType":"str","x":1160,"y":980,"wires":[[]]},{"id":"44b2605f.5d41","type":"server","name":"Home Assistant","version":2,"addon":true,"rejectUnauthorizedCerts":true,"ha_boolean":"on","connectionDelay":false,"cacheJson":false,"heartbeat":false,"heartbeatInterval":"30"}]
Screenshot âŚ
I have set this up for my solar generation sensor (a Reimann Integral sensor from inverter power sensor) which updates every 20 seconds, so I have set a narrow 1 minute window for HA history. The default fetch is 24 hours which returns in this situation a very big array and I only want the first item. History node will return the state as it was at the point of the start datetime (and then for every state change within the start-end window) and current state node will return state now, so this should work no matter what the target entity state update frequency is.
You will, of course, need Node Red running (either as an add-on alongside HA or on another device) all the node types used (ie add node-red-contrib-moment using the pallet) a configuration node configured for the server connection to HA, and to change the local time zone (moment node), entity name (history, current state) and the code if you need to change rounding etc.
I hope this is of value to you or anyone else who wants to read and manipulate values from HA history recorder. Much as I like HA, there are certain things that appear to be overly difficult if not impossible to do with templates that take just a small amount of work in Node-Red, the get-history-node being one example.
4 Likes
I will wait until it is fixedâŚ
1 Like
Noty sure why to use the sensor update interval, as the number of updates might not always be consistent.
Iâm using:
start: "{{ now() - timedelta( hours = 24 ) }}"
end: "{{ now() }}"
This seems to work reliably as it is purely based on time.
You are confusing the history stats and statistics sensors. The statistics sensor does not have those options.
statistics would be lost at every HA restart, looking for alternativesâŚ
I was looking for the same thing and didnât find anything good enough, so worked out another solution using pyscript.
Sharing in case itâs useful to someone else, and in case anyone has ideas to make it better.
It works by creating for each monitored sensor.example a new sensor pyscript.24h_example containing the list of (differences of) values for each hour of the last 24h, the time of last update, the value of last update, and (as the main sensor value) the sum of all differences in the last 24h. Each time the script is run, it computes the difference between the current value and the value of last update and adds it to the list while removing the listâs first value. It checks if the update is run at each hour. If not it either doesnât update the sensor, or guesses approximate values if last update was between 2 and 23h ago (not precise, but good enough if it doesnât happen too often). If the system was offline for more than 24h it resets the sensor. Writes errors in log to help debug.
Not perfect because
- sensors must be in the pyscript domain to make them persistent across restarts (so they cannot be reused with some platforms like utility meter, not that it would make much sense with this one in particular^^)
- this hack stores only hourly values, itâs not nearly real time, but thatâs good enough for my needs
- needs an automation to run the script every hour.
- you need to list the sensors you want to monitor in the script
(you can use an argument for the service function instead, itâd be cleaner) - you could modify it to store and run every 30min but not sure it would be wise^^
Here is the script (to be put in config/pyscript/anyfile.py once pyscript is installed):
(updated on jan 30th because persistence across reboots did not always work)
import datetime
# list of sensors to monitor
sensors = ["sensor.example1_total_energy", "sensor.example2_total_energy"]
@service
def update_energy_stats():
"""Script to update energy stat sensors"""
#necessary to kill the script
task.unique("energy_stats")
today_datetime = datetime.datetime.now()
today_str = today_datetime.strftime("%d/%m/%Y %H:%M:%S")
for sensor in sensors:
daily_sensor = sensor.replace("sensor.", "pyscript.24h_")
try:
energy_value = float(state.get(sensor))
except:
# sensor does not exist. Log error and continue with the next item
log.error("sensor " + sensor.replace('sensor.', '') + " does not exist")
else:
if energy_value is None:
# sensor is null. Log error and continue with the next item
log.error("sensor " + sensor.replace('sensor.', '') + " has no value")
else:
go = True
# make the sensor "restart-proof"
state.persist(daily_sensor)
attributes = state.getattr(daily_sensor)
if attributes is None:
# sensor does not exist, we'll need to create it
values = [0] * 24
else:
last_value = float(attributes["last_value"])
value_difference = energy_value - last_value
last_update = attributes["last_update"]
values = attributes["values"]
datetime_last_update = datetime.datetime.strptime(last_update, '%d/%m/%Y %H:%M:%S')
time_ago = round((today_datetime - datetime_last_update).seconds /3600, 2)
if time_ago >= 23.1:
values = [0] * 24
log.error("sensor " + daily_sensor.replace('sensor.', '') + " was updated " + str(time_ago) + " hours ago, so we reset it to zero.")
elif 0.9 <= time_ago < 23.1:
hours_ago = round(time_ago)
# if hours_ago == 1 everything is good
# otherwise, we missed one or several updates. Let's split the value difference into equal parts for the missing time (best guesstimate)
# this also works for hours_ago == 1
value_slice = value_difference / hours_ago
values = values[hours_ago:] + [value_slice] * hours_ago # slide the list of values and add the new one(s)
if hours_ago != 1: # log this for debug
log.error("sensor " + daily_sensor.replace('sensor.', '') + " was updated " + str(time_ago) + " hours ago, so we split it into " + str(hours_ago) + " equal parts :")
else:
log.error("Not updating sensor " + daily_sensor.replace('sensor.', '') + " (updated " + str(time_ago) + " hours ago)")
go = False
if go:
last_24h_value = sum(values)
attributes = {
"last_update": today_str,
"last_value": energy_value,
"values": values,
"device_class": "energy",
"unit_of_measurement": "kWh"
}
state.set(daily_sensor, value=last_24h_value, new_attributes=attributes)
@time_trigger("startup")
def persistOnStartup():
log.error("Persist pyscript energy stats on startup")
for sensor in sensors:
daily_sensor = sensor.replace("sensor.", "pyscript.24h_")
state.persist(daily_sensor)
@time_trigger("shutdown")
def persistOnShutdown():
log.error("Persist pyscript energy stats on shutdown")
for sensor in sensors:
daily_sensor = sensor.replace("sensor.", "pyscript.24h_")
state.persist(daily_sensor)
Then I run the script hourly with an automation :
alias: Conso - Update energy hourly
description:
trigger:
- platform: time_pattern
minutes: "0"
action:
- service: pyscript.update_energy_stats
metadata: {}
data: {}
mode: single
Works well. Anyone has ideas/critics ?
I had the same requirement and did some web searches. The proposels here came closest to what I was looking for.
However I am now using a little adopted solution:
I have an always incresing sensor which is counting my gas consumption and I wanted to see the consumption of the last hour and of the last 24 hours.
In the graphical user interface I added two helpers using the statistics integration.
One for the last hour, one for the last 24 hours.
For me the trick was to use sum_differences as state_characteristic.
For max_age I used 1h and 24h. I left sampling_size empty.
1 Like
If you want a sensor that you can use on your ESPHome device, to take in a source that has an ever increasing âlifetimeâ number of litres used and output the consumption of litres for Today, Yesterday, Week, Month and Year then I have a custom_component you can use for this.
I use it on my âUtilities Monitorâ that is on a Kincony s3 Core board with a few reed switches connecting it to the town water meter, gas meter and a BME280 environmental sensor.
Hereâs the local configuration for the device in ESPHome builder:
#########################################################################
#
# Environmental monitoring using ESP32-S3 chipset and BME280 sensor(s)
# for temperature, humidity, pressure & dew point.
#
substitutions:
###############################
# Device Details:
#
# Default name
name: "utilities-monitor"
# Default friendly name
friendly_name: "Utilities Monitor"
# Description as appears in ESPHome & top of webserver page
device_description: "Gas, Water & Environmental Monitor (Temp./Humidity/Pressure) ESP 2024.11.3 & Stats with Globals"
###############################
# Location & Site Details:
#
# Allows ESP device to be automatically lined to an 'Area' in Home Assistant. Typically used for areas such as 'Front Gate, Decontamination, Shed #XYZ etc
location: "Garage"
# Location geographical height, in meters. Used for calculations in sensors such as BME280/BME680 and Water Tank sensors.
Average = 666m.
location_height: "673"
# The phase in the locations power supply, upon which this device is utilsed. 1p = A. 3p = A, B or C. (GPO 1 = Phase A, GPO 2 = Phase B, GPO 3 = Phase C).
# Used to populate a Text Sensor with this information, that may be used to loadshed in future, or trace outages to electrical faults.
power_circuit: "TBA"
##########################
# Device Settings:
#
# Set the update interval for sensors (Default is 10s)
sensor_update_interval: 10s
# Define logging level: NONE, ERROR, WARN, INFO, DEBUG (Default), VERBOSE, VERY_VERBOSE
log_level: "DEBUG"
###############################
# Network Settings - Static IP
#
# Network - Static IP - Address
network_ip: ""
# Network - Static IP - Gateway
network_gateway: ""
# Network - Static IP - Subnet Mask
network_subnet: ""
# Network - Static IP - DNS Servers
dns_server1: ""
dns_server2: ""
# Define a domain for this device to use. i.e. iot.home.lan (so device will appear as athom-smart-plug-v2.iot.home.lan in DNS/DHCP logs)
dns_domain: ""
###############################
# BME 280 / BME680 Sensor
#
# Location of BME280/680 Sensor #1. Used to prefix sensor if there are two. Leave empty if only 1. Add a " - " to the end of each entry if used.
bme_sensor1_location: ""
# Location of BME280/680 Sensor #2. Used to prefix sensor if there are two. Leave empty if only 1. Add a " - " to the end of each entry if used.
bme_sensor2_location: ""
# Calibrate the outpute of the temperature sensor(s).
temperature_1_offset: "0.0"
temperature_2_offset: "0.0"
# Calibrate the outpute of the pressure sensor(s).
pressure_1_offset: "0.0"
pressure_2_offset: "0.0"
# Calibrate the outpute of the humidity sensor(s).
humidity_1_offset: "0.0"
humidity_2_offset: "0.0"
###############################
# TL-136 / LIQ-136 Liquid Pressure Sensor
#
# Name of the type of tank the sensor is being used to monitor. Omit entering 'Tank'.
# i.e. Water (Default), Sewerage, Disinfectant etc.
tank_name: "Water"
# Tank Inlet Height - Maximum level of water in centimetres (e.g. 241).
inlet_height: "241"
# Tank Diameter (i.e. distance from one side to the other) in centimeters.
tank_diameter: "238"
# Tank Radius - For a cylindrical tank. This will be half the diameter, in centimeters.
tank_radius: "119"
# Tank Level Offset. Add 0.01847cm to total, to correct to 0L when voltage is 0.469V
tank_level_offset: "0.01847"
# Note: Default measurements are based upon a TankWorld 10,000L Tall Straight Wall Tank
# https://www.tankworld.com.au/product/10000l-tall-straight-wall-tank/
# Tank Capacity: 10,000L | Diameter Width: 2380mm
# Inlet Height: 2410mm | Tank Height: 2485mm
########################## End of Substitutions #########################
esphome:
name_add_mac_suffix: false
#########################################################################
#
# Remote Packages to be utilised
#
packages:
remote_package:
url: https://github.com/Roving-Ronin/myHomeAssistant/
ref: main
files: [esphome/utilities-monitor.yaml]
refresh: 0s
https://github.com/Roving-Ronin/myHomeAssistant/blob/8b17e87faf4cdc9762913f3661c62d7cdbb0296d/esphome/utilities-monitor.yaml
You just need to update the substititions to suit and GPIO if using different hardware. From the packages section it pulls the primary yaml for this device from myHomeAssistant/esphome/utilities-monitor.yaml at 8b17e87faf4cdc9762913f3661c62d7cdbb0296d ¡ Roving-Ronin/myHomeAssistant ¡ GitHub and in turn calls other packages for setting base config, network config, bme280 sensors, water tank sensors, and consumption for gas and town water.
The town water sensor yaml ( myHomeAssistant/esphome/sensors/utilities-town-water.yaml at 8b17e87faf4cdc9762913f3661c62d7cdbb0296d ¡ Roving-Ronin/myHomeAssistant ¡ GitHub ) is loaded and uses the custom component âwater_statisticsâ (referred by - platform: âwater_statisticsâ ). If you just want to use this component, this is the file you should look at to replicate in your ESPHome config, just make sure you call the custom component using this in your device yaml (you can remove the gas references below if not needed:
external_components:
# - source: github://roving-ronin/myhomeassistant/components@statistics-partial-times
# refresh: 0s
- source:
type: git
url: https://github.com/roving-ronin/myhomeassistant/
ref: main
refresh: 3d
components: [ gas_statistics, gas_statistics_mj, water_statistics ]
This custom_component is located at : myHomeAssistant/components/water_statistics at main ¡ Roving-Ronin/myHomeAssistant ¡ GitHub and uses a source sensor of the ever_increasing (aka lifetime litres) sensor, that the platform: pulse_meter and its âtotal:â option provides.
ESPHome webserver looks like this, with all these sensors automatically being available in HA (with no need to fiddle with utility_meter etc). As readings are saved in ESP flash, the solution has been tested and confirmed to keep its readings whennever the ESP is reflashed / updated.
Hope this is of assistanceâŚ
PS. Custom component is also there to do the same for Gas in MJ and m3 also, as well as another variation used for tracking energy consumption.
Edit: Todays are showing 0 as just noticed its after midnight where I am, so its rolled over. The water tank readings (top) are no live, as needing to run cabling to rainwater tank currently, but yaml is sound.