Both flow and total values are sent to HA every 60s.
This works fine.
The problem is on ESP32 restart - total pulses value is reset to 0.
I use total pulses value to synchronize total water meter value in HA with value on mechanical water meter so I need to restore the total value after ESP32/ESPHome restart.
I thought about following methods:
Store total value in ESP32 flash - is that possible? It would be the best solution as ESP32 become fully independent counter. How this affect lifetime of ESP32 flash?
Keep the total value in HA and send back to ESPHome when HA total value < ESPHome total value. Or something like that. Would be that possible?
Count the value from flow sensor (in l/min) in HA. I spent plenty of time to find the way how to do it but no success.
Does anyone have a solution for this?
(this should be same for water drop meter, gas meter etc.)
Hi, I spent a lot of time playing around with my electricity meter in ESPhome over a year ago and eventually settled on the implementation detailed below which you may be able to adapt for your use case. The two things which I did which resulted in a solid and relabile solution that never seems to reset is the following:
Use the total_daily_energy platform is ESPhome (I originally tried the integration platform but had problems with it spiking to an infinite value every couple of months - I was never able to solve this).
Use the total_daily_energy sensor as the source for utility_meter sensors in HA (note even though the total_daily_energy sensor from ESPhome resets daily, the utility_meter keeps counting up, I don’t know why but it seems to work well).
Interesting topic. But what I do not get is why you recommend the
How is that related to water usage? It is all about W and kW, how does that relate back to L (Liters)?
Or do you have some formula to translate the output of the energy sensor?
Hi, I don’t think it matters what you are measuring … it could be water, electricity, gas etc. as all you are measuring in reality is pulses. These are then converted to the units you want to measure using appropriate conversion factors.
In theory you could use the integration platform but I had problems with that and found that the total_daily_energy sensor worked better (I tried raising an issue on Github but that did not go anywhere). However, as long as you scale your pulse_counter sensor correctly then the “total_daily_energy” sensor would actually be reporting total daily litres of water and you could name it appropriately.
As you say, the counts will get reset when the microcontroller is rebooted. The way I got around this was by using the utility_meter in HA. To give an example, if the counts went 0 > 10, there was a reboot, then the counts went 0 >20, the utility meter would have counted to 30.
I don’t have a pre-packaged solution for you but I was just attempting to give you some ideas to try and prevent you from facing some of the problems I did (which took me many months of testing to iron out as the problem I had occurred so infrequently).
Hi Mark, thank you for your suggestions, they are very appreciated.
I already use utility meter in HA, it works for me.
But I’m trying to find a solution for a lifetime meter that will be resistant enough to data loss and provides accurate results.
Yes, there is a lot of workarounds like energy meter, integration etc. but I need just simple pulse counter that will be able to read and store values from ESPHome. I do not see a point to use integration functions to count few integer numbers, e.g. 5+6= 11 and not 10,999 I believe there must be some functionality in HA that simply counts values coming from sensor. Something like counter sensor but able to increment the state with variable values.
I tried a lot of workarounds like what you suggesting above but this is not correct solution to my question, I would like to come back to my original question please.
Im a struggling with the same issue as well. I am using energy meters with Home Assistant for quite some time (first MySensors then ESPHome). I think a native solution based on reliably storing the value in Home Assistant and restoring it when an ESPHome device is restared should be added to the ESPHome integration in HA. In the meantime there are multiple options.
The correct choice depends also on the reason for the restart of the ESP32.
Store in the flash of the esp device.
In theory that works fine. (and ESPHome supports it). But: Storing the value on every change (pulse) will generated lots and lots of flash writes. For me about 650 000 a year. AFAIK they do not use wear leveling. That will kill the flash in days or weeks.
Options to improve it: Don’t store every pulse maybe only once every day. (every hour seems stil to often). But you will always lose some data.
A variant: Store in flash only for clean reboots (OTA, software triggered reboots).
An other idea: (more complicated) Store in flash but implement your own wear leveling. You definitely need C/C++ code that. Not sure if it will help enough. But together with storing only once every hour or 15 minutes it may work.
Store in RTC
This will not cause any wear and should help for reboots (caused by an OTA update for example).
Add a “big” capacitor and store in flash on power loss and reboots
You can google for examples how to implement the hardware for that. Software side is also not that complicated. The issue: Will not work when it crashes.
Same but with a battery
Just power the ESP32 with a Lithium battery that charges itself when connected to USB. (The simplest solution: Get a USB power bank that can be charged and used at the same time). When the voltage goes to low -> store in flash.
ESP32 supports SD-Cards. They implement wear leveling and are big enough that it will live for years storing the pulses. (the bigger the card the longer it should last)
Build some form of contraption using InputNumber objects in HA and automations.
That’s what I am doing and does not always work correct Will post the automations below. Maybe somebody can help fixing it.
Use MQTT retained messages to store the value in the MQTT broker.
This should work great but depends on the MQTT broker not getting reset. The ESP32 pushes the value to MQTT with the retain flag. And also subscripe to the topic. At boot it should get the message and should use it to initialize a global value with the last written value that you can add to the sensor value in a lambda of the sensor. (did not test it but that should work, maybe I try to set it up later). You can use MQTT and HA native api a the same time.
Use an app daemon script that always stores the sensor value from HA somewhere (a simple file, a database, whatever) and that will push (using a HA service created by the ESPHome using “api:”) the last value to ESPHome when needed. (ESPHome could ask for it for example using a binary template sensor).
That should work great if you know Python/Appdaemon or a willing to learn.
Just always use the integral of the current flowrate/wattage for everything.
That is what I do as backup as my current solution does not work perfectly. I push every momentarily wattage value to HA. HA pushes it to InfluxDB. InfluxDB can easly calculate total usage values by integrating the wattage values over time. Pushing a new wattage value on every pulse or the average for for every minute will make this exact in theory.
A variant: Store the reseting (to 0) value of the total usage in InfluxDB. Use the absolute derivative function to build power/flow rate values and integrate those to have to total usage value.
But well that seems to complicated…
My solution using input number and automations that does NOT work in 100% of the cases:
In automation in HA
- alias: store last usage when sensor goes offline
trigger:
platform: state
entity_id: sensor.power_usage
to: unavailable
action:
- service: input_number.set_value
data_template:
entity_id: input_number.stored_power_usage
value: '{{ trigger.from_state.state }}'
OK, I tried the “store in MQTT” method and it works great. If you have an MQTT broker and can live with the fact that the value gets lost of you reset the broker that seems like very simple and stable solution:
ESPHome yaml: (it counts the pulses of the BOOT button on my ESP32 -> one press -> 1 liter of use
esphome:
name: mqtt_save_poc
platform: ESP32
board: featheresp32
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
# Enable logging
logger:
ota:
password: !secret homeassistant_api_password
# Enable Home Assistant API
api:
password: !secret homeassistant_api_password
mqtt:
broker: mosquitto
discovery: False
on_message:
- topic: "mqtt_save_poc/sensor/total_water_new/state"
then:
lambda: |-
static bool already_restored = false;
if (already_restored)
return;
auto value = esphome::parse_float(x);
if (value.has_value()) {
id(last_usage) = value.value();
already_restored = true;
}
globals:
- id: last_usage
type: float
restore_value: no
initial_value: NAN
sensor:
- platform: pulse_counter
# counts pulses of the BOOT button on my esp32
pin: GPIO0
name: "Pulse Counter"
update_interval: 10s
total:
unit_of_measurement: 'l'
name: 'Total Water_new'
filters:
- multiply: 1
- lambda: |-
static float restored_value = NAN;
if (isnan(restored_value) && !isnan(id(last_usage)))
{
restored_value = id(last_usage);
ESP_LOGI("main", "Restored value: %f", restored_value);
}
return !isnan(restored_value) ? restored_value+x : NAN;
- filter_out: NAN
The first time you boot the ESP32 after flashing you have to push the initial value to the mqtt broker yourself (or change the yaml to do it somehow (use a button for that?)) like so:
Hi Daniel, thank you for your perfect analysis ! It gave me a lot of points to think about !
It seems there is plenty of options but no one is no ideal
Base on your findings I see following options (after I rejected complicated solutions ):
Store total counter value on SD card
Store total counter value in HA and in case that ESPHome total value resets to 0, HA will:
a) restore value in ESPHome
b) HA will continue counting total lifetime = total ESPHome + last stored value
I have on idea how to setup any of these
Any workarounds like MQTT, RTC, databases, Rieman integration (why to use integration to count integer values?) etc. are IMHO too complicated - I just need to simply count few values !
I found interesting that utility meter / daily shows 229 liters and total count 292 litres. I believe that is because some data transfer has been lost. It´s another reason why to have an opportunity to read stored values from ESPHome.
Related to previous comment regarding MQTT, I agree that it would be a solution how to prevent lost counts. Personally I would prefer to have an option in ESPHome pulse_counter to verify the data if they were received by HA and if not, to send them again. It does not make a sense to me to replace native communication between ESPHome and HA with another protocol because native communication does not work properly.
We had long discussion at our local Czech HA facebook group yesterday.
I got explained that utility meter does not work as expected - it counts values that are different from previous only so if there pulse input 2+2+2+4 it counts 2+4 only. This causes lost pulses.
The solution is to use total value from ESPHome - it is always growing so all pulses are counted properly.
No matter if some pulses are lost due to communication problem between ESP and HA as pulses are counted next time.
Utility_meter stores stores values internally so no doubts about lost pulses during restart.
Only the pulse lost will be there in case both ESP and HA will be restarted in one time or ESP will be offline. But there is not too much to do with it.
I run this solution since yesterday night till now and it seems all pulses have been counted properly !
This fixes troubles with utility meter counts.
The next step I would like to do is to setup lifetime meter in HA that will be synchronized with mechanical water meter.
Currenty I use year value from utility meter + helper manual input value = total lifetime meter.
This works for now but it is still workaround. I would like to setup a sensor variable that will count flow values from pulse meter without need to use utility meter. Any idea?
Hi @TomasCZ ! Could you share your final configuration in ESP and in HomeAssistant? I am trying to setup the same (water meter). Many thanks in advance!!
I believe there must be some functionality in HA that simply counts values coming from sensor. Something like counter sensor but able to increment the state with variable values.
Will post the automations below. Maybe somebody can help fixing it.