How to count pulse frequency accurately with ESPHome?

stevebaxter · December 24, 2020, 12:02pm

I’m just going round this loop as well. I have a NodeMCU with two inputs, one from a hall effect sensor on my gas meter (which outputs one pulse per litre), and one from a photoresistor on the electricity meter (which outputs 1000 pulses per kWh, so 1 per Wh).

The problem is that using pulse_counter, with an update every minute, the resolution of the electricity meter is limited to 60W. To get to 1W of power resolution I would need to integrate over an hour which would give very poor temporal resolution.

Ideally, I think I want to do two things:

Measure the pulse width (both sensors are active low, so we can measure the width of the “high” state to give the time between pulses). This needs to output on every pulse, not at a fixed time interval (because we know the power at every pulse).
Count the total pulses - this gives an accurate total usage figure, e.g. for echoing the meter reading

Unfortunately, neither pulse_width or pulse_counter quite do what I need. pulse_width doesn’t seem to output on every pulse, pulse_counter doesn’t actually count the total pulses (though I see that https://github.com/esphome/esphome/pull/1173 has been merged but not yet released which should do this). Also, using pulse_counter and pulse_width on the same pin doesn’t seem to work (if you use pulse_width, the counter doesn’t see any pulses).

I’m thinking of writing a new sensor called pulse_meter that will give both an instantaneous measurement (by measuring the time between edges rather than the pulse width and outputting on every pulse) and also count the total pulses. Before I start tinkering with this, is there another way of doing this that I might have missed?

JeeCee · December 25, 2020, 11:25am

Here is what I did:

globals:
# https://esphome.io/guides/automations.html#bonus-2-global-variables
- id: us_previousTriggerTime
  type: time_t
  restore_value: no
  initial_value: '0'
- id: us_pulsetime
  type: float
  restore_value: no
  initial_value: '0'
- id: us_power
  type: float
  restore_value: no
  initial_value: '0'

binary_sensor:
  - platform: gpio
    name: "Elecmeters US - Pulse"
    id: us_pulse
    pin:
      number: 27
      mode: INPUT_PULLUP
      inverted: yes
    # https://community.home-assistant.io/t/publish-timestamp-into-text-sensor/122531/10
    # 1 'unit'/hr = 3600 / pulsewith (seconds)
    # 1000 pulses = 1kWh
    # 1 pulse = 1 Wh
    # 1 pulse per 'unit' (Wh)
    # 3600/1 = 3600
    # Max power 17250W (25A)
    on_press:
      then:
        - sensor.template.publish:
            id: template_us_electricity_power
            state: !lambda |-
              auto currTime = millis();
              id(us_pulsetime) = (float) (currTime - id(us_previousTriggerTime));
              id(us_previousTriggerTime) = currTime;
              id(us_power) = 3600000.0 / id(us_pulsetime);
              if (id(us_power) < 18000) return id(us_power);
              else return {};

(Code borrowed and adapted from esp_easy forums).

I used to have a service call in this code block to increase a counter as well, keeping track of consumption (kWh).

However, it turns out that apparently the gpio platform does not utilise interrupts, cause I was seeing a discrepancy between my counter and the meter in the electricity cabinet.
I can even observe this with the kW sensor, with high loads (10kW) it sometimes drops to 5kW briefly, indicting a missed pulse.
I opened an issue on the ESPHome repo, but it received 0 replies…

So I added the pulsecounter platform to count pulses (kWh) and accepted the fact that the power sensor sometimes misses a pulse.

Sidenote 1:
Using a service call directly to HAS avoids loosing pulses when the ESP reboots (I want to avoid writing to memory in ESPhome, prolonging flash life), but now I have the problem I’m loosing pulses when HAS reboots. I am building an automation which at the end of the day compares daily pulses from HAS and ESP, and increases the counter with the difference.
EDIT: another downside of directly incrementing a HAS counter is that during WiFi disconnects (a rather common occurrence on ESPs) we miss pulses as well.

Sidenote 2:

Also, using pulse_counter and pulse_width on the same pin doesn’t seem to work (if you use pulse_width , the counter doesn’t see any pulses).

I have noticed this behavior on an ESP8266 as well when using the gpio and pulsecounter platform for the same pin. On an ESP32 this works.
A jumper wire solves this problem, but sacrifices an input pin (which are scarce on the 8266).

stevebaxter · December 26, 2020, 12:49pm

Thanks for the info! The GPIO component just uses a loop to monitor the input so is pretty much guaranteed to miss some pulses, particularly if they are short. We would need to use an interrupt (like the pulse counter component) to ensure that the pulses are detected correctly (along with the debouncing that the pulse counter provides).

I will hopefully have time to look at the C++ code over the next few days and put something together.

JeeCee · December 26, 2020, 2:45pm

That would be great! Are you a ESPHome contributor?

stevebaxter · December 27, 2020, 6:08pm

Not yet! I will submit this back to the project though.

I have this working, just testing a bit and then I’ll post it here for people to play with.

stevebaxter · December 27, 2020, 6:36pm

OK, the attached code implements a new sensor called pulse_meter. This has a very similar setup to pulse_counter, with two key differences:

It measures the time between the pulses and outputs a measurement for every pulse rather than a measurement every minute. This gives much better resolution.
It has an additional “total” sensor that outputs the total number of pulses (this is the same as the current development branch of pulse_counter).

These two together allow you to get the best possible temporal resolution (a measurement for every pulse) and accurate totals (as we can really count the pulses rather than trying to integrate the frequency).

Here’s an example config for my electricity meter that outputs 1000 pulses per kWh:

sensor:
  - platform: pulse_meter
    name: "Electricity Meter"
    pin: D5
    unit_of_measurement: "kW"
    internal_filter: 5us
    accuracy_decimals: 3
    timeout: 5min
    total:
      name: "Electricity Meter Total Count"
    filters:
      - multiply: 0.06

The setup is much the same as pulse_counter with a couple of changes:

You can’t specify rising or falling edges, we always look for rising edges (you can invert the pin if you want falling edges for some reason). In particular we can’t count both rising and falling, that doesn’t make sense for this sensor.
I’ve added a timeout value, this defaults to 5 mins. If we don’t see a pulse for this amount of time we assume 0 pulses/min. If we don’t do this then we get “stuck” on the previous measured value if the pulses stop (because we normally update the value on each pulse).

Here is some output showing pulse_meter vs pulse_counter:

You can see that the new sensor (blue) tracks the usage much more quickly than the pulse counter (red). In particular it shows spikes that the pulse counter will miss, and doesn’t hunt up and down (see the period from 14:39 to 15:09 as a good example). It does however track the old sensor pretty well which is reassuring.

You can download the new sensor here:

Unpack it into your esphome directory (this will be in the config directory for hassio).

DarthMob · January 4, 2021, 8:52pm

Thanks @stevebaxter for the improvement on the counter!

Sadly it does not work on my side.
I also have a counter 1 pulse = 1 Wh
If i integrate the sensor 1:1 copy of our senor code i have 2 issues:

the “Electricity Meter” continuously shows 0
the “Electricity Meter Total Count” increases much too fast (i only have like 1 kW running)

any idea what is happening here?

Thanks!

stevebaxter · January 5, 2021, 12:57pm

Hmm, that’s odd! Can you share the output from the ESPHome device log? That will give me the full config and allow me to see exactly what is happening.

One thing I had to do was to use much longer values for the “internal filter” (around half the minimum pulse width), as noise on the pulses will cause very high readings. Even then I had a problem with occasional spikes, I used a median filter to remove that. That doesn’t explain what you’re seeing though.

This is my full sensor config for reference:

sensor:
  - platform: pulse_meter
    name: "Electricity"
    pin: D5
    unit_of_measurement: "kW"
    internal_filter: 100ms
    accuracy_decimals: 3
    timeout: 2 min
    filters:
      # Filter outliers
      - median:
          window_size: 3
          send_every: 1
          send_first_at: 1
      # Convert pulses/min (Wh/min) to kW
      - multiply: 0.06
    total:
      name: "Electricity Meter"
      unit_of_measurement: "kWh"
      accuracy_decimals: 3
      filters:
        - multiply: 0.001

  - platform: pulse_meter
    name: "Gas"
    pin: D6
    unit_of_measurement: "kW"
    internal_filter: 4s
    accuracy_decimals: 3
    timeout: 2 min
    filters:
      # Convert litres (one pulse) to kWH/min using:
      #
      # 1.022640 * 39.2 / 3.6 / 1000 = 0.01113541333
      #
      # Then kWH/min to kW by multiplying by 60 = 0.66812
      - multiply: 0.66812
    total:
      name: "Gas Meter"
      unit_of_measurement: "m³"
      accuracy_decimals: 3
      filters:
        - multiply: 0.001

For the electricity meter my filter is 100ms (5 pulses a second is 18kW which is way more than our house would ever use). For gas it’s even longer (4 seconds).

I’ve updated the custom components archive with the latest code (there were some changes which were only cosmetic but maybe I missed something). You can find it at the same URL (https://www.dropbox.com/s/6tn8th7mmtvwlup/custom_components.zip?dl=0).

stevebaxter · January 5, 2021, 1:00pm

Actually, are you sure your multipliers are right in the filter? Looking at your graph, is that over 5 seconds or 5 mins? If it’s 5 mins there appear to be 14-15 steps each min, that’s 840-900/hour or nearly 1 kW which sounds about right. The raw data from the log will tell us though!

DarthMob · January 5, 2021, 6:39pm

Thank you @stevebaxter!

With the new code it is working (don’t ask me why )

I have this energy counter (sorry only found it in German) - 1000 imp/kWh
SDM72D

Log output for reference:

[19:33:28][C][logger:185]: Logger:
[19:33:28][C][logger:186]:   Level: DEBUG
[19:33:28][C][logger:187]:   Log Baud Rate: 115200
[19:33:28][C][logger:188]:   Hardware UART: UART0
[19:33:28][C][pulse_meter:052]: Pulse Meter 'Electricity'
[19:33:28][C][pulse_meter:052]:   Unit of Measurement: 'kW'
[19:33:28][C][pulse_meter:052]:   Accuracy Decimals: 3
[19:33:28][C][pulse_meter:052]:   Icon: 'mdi:pulse'
[19:33:28][C][pulse_meter:053]:   Pin: GPIO12 (Mode: INPUT)
[19:33:28][C][pulse_meter:054]:   Filtering pulses shorter than 100000 µs
[19:33:28][C][pulse_meter:055]:   Assuming 0 pulses/min after not receiving a pulse for 120s
[19:33:28][C][homeassistant.time:010]: Home Assistant Time:
[19:33:28][C][homeassistant.time:011]:   Timezone: 'CET-1CEST-2,M3.4.0/2,M10.5.0/3'
[19:33:28][C][captive_portal:169]: Captive Portal:
[19:33:28][C][ota:029]: Over-The-Air Updates:
[19:33:28][C][ota:030]:   Address: stromzaehler.local:8266
[19:33:28][C][api:095]: API Server:
[19:33:28][C][api:096]:   Address: stromzaehler.local:6053
[19:33:31][D][sensor:092]: 'Electricity Meter': Sending state 0.00100 kWh with 3 decimals of accuracy
[19:33:31][D][api.connection:604]: Client 'Home Assistant 2020.12.2 (192.168.123.60)' connected successfully!
[19:33:31][D][time:040]: Synchronized time: Tue Jan  5 19:33:31 2021
[19:33:38][D][sensor:092]: 'Electricity': Sending state 0.24951 kW with 3 decimals of accuracy
[19:33:38][D][sensor:092]: 'Electricity Meter': Sending state 0.00200 kWh with 3 decimals of accuracy
[19:33:45][D][sensor:092]: 'Electricity': Sending state 0.49903 kW with 3 decimals of accuracy
[19:33:45][D][sensor:092]: 'Electricity Meter': Sending state 0.00300 kWh with 3 decimals of accuracy
[19:33:52][D][sensor:092]: 'Electricity': Sending state 0.50293 kW with 3 decimals of accuracy
[19:33:52][D][sensor:092]: 'Electricity Meter': Sending state 0.00400 kWh with 3 decimals of accuracy
[19:33:59][D][sensor:092]: 'Electricity': Sending state 0.50293 kW with 3 decimals of accuracy
[19:33:59][D][sensor:092]: 'Electricity Meter': Sending state 0.00500 kWh with 3 decimals of accuracy
[19:34:07][D][sensor:092]: 'Electricity': Sending state 0.49924 kW with 3 decimals of accuracy
[19:34:07][D][sensor:092]: 'Electricity Meter': Sending state 0.00600 kWh with 3 decimals of accuracy
[19:34:13][D][sensor:092]: 'Electricity': Sending state 0.49924 kW with 3 decimals of accuracy
[19:34:13][D][sensor:092]: 'Electricity Meter': Sending state 0.00700 kWh with 3 decimals of accuracy
[19:34:19][D][sensor:092]: 'Electricity': Sending state 0.54836 kW with 3 decimals of accuracy
[19:34:19][D][sensor:092]: 'Electricity Meter': Sending state 0.00800 kWh with 3 decimals of accuracy
[19:34:26][D][sensor:092]: 'Electricity': Sending state 0.57398 kW with 3 decimals of accuracy
[19:34:26][D][sensor:092]: 'Electricity Meter': Sending state 0.00900 kWh with 3 decimals of accuracy
[19:34:32][D][sensor:092]: 'Electricity': Sending state 0.57398 kW with 3 decimals of accuracy
[19:34:32][D][sensor:092]: 'Electricity Meter': Sending state 0.01000 kWh with 3 decimals of accuracy
[19:34:38][D][sensor:092]: 'Electricity': Sending state 0.57361 kW with 3 decimals of accuracy
[19:34:38][D][sensor:092]: 'Electricity Meter': Sending state 0.01100 kWh with 3 decimals of accuracy

Thanks again for the code - looking forward to using this module for energy counting

stevebaxter · January 5, 2021, 8:07pm

Thanks for replying! Possibly there was some bug in the original code, apologies for that. Glad it’s working!

depen · January 6, 2021, 8:38pm

I´m also trying your custom component. It validates just fine with Esphome v1.15.3, but when I compile it I get some errors:

In file included from src/main.cpp:44:0:
src/pulse_meter_sensor.h:11:7: error: redefinition of 'class esphome::pulse_meter::PulseMeterSensor'
 class PulseMeterSensor : public sensor::Sensor, public Component {
       ^
In file included from src/esphome.h:23:0,
                 from src/main.cpp:3:
src/esphome/components/pulse_meter/pulse_meter_sensor.h:11:7: error: previous definition of 'class esphome::pulse_meter::PulseMeterSensor'
 class PulseMeterSensor : public sensor::Sensor, public Component {
       ^
*** [/data/esp32_01/.pioenvs/esp32_01/src/main.cpp.o] Error 1

I have put pulse_meter_sensor.h, pulse_meter_sensor.cpp and sensor.py in
the directory: config/esphome/custom_components/pulse_meter/

Code snippet:

esphome:
  name: ${esp_name}
  platform: ESP32
  board: mhetesp32devkit
  includes:
    - custom_components/pulse_meter/pulse_meter_sensor.h

  - platform: pulse_meter
    name: "main_power_usage"
    pin: GPIO12
    unit_of_measurement: "kW"
    internal_filter: 100ms
    accuracy_decimals: 3
    timeout: 2 min
    filters:
      # Filter outliers
      - median:
          window_size: 3
          send_every: 1
          send_first_at: 1
      # Convert pulses/min (Wh/min) to kW
      - multiply: 0.06
    total:
      name: "main_energy_usage"
      unit_of_measurement: "kWh"
      accuracy_decimals: 3
      filters:
        - multiply: 0.001

What am I doing wrong ??

stevebaxter · January 6, 2021, 10:40pm

Can you check you definitely haven’t got pulse_meter_sensor.h anywhere else in your config directory? It looks like two versions are being included from different directories.

depen · January 7, 2021, 5:35pm

Yes, the three files are only located in
config/esphome/custom_components/pulse_meter/

I made a clean build but it didn´t help. I run the esphome add-on from the HA community, but it is in sync with the Esphome repository and shouldn´t matter…?

stevebaxter · January 7, 2021, 8:11pm

Ah the problem is the:

  includes:
    - custom_components/pulse_meter/pulse_meter_sensor.h

This is already in the custom_components folder, so it will just work without that line. Delete it and you should be good!

depen · January 7, 2021, 9:13pm

Yes, Thanks! Unfortunately I got a new error

Linking /data/esp32_01/.pioenvs/esp32_01/firmware.elf
/data/esp32_01/.pioenvs/esp32_01/src/pulse_meter_sensor.cpp.o: In function `esphome::pulse_meter::PulseMeterSensor::gpio_intr(esphome::pulse_meter::PulseMeterSensor*)':
/config/esphome/esp32_01/src/pulse_meter_sensor.cpp:58: multiple definition of `esphome::pulse_meter::PulseMeterSensor::gpio_intr(esphome::pulse_meter::PulseMeterSensor*)'
/data/esp32_01/.pioenvs/esp32_01/src/esphome/components/pulse_meter/pulse_meter_sensor.cpp.o:/config/esphome/esp32_01/src/esphome/components/pulse_meter/pulse_meter_sensor.cpp:58: first defined here
/data/esp32_01/.pioenvs/esp32_01/src/pulse_meter_sensor.cpp.o: In function `esphome::pulse_meter::PulseMeterSensor::setup()':
pulse_meter_sensor.cpp:(.text._ZN7esphome11pulse_meter16PulseMeterSensor5setupEv+0x0): multiple definition of `esphome::pulse_meter::PulseMeterSensor::setup()'
/data/esp32_01/.pioenvs/esp32_01/src/esphome/components/pulse_meter/pulse_meter_sensor.cpp.o:pulse_meter_sensor.cpp:(.text._ZN7esphome11pulse_meter16PulseMeterSensor5setupEv+0x0): first defined here
/data/esp32_01/.pioenvs/esp32_01/src/pulse_meter_sensor.cpp.o: In function `non-virtual thunk to esphome::pulse_meter::PulseMeterSensor::setup()':
pulse_meter_sensor.cpp:(.text._ZThn160_N7esphome11pulse_meter16PulseMeterSensor5setupEv+0x0): multiple definition of `non-virtual thunk to esphome::pulse_meter::PulseMeterSensor::setup()'
/data/esp32_01/.pioenvs/esp32_01/src/esphome/components/pulse_meter/pulse_meter_sensor.cpp.o:pulse_meter_sensor.cpp:(.text._ZThn160_N7esphome11pulse_meter16PulseMeterSensor5setupEv+0x0): first defined here
/data/esp32_01/.pioenvs/esp32_01/src/pulse_meter_sensor.cpp.o: In function `esphome::pulse_meter::PulseMeterSensor::dump_config()':
pulse_meter_sensor.cpp:(.text._ZN7esphome11pulse_meter16PulseMeterSensor11dump_configEv+0x0): multiple definition of `esphome::pulse_meter::PulseMeterSensor::dump_config()'
/data/esp32_01/.pioenvs/esp32_01/src/esphome/components/pulse_meter/pulse_meter_sensor.cpp.o:pulse_meter_sensor.cpp:(.text._ZN7esphome11pulse_meter16PulseMeterSensor11dump_configEv+0x0): first defined here
/data/esp32_01/.pioenvs/esp32_01/src/pulse_meter_sensor.cpp.o: In function `non-virtual thunk to esphome::pulse_meter::PulseMeterSensor::dump_config()':
pulse_meter_sensor.cpp:(.text._ZThn160_N7esphome11pulse_meter16PulseMeterSensor11dump_configEv+0x0): multiple definition of `non-virtual thunk to esphome::pulse_meter::PulseMeterSensor::dump_config()'
/data/esp32_01/.pioenvs/esp32_01/src/esphome/components/pulse_meter/pulse_meter_sensor.cpp.o:pulse_meter_sensor.cpp:(.text._ZThn160_N7esphome11pulse_meter16PulseMeterSensor11dump_configEv+0x0): first defined here
/data/esp32_01/.pioenvs/esp32_01/src/pulse_meter_sensor.cpp.o: In function `esphome::pulse_meter::PulseMeterSensor::loop()':
pulse_meter_sensor.cpp:(.text._ZN7esphome11pulse_meter16PulseMeterSensor4loopEv+0x0): multiple definition of `esphome::pulse_meter::PulseMeterSensor::loop()'
/data/esp32_01/.pioenvs/esp32_01/src/esphome/components/pulse_meter/pulse_meter_sensor.cpp.o:pulse_meter_sensor.cpp:(.text._ZN7esphome11pulse_meter16PulseMeterSensor4loopEv+0x0): first defined here
/data/esp32_01/.pioenvs/esp32_01/src/pulse_meter_sensor.cpp.o: In function `non-virtual thunk to esphome::pulse_meter::PulseMeterSensor::loop()':
pulse_meter_sensor.cpp:(.text._ZThn160_N7esphome11pulse_meter16PulseMeterSensor4loopEv+0x0): multiple definition of `non-virtual thunk to esphome::pulse_meter::PulseMeterSensor::loop()'
/data/esp32_01/.pioenvs/esp32_01/src/esphome/components/pulse_meter/pulse_meter_sensor.cpp.o:pulse_meter_sensor.cpp:(.text._ZThn160_N7esphome11pulse_meter16PulseMeterSensor4loopEv+0x0): first defined here
collect2: error: ld returned 1 exit status
*** [/data/esp32_01/.pioenvs/esp32_01/firmware.elf] Error 1

stevebaxter · January 7, 2021, 9:54pm

Try cleaning the build files (… menu and “Clean build files”). Things might be a bit messed up.

DarthMob · January 8, 2021, 7:23pm

For reference if interesting for others:

I wanted to show the daily value for the power usage (with daily reset).

I tried doing this with the Total Daily Energy Sensor from ESPHome (basically integration of kW over time)

This however results in ~400 Wh more on a day where i used 6 kWh. (see blue graph).

I then used the HA Utility Meter on the Pulse/ kWh sensor (red graph).
This then returns the correct value.

mhooghiemstra · January 26, 2021, 11:54am

@stevebaxter First of all thanks for the code and your explanations in this thread. It helped met to setup a puls meter for my Solar Panels.
I was wondering if you could help me with one issue I have. Sometimes I get big spikes.
I already set my internal_filter to 500ms which helps a little. When I set a median the spikes are gone but then the out ut never reaches 0, it hovers around 36W.

Could you help me remove the spikes but still get the output to 0. What am I missing?

massive · January 27, 2021, 4:39pm

Just chipping in here to thank @stevebaxter! I found your pulse meter quite accurate. I’ve built a power monitor using M5Stick and accompanied LDR that reads impulses from a led flashes our electricity meter generates.

It is surprising that pulse meter was so much more accurate than built in pulse counter. In fact, pulse counter reported values that were off over 30% from the actual numbers.

I’m comparing the numbers to my electricity distributor numbers that I can check from their app a few days after the fact.

So just saying that I appreciate your work, and as it seems to work so good, ask if it could be merged to the esphome main line?

I had a few issues here and there. I also got huge spikes which I had to filter by tweaking the median filter and changing input pin to inverted. Additionally, I’m seeing some jitter in the power numbers, but I suspect that it’s probably some underfloor heating system going on and off. This needs to be verified.