Idea how to filter out SML Energy sensor glitches

[Mr_Photonics]

I am bit new to custom sensors in the configuration.yaml. However the recent increase in sensor glitches made me try to get a bit more into it.
For my current power consumption I wrote a static limit within the value has to be to be updated.
For increasing sensors like kWh counters, especially if they are used in the energy dash board is seems not that easy.
My first idea would be to compare it to a value what was recorded shortly before to check if it is not smaller or increased by an implausible amount. I saw an approach with trigger sensors to capture a value/state every few minutes.
What I tried to derive from all me web findings looks like this:

template:  
 -trigger:  
   - platform: time_pattern 
     minutes: "/5"
     sensor:
       - name: Prev_Grid_in
         entity_id: Prev_grid_in_5m
         unit_of_measurement: "kWh"
         state: "{{ states('sensor.tasmota_sml_total_in') | float }}" 
       
 -trigger:  
   - platform: time_pattern 
     minutes: "/5"
     sensor:
       - name: Prev_Grid_out
         entity_id: Prev_grid_out_5m
         unit_of_measurement: "kWh"
         state: "{{ states('sensor.tasmota_sml_total_out') | float }}" 
 
 -sensor:
   - name: Grid_Power_out  
     unique_id: Main_Grid_Out
     unit_of_measurement: "kWh"
     device_class: energy
     stats_class: total_increasing
     state: >
     { % if  float(states('trigger.Prev_grid_out_5m')) < float(states('sensor.tasmota_sml_total_out')) < (float(states('trigger.Prev_grid_out_5m')) + 1)
     {{states('sensor.tasmota_sml_total_out')}}
     {% endif %}
   
   
 -sensor:
   - name: Grid_Power_in  
     unique_id: Main_Grid_In
     unit_of_measurement: "kWh"
     device_class: energy
     stats_class: total_increasing
     state: >
     { % if  float(states('trigger.Prev_grid_in_5m')) < float(states('sensor.tasmota_sml_total_in')) < (float(states('trigger.Prev_grid_in_5m')) + 3)
     {{states('sensor.tasmota_sml_total_in')}}
     {% endif %}
       

In short: The actual sensor values are read every 5min as reference. Then the sensor template checks if the SML value is bigger that the reference but also not bigger by more than 1 or 3 kWh. Even if a glitch was captured as reference, the worse thing that can happen is that the sensor won’t display a new value/state for 5 min.
Since I run HA core and can’t access the yaml that easy I was hoping to get an opinion and error correction before I cripple my HA

[tom_l]

You are missing spaces between the dashes and sensor/trigger. e.g.

-trigger:

Should be:

- trigger:

As well as this the {% if %} test requires an {% else %} case. This is not optional.

In short what you have is not good.

What sort of “glitches” are you getting with your energy sensor?

What integration is providing these sensors?

You may be able to solve the issue with an availability template, or the integration may need issues fixed.

e.g. where does this sensor come from, sensor.tasmota_sml_total_out ?

Is it a template sensor?

[Mr_Photonics]

Hello Tom,
thanks for your reply.
The sensors tasmota_sml_total_out and _in are provided by an ESP based power sensor that is connected to my main power meter in the house. The information is then provided as an sensor via the Tasmota integration.
I don’t know were this error originates since the main power meter does not have any glitches.
The glitches are random values ranging from 0.000 to 99999999999.999, however it can also be a number like 250.258. This makes it difficult to work with fixed limit to accept data. At the moment the power meter has around 450kWh, but it will increase over time, and so the limits would have to adapt.
Therefor I just wanted to compare the current value with a value it read 5 minutes ago.

The value in the energy dashboard basically sums up all the changes in between the sensors grid_in and _out (kWh) readings within 1 or 2 hours. Sensor glitches therefor produce insane high energy consumptions (kWh).
At no time the state/value is unknown or unavailable.

This is how is should work.

• read the state value of the tasmota sensors in and out every 5 min, e.g. 455.123
• if the tasmota sensor value changes the template sensor should check if the new value is bigger than the value 5min ago. On the other hand the value cannot be bigger than the 5min reference value + 3 kWh (because I technically cannot draw more Energy in 5 minutes).

[tom_l]

It is going to be very difficult to filter out those glitches. You need to fix the sensor.

Sounds to me like there is a problem with Tasmota or its configuration.

Searching their issues I discovered this:

SML (Smart Meter) does not check checksum?! · arendst/Tasmota · Discussion...

So they seem to be refusing to implement the checksum (which would reject invalid readings) and request that you check your serial installation.

Seems like a terrible answer to me. Any brief understanding of Shannon’s Theorem would tell you that you can not eliminate noise in a communication channel completely. Using the checksum is wise. All sorts of impulse noise could exist near your power switchboard that could cause bad data.

Try to eliminate this by checking your installation, in particular:

1. positioning of the IR head
2. use the shortest serial cabling possible, consider extending the power cabling instead.
3. use screened serial cable grounded at one end.
4. consider using ESPHome instead. Particularly if you have a Holley DTZ541 smart meter. SML (Smart Message Language) — ESPHome

If you still have the problem, you could try joining the Tasmota discussion to try and convince them that this is an issue for more than one person.

[Mr_Photonics]

Yeah, as assumed there is no error correction in the transmission…
Wifi is stable, position is good.

Well, I am still wondering if there could be a dirty fix for now.
Where I am stuck now is actually how to write the current sml_total_* state into prev_grid_in_5m or something i can access in the sensor grid_power-* state calculations.

The trigger might be triggering by time pattern, but I don’t get a entity_id to access.

Long term solution might be to get a shelly…

[tom_l]

It’s the serial connection that is the issue not the wifi. Shorten the leads. Screen the cable.

[Mr_Photonics]

I see… it is already a compact module, but I just disassembled everything and cleaned/bend the contacts. Keep fingers crossed

[Mr_Photonics]

Sadly no luck so far. I partially re-soldered the whole thing but no significant changes. I also removed all other Mqtt traffic that could cause some trouble on my server side. But since I saw the glitches also on the Tasmota web interface, it is clear that the glitches must occur directly on the read-out side.

For now the only solution is to work with tight limits, but any dynamic filter would be better. Is there no way to implement a filter (or something like my first idea) in the yaml?
Another work around might rely on statistics. Right now the value is updating every few seconds. If a value is fetches every 5min, the chance of fetching a glitch would be at least a bit lower.
So I am still open for all temporary work arounds better than this:

 - sensor:
   - name: Grid_Power_out  
     unique_id: Main_Grid_Out
     unit_of_measurement: "kWh"
     device_class: energy
     stats_class: total_increasing
     state: >
      {% if 220 < float(states('sensor.tasmota_sml_total_out')) < 750 %}
      {{states('sensor.tasmota_sml_total_out')}}
      {% else %}
      {% endif %}

EDIT:
I found filters and created:

sensor:
  - platform: filter
    name: Grid_Power_in_filtered
    entity_id: sensor.tastmota_sml_total_in
    filters:
     - filter: outlier
       window_size: 4.0
       radius: 10.0

this seems to do what I want. So I created a new template sensor from that value [ state: filter…] to get into the device class energy with the correct unit of measurement.
I hope this will work …

EDIT2
following settings work perfect so far:

 window_size: 30.0
 radius: 1.0

[PeteRage]

FWIW, I put filters on lots of sensors. Weird stuff happens, software and hardware have bugs…even the sensors that don’t may