Sensor normalization using filters

Configuration
1

I have a question on a specific sensor, its my total kWh yield sensor.

Actually I am using this sensor as source for the energy dashboards,which is doing fine, with one exception.

This sensor is a calculation of 2 solar systems, which I am adding both to this new sensor.
However, one of the source sensors is coming from the internet (GoodWe portal) and sometimes this portal is not reachable for a couple of minutes. What happens then, is that the total kWh yield sensor is not accurate, its just decremeting the value of the GoodWe source, since it cannot fetch that information.

I am looking for a way to fix this, either just not add the wrong values or something else.
I saw the Filter options, but these seem quite complicated, probably somebody can help me, for example with a little piece of configuration of a similar issue.

To visualize: the picture below shows to 4 spikes down, those are the moments the GoodWe portal was not reachable for a short periode of time. I would like to see a flat line increasing everyday with the solar yield.

sensoree

1 Like
2

Post the configuration you have for this sensor.

Also, which integration is used for the sensor that gets data from GoodWe? Is it a REST sensor or something else?

3

was looking for a solution to that too, unfortunately we’re on our own I fear: https://github.com/home-assistant/core/issues/55565

and I am not sure the filter option filters these spikes, as I read that documentation, it will smooth them, which isnt what we need of course…

4

It appears that japie101 is using a Template Sensor to calculate the sum of the two solar systems. There may be an opportunity to mitigate the problem with the template or the availability option. However, more information is needed before proceeding.

I read your link and I agree there should be a way to delete bad long-term data that skews the statistics. Without that ability, it remains on record indefinitely.

5

yes, and I just read a post by Balloob that we can not do that from the UI, and need to go into the Sql tools for that… Discord

dixit balloob:

not possible via the UI, you can always go deleting rows via SQL

hardly a user friendly method, for a system that is as glitchy as it is in real life. My energy sensors are mostly Z-wave, and experience with z-wave is not that positive at all, with frequent resets and spikes/dips as displayed. Added to that the z-wave mesh network is very difficult to analyze, so trying to find which interference is causing the communication issues is practically impossible.

I would hope HA will find a way to deal with these random disturbances in the near future. The graphs would indicate these issues could be determined quite easily?

6 
 lifetime_energy_combined_kwh:
        value_template: "{{ ((states('sensor.ecu_lifetime_energy') | float ) + (states('sensor.pv_etotal') | float)) | round(1) }}" 
        unit_of_measurement: 'kWh'
        friendly_name: "Lifetime opbrengst kWh"

The Goodwe data comes from the GoodWe SEMS API integration

7 
  - platform: filter
    name: "Filtered Lifetime opbrengst kWh"
    entity_id: sensor.lifetime_energy_combined_kwh
    filters:
      - filter: outlier
        window_size: 4
        radius: 40

I created this filter, which is working somewhat, but it is replacing a spike with a median value.
This is better than before, but there should be an addition to the outlier filter containing just do nothing with an outlier value. (no replacement of the value)

8

Which one of the two sensors is the GoodWe system? What does that sensor report as its value when it fails to get data from the GoodWe API? Is it unknown or 0 or something else?

The goal here is not to mathematically ‘smooth out’ the incorrect value but to ignore it completely and use the previous good value.

9

maybe we can use the phpMyAdmin add-on, to edit the SQL database, but I wouldn’t know where to start looking… it is rather daunting, and my SQL days are way behind my memory :wink:

10

Hi thanks for your reply.
The GoodWe sensor is: sensor.pv_etotal

This is how this individual sensor looks like:
Seems its just skipping the value for a period of time?

goodwe sensor

11

I need to know what is the value it reports each time a gap appears in the graph. Look for that information in the Logbook.

It shouldn’t be too difficult to find. Look for it around 08:00 today.

12

I checked the logbook but it did not contain any info on this sensor.

Probably I am looking at the wrong logbook or I should enable logging?

13

I did some extra investigation, to be more detailed: I installed the community addon SQLite web which lets you discover the Home Assistant database.

Using a simple SQL query:

SELECT *
FROM "states"
WHERE entity_id is "sensor.pv_etotal"

I get the exact values written in the database.

When the value is not available, I see either two options: just a blank entry, or unknown is displayed.

sensor_blank
sensor_blank929×372 39.9 KB

sensor_unknown
sensor_unknown928×470 50.1 KB

14

In that case, sensor.lifetime_energy_combined_kwh should be considered as being unavailable when the value of sensor.pv_total is either unknown or blank (an empty string).

The following template reports true when pv_total is not unknown or blank. Otherwise it reports false.

{{ not states('sensor.pv_etotal') in ['unknown', ''] }

Use it with the availability_template option. When the template reports false, the computed value should will not be used and it will report its previous value.

      lifetime_energy_combined_kwh:
        value_template: "{{ [states('sensor.ecu_lifetime_energy'), states('sensor.pv_etotal')] | map('float') | sum | round(1) }}"
        unit_of_measurement: 'kWh'
        friendly_name: "Lifetime opbrengst kWh"  
        availability_template: "{{ not states('sensor.pv_etotal') in ['unknown', ''] }}"