Fix bad data in energy tab

as i messed up a little with short term in sql already i had to finish it like that. but the second sensor from statistic tab went perfectly.
Anyway, all fine now, looking good again :slight_smile:
Thanks for all assistance!

EDIT,
I’ll just put a link here because it’s nicely described and will help before messing up with ale SQL mining

4 Likes

I’m sruggling with correcting the data :frowning:

What I have is Huawey SUN2000 PV inverter which gives power but not energy. For Energy i use integration platform and calclulate sensor_pv_energy:

  - platform: integration
    source: sensor.sun2000_10ktl_m1_sun2000_12_bt2150240752
    name: pv_energy
    unit_prefix: k
    round: 2

Now this night, my inverter sent some wrong wattage peak :

and because of that pv_energy sensor was calclulated wrong (this is already post db changes picture:

And of course energy dashboard is now off with a peak of 300kwh solar power.

I used this guide here and corrected the pv_energy values in:

  1. States
  2. statistics
  3. statistics_short_term

Everything was ok, but then probably integration platform calcluated again the difference and wrong numbers wehre in states, statistics_short_term and statistics again.

Then I thought that I should update the sensor.sun2000_10ktl_m1_sun2000_12_bt2150240752 wrong value as well, because integration platform uses it. Only place in db i saw these values, was states table. I corrected the data there, but in the UI it still shows the peak with wrong data and pv_energy is calclulated wrong again by integration platform.

So my question is, what to look more to solve this issue?

EDIT: Interesting is that i can’t see this wrong data in the dev toold statistic tab:


I have played with the hours and always 0. Maybe it’s because of it’s the “integration platform” sensor?

As you see in Developer tools you can correct the statistics, aka long term statistics.

If you corrected the statistics for your energy sensor then it should show the right values in your Energy tab.

You shouldn’t care about short term statistics as it will be discarded after a few days unless you are storing all values in an InfluxDB to display it with Grafana, then you should correct your value there.

Obviously your pv_energy short term statistics has been fudged not at 4AM, but after 8AM, so you have made some magic wrong there.

I did correct everything, but it came back. Even in statistics (not shot term):

So what i just did:

  1. Stopped Home Assistant docker

  2. states corrected

update states set state = round(state - 301.71, 1)
where 
  entity_id ='sensor.pv_energy'
  and state_id between 12580414 and 12580568;

Result: no wrong states:

  1. statistics_short_term corrected
update statistics_short_term set sum = sum -301.63
where 
  metadata_id == 8
  and id between 3990052 and 4001703;

and

update statistics_short_term set sum = sum -301.63
where 
  metadata_id == 8
  and id between 3990052 and 4001703;

Short term stats clean:

  1. statistics corrected
update statistics set sum = sum -301.63
where 
  metadata_id == 8
  and id between 333605 and 334505;

and

update statistics set state = state -301.63
where 
  metadata_id == 8
  and id between 333605 and 334505;

Result:

Checked PV states with a command:

SELECT *
FROM "states"
WHERE 
  entity_id ='sensor.sun2000_10ktl_m1_sun2000_12_bt2150240752'
  and last_updated between '2022-10-11 00:00:00' and '2022-10-12 23:59:00'

There is no wrong data there, as I cleaned it this morning.

I’ll start the HA now, and post what happens in db

1 Like

There it is again

If you see it there, then it means you can fix it through developer tools. You just need to find the hour where it is happening and set it to 0.

How does the short term looks?

just did it myself, works as a charm :wink:

You just have to find the sensor and the date🤔

I did as the instruction but no luck:

Screenshot 2022-10-12 at 13.46.41

I even scrolled through every hour that date, but no luck

I think you have the wrong sensor?
Try looking for Consumed Solar (if that is the one that’s wrong) :wink:

Its the excact senor:

As you can see from the picture, I changed the integration and from now on, i can probably correct the issues, as this integration offers energy sensor as well.

My guess is that i cannot correct, beaduse the old sensor was the integration sensor, which cannot be corrected:

  - platform: integration
    source: sensor.sun2000_10ktl_m1_sun2000_12_bt2150240752
    name: pv_energy
    unit_prefix: k
    round: 2

Bad thing is that i can’t delete the old sensor from Energy Dashboard, because i will lose all the previous data. So i just have to keep in mind that there is this 300kwh difference. One thing i can probably do is change the pv_energy senor to some static number and correct the bad data in db.

Have you changed the minutes as well? Because every time when you show this picture you show only the whole hour.

The data will be stored somewhere during the hour, not just at the start or end of the hour.

So i commented out the integration sensor in configruation.yaml , corrected the DB, switched to new pv integration and looks like it’s solved.

And yes, i changed minutes as well.

If it shows a spike there on the Energy Tab, that means it has a huge change there recorded in the long term statistics. It should be visible through the developer tools as well.

By the way, have you set the state of the sensor to the correct value in developer tools as well, before you started all the modifications?

My SUM tabel is ruined (maybe I did it myself using the query incorrectly).

But now need to fix it:

How to rebuild the SUM column based on the difference in de STATE column?
Any query to do this?

Thanks to this topic I managed to edit/recalc all my sum values by updating the sqlite stat tables. Now my reports finally make sense.

However, as soon as the hourly integration runs on my Electicity Meter Reader digitizer value, it adds the huge offset again once more in the ‘sum’ (not in the state, that one is correct, see below).
Even if I turn HA off at that time. Is this value stored somewhere? If so, where to change it?

## DIGITIZER METER READERS ###
- device_class: "energy"
  state_class: "total_increasing"
  name: Electricity Meter Reader
  state_topic: "digitizer/elecmeter/main/value"
  unit_of_measurement: "kWh"


image
image

If your sensor still has the bad value when you look at it you will need to also set the correct value with:

@ORi I had the problem… You can just edit the sensor value in the development tools → statistics

You search for the sensor you want to update and then click on the far right “sum adjustment” (not sure it is the exact wording as it is translated from the my french version “ajuster la somme”)… Then you look at the date and time where your sensor is going “banana” and adjust the value to the correct number… Done !

Normally all the statistics where this sensor is involved are automatically updated…

1 Like

Thanks for the suggestion @browetd. That’s actually not the problem here as you can see from below screenshot:

In fact the sensor value is completely correct, also the ‘adjust sum’ value is correct at that time. My best guess is that the integration that calculates the energy consumption over the entire hour somehow stores the incorrect sum somewhere.

EDIT1: i’ll experiment with the correcting the statistics_short_term table and provide feedback if it works.

EDIT2: it works, updating the ‘statistics_short_term’ table in the same way (SET sum = sum - $offset) resulted in the values being correct and the integrations calculate the power consumption correctly in the energy dashboard. No need to stop HA if it’s done this way.

Hopefully this helps anyone in the future as well.