Check the history of your sensor, does it drop to zero then back? If yes, you need to look at your MQTT sensor configuration, specifically what happens if the data in the topic is invalid. It should make the sensor unavailable. If it makes the value 0, that will be built into your results. Can you post your discovery code for the MQTT sensor or it’s configuration?
The configuration for the spike I have today is:
- platform: mqtt
name: _some name here_
state_topic: SmartMeter_day_used_kWh
unit_of_measurement: kWh
state_class: total_increasing
device_class: energy
here are the records that are off in the statistics_short_term table:
[
{
"id": 877930,
"created": "2022-03-25 09:25:11.997609",
"start": "2022-03-25 09:20:00",
"mean": null,
"min": null,
"max": null,
"last_reset": null,
"state": 2688.247,
"sum": 10921.678999999887,
"metadata_id": 12
},
{
"id": 877883,
"created": "2022-03-25 09:20:12.552433",
"start": "2022-03-25 09:15:00",
"mean": null,
"min": null,
"max": null,
"last_reset": null,
"state": 2688.247,
"sum": 10921.678999999887,
"metadata_id": 12
},
{
"id": 877836,
"created": "2022-03-25 09:15:11.043967",
"start": "2022-03-25 09:10:00",
"mean": null,
"min": null,
"max": null,
"last_reset": null,
"state": 2688.247,
"sum": 8233.431999999888,
"metadata_id": 12
},
{
"id": 877789,
"created": "2022-03-25 09:10:10.314489",
"start": "2022-03-25 09:05:00",
"mean": null,
"min": null,
"max": null,
"last_reset": null,
"state": 2688.247,
"sum": 8233.431999999888,
"metadata_id": 12
}
]
and this is the statistics table
[
{
"id": 73364,
"created": "2022-03-25 10:00:11.896423",
"start": "2022-03-25 09:00:00",
"mean": null,
"min": null,
"max": null,
"last_reset": null,
"state": 2688.289,
"sum": 10921.720999999889,
"metadata_id": 12
},
{
"id": 73317,
"created": "2022-03-25 09:00:14.952358",
"start": "2022-03-25 08:00:00",
"mean": null,
"min": null,
"max": null,
"last_reset": null,
"state": 2688.247,
"sum": 8233.431999999888,
"metadata_id": 12
}
]
And this is the meta information for that sensor:
[
{
"id": 12,
"statistic_id": "sensor.total_day_usage",
"source": "recorder",
"unit_of_measurement": "kWh",
"has_mean": 0,
"has_sum": 1,
"name": null
}
]
Is this the information you’re looking for?
what’s in the state topic? There’s nothing that explains to HA how to take the value from that topic and use it. I.e. i’ts missing a value_template
I’m interested in the state changes that home assistant sees. This is the information that’s a result of the home assistant state changes, so it’s the wrong table.
I can’t immediately tell where to get that mqtt history information, I can look into adding some kind of firehose on that topic so that I can read through the data and verify if there’s anything incorrect. Or is there a way to retrieve historical information through another way, is home assistant storing this somewhere?
In terms of configuration, I believe I followed the setup information when this feature was launched, if there’s anything missing from that, do you have any pointers where to look for an up to date config? The configuration itself to tell home assistant how to read these values is to my knowledge done by specifying the device_class, the unit_of_measurement and assigning it to the respective property of the energy feature? Let me know if I did miss something!
Thanks for having a look btw!
You should just be able to look at your history graph. That’s made from the non-long-term tables inside your database.
As for your configuration, that will depend on the topic. How is the device posting to your topic? Your current configuration has no safeguards against non numeric data.
The script posting to the topic is a (small) adaptation from GitHub - tribp/DSMR-Fluvius-MQTT-Openhab: How to connect your P1 port from the Fluvius Smart Meter to your home automation with MQTT.
Looking at the history graph there’s indeed a 400kWh drop (2688 → 2288), I’m curious now to find that information in the database but I can’t at least pin point it in the stastics_short_term.
Either way, that does indeed point to invalid data being pushed through, I’ll make sure to check the script that does the pushing and see if I can add some safety measures.
Is there anything possible on the receiving end btw? You mentioned value_template and safeguards, I’m assuming these will need to be set through a template sensor which takes the value from the mqtt and applies such guards? Can those evaluate that the state never can be below the state it had received before? (which in this case seemed to have happened)
you can add an availability to your mqtt configuration to look for numbers
availability:
- topic: your_state_topic
value_template: "{{ value | is_number }}"
This will make your sensor go unavailable when it’s not a number. When it returns, it won’t produce a spike.
This is assuming that the bad number is not coming from your custom script. You should look at the script and ensure 0 is not sent as a response to an error.
Will do, thank you again for your help on this.
Thanks @petro for looking at this - I too am having issues, although I have tried to fix this with the availability code below to no avail. I am using ModbusTCP2MQTT to ingest my inverter data.
- platform: mqtt
name: "daily_purchased_energy"
state_topic: inverter/tele/SENSOR
value_template: "{{ value_json.daily_purchased_energy }}"
unit_of_measurement: 'kWh'
state_class: total_increasing
device_class: energy
# availability: "{{ states('sensor.daily_purchased_energy') > 1 }}"
# last_reset: "2021-08-05T08:05:00+00:00"
availability:
- topic: inverter/tele/SENSOR
value_template: "{{ value | is_number }}"
payload_not_available: "offline"
unique_id: 394857362528
Your availability template should be checking to see if what value_tempalte is pulling is a number, not just value.
Thanks - the above doesn’t like to work for me, I fail to get any data once the availibility is added:
availability:
- topic: inverter/tele/SENSOR
value_template: "{{ value | is_number }}"
My apologies if it’s something I’m overlooking … maybe I’m formatting this wrong?
Your value_template, which produces your state is {{ value_json.daily_purchased_energy }}, that’s what’s making your number. | is number checks to see if a variable/equation is a number. So what do you think your value_template should be for availability?
Thanks for your help - below is the data I am receiving, which in this case shows the “daily_purchased_energy” as “28.9” (increasing / 30 sec).
So I think indeed it needs to be a number, but why will the entity fail if the availability as suggested is entered?
INFO:root:Published to MQTT
INFO:root:{‘daily_power_yield’: 26700.0, ‘total_power_yield’: 92.75, ‘internal_temp’: 39.4, ‘pv1_voltage’: 0.0, ‘pv1_current’: 0.0, ‘pv2_voltage’: 0.0, ‘pv2_current’: 0.0, ‘total_pv_power’: 0, ‘grid_voltage’: 235.7, ‘inverter_current’: 0.0, ‘total_active_power’: 0, ‘grid_frequency’: 50.0, ‘export_power’: 957, ‘export_power_indicator’: 0, ‘power_meter’: 957, ‘daily_purchased_energy’: 28.9, ‘daily_energy_consumption’: 21800.0, ‘total_energy_consumption’: 3243.4, ‘timestamp’: ‘2022-03-30T23:25:07’}
This is one example of the spike and the apparent totals:
I don’t think you understood my response. Your value_template for availability is not correct. My response was to help you correct your value template by leading you down the correct path.
Very much appreciate the help but I’m an absolute amateur when it comes to any sort of language other than english… but that said, and this is a guess based on my logic only, should it be “> 1” ?
Hi,
had the same issue, have a sensor integrated via ESPHome.
Had the spikes too. Was a problem at the sending device.
Found a workaround using a median.
See the last four entries.
ISSUE: abnormal data in home assistant · Issue #16 · DomiStyle/esphome-dlms-meter (github.com)
Maybe that helps.
1 Like
Let me try to make it clear (without giving you the answer, so that you can learn):
This is where your data is coming from:
This is your availability template:
So, your data is coming from value_json.daily_purchased_energy. You want to change {{ value | is_number }} so that it’s looking at value_json.daily_purchased_energy instead of value. So how would you change the template?
1 Like
I to am having similar issues with both gas and electric sensors. I am using Hildebrand Glow CAD, with MQTT bridged to pull data from Hildebrand.
An example is from today for gas (electric also had spike at the same time however gas should only get updates every half hour from Hildebrand):
My sensor config is:
- platform: mqtt
name: "Home Gas M3 Raw 2"
state_topic: !secret smart_hild_state_topic
unit_of_measurement: m³
value_template: "{{ value_json['gasMtr']['0702']['00']['00']|int(base=16) * value_json['gasMtr']['0702']['03']['01']|int(base=16) / value_json['gasMtr']['0702']['03']['02']|int(base=16) }}"
icon: 'mdi:flash'
device_class: gas
state_class: total_increasing
@petro from your previous pointers, would that mean the config should change to the following?
- platform: mqtt
name: "Home Gas M3 Raw 2"
state_topic: !secret smart_hild_state_topic
unit_of_measurement: m³
value_template: "{{ value_json['gasMtr']['0702']['00']['00']|int(base=16) * value_json['gasMtr']['0702']['03']['01']|int(base=16) / value_json['gasMtr']['0702']['03']['02']|int(base=16) }}"
availability:
- topic: !secret smart_hild_state_topic
value_template: "{{ (value_json['gasMtr']['0702']['00']['00']|int(base=16)) | is_number }}"
icon: 'mdi:flash'
device_class: gas
state_class: total_increasing
The following is from statistics_meta table:
The following is from ‘statistics’ table (includes the previous record and next record):
The following is from ‘states’ table (includes the previous record and next record):
Surely it wasn’t that simple! 
Thank you for your help @petro
availability:
- topic: inverter/tele/SENSOR
value_template: "{{ value_json.daily_purchased_energy | is_number }}"
EDIT: Nope, sensor has gone unavailable with that code.
availability:
- topic: inverter/tele/SENSOR
value_template: "{{ sensor.daily_purchased_energy | is_number }}"
And no again. Can I phone a friend or ask the audience please?
Could HA implement some sort of optional filter that could help to iron out massive discrepencies in data like this?
it is that simple. As for your edit, post your current configuration because what you have written prior to your edit should work. Your post edit template is jibberish.