Calculating derivatives - can HA accept timestamps from a sensor instead of its own?

Development
#1

Hello everyone,

I’m facing a problem with processing sensor data in Home Assistant and need help to fix it. My power sensor takes readings at regular time intervals, but because of delays in wireless communication (from ZigBee bridge to WiFi router), Home Assistant ends up processing and storing the readings at uneven time intervals.

Right now, the times used are based on when Home Assistant receives the readings, not when the sensor actually takes them. Even though the sensor includes a timestamp in its report, Home Assistant isn’t using it. It’s making the calculated derivative unstable, which is not adequately fixed using a time window.

I exported the sensor’s data and HA’s derivative to verify this. I plotted them and then generated my own derivative by massaging the sensor data such that all samples were evenly spaced.
The following diagram shows a portion of my data with the issue I’m trying to remedy. (Please look past the orange/blue plots not being “in-phase”)

2023-04-22 16 08 47 - python3.10
2023-04-22 16 08 47 - python3.10901×547 77.8 KB

While trying to diagnose/fix this issue, I’ve considered that it might be an artefact of smoothing or the time window that HA’s derivative is set to use, but no matter what I set or simulate, I always get some events that don’t produce the derivative I’m expecting. I’ve only been able to get consistently correct results when the samples are spaced evenly.

Here’s a histogram showing the interval between samples, calculated from HA’s timestamps:

2023-04-22 16 18 11 - chrome
2023-04-22 16 18 11 - chrome774×496 26.7 KB

You can clearly see that the intervals between samples are normally quite stable, but plenty of cases aren’t. The samples that arrive at “exactly the same time” - essentially identical timestamps in the exported data - are particularly damaging to the derivative.

Is it possible for Home Assistant to handle sensor readings with their own timestamps? I’m guessing if yes, I’ll need to make code changes to the integration that’s providing the sensor data, which I’m comfortable doing.

1 Like
#2

Maybe you can use the spook integration to store the data as historic data.

Spook is available through HACS

#3

Do you know if “importing” real-time data here will work normally? IE. Still trigger automation, downstream template sensors, etc?

As far as I was about to tell when I came across the async_add_external_statistics and async_import_statistics APIs, they don’t actually change the current state of an entity… So that’ll be good for viewing the data, but not much else, I think.

Been thinking about this a fair bit over the last few days, and I’m starting to settle on developing one of these approaches:

  1. Custom “delayed derivative” sensor - handles input data with an input timestamp and produces a steady ongoing derivative.

  2. A sensor that’s “just aware” of the timestamps; buffer state updates to mitigate network unreliability at the cost of data being delayed by some number of samples.

Approach 1 is better than 2 if I only care about this specific derivative issue, as I don’t actually need the output of the derivative to have even spacing. It just needs to use evenly-spaced samples in its calculations.

… now, having written that out, I realise that I can compute my own derivative using the individual samples rather than time - the whole point of trying to get a reliable output from the derivative is so it can be used for triggers.

I’ve just tested that theory and am happy to report that it works as I wanted, just with a pair of statistics sensors. One to calculate delta over two samples, the other to sum the previous three samples together. For a “true” derivative, I’d want to divide the sum’s output by three, but it doesn’t matter for my use (and it’s convenient having the output have the same scale as the input).

Here’s a fun screenshot showing my kettle boiling:
image

And the sensors:

- platform: statistics
  name: "Electricity: Consumption (Delta)"
  entity_id: sensor.meter_electricity_power
  state_characteristic: change
  sampling_size: 2
  precision: 3

- platform: statistics
  name: "Electricity: Consumption (Delta Sum)"
  entity_id: sensor.electricity_consumption_delta
  state_characteristic: sum
  sampling_size: 3
  precision: 3
#4

It does not seem to do.
It just writes the data to the database it seems.