Hello everyone
tl;dr I want the derivative (rate of change) of a signal, and I’m doing it terribly wrong
I have a sensor which provides the temperature of my Domestic Hot Water (DHW). What I’m trying to do is to get the rate of change of this temperature, so as to detect whenever I’m using DHW (thus lowering the temperature) or when I’m re-heating the water.
The raw data itself is a bit noisy, especially when the sensor “hesitates” between two values within its threshold. As a result, when I take the derivative of that, I get something that is super jumpy and not really usable. Here is a graph of the raw data, and of its derivative:
As you can see, there is a lot of jumpiness, and also I have a non-zero derivative when the temperature is not moving (which is wrong: when the temperature is not moving, its derivative should be zero).
These are the settings I used for the derivative:
- platform: derivative
source: sensor.espaltherma_temp_ecs
name: Changement température ECS brute
round: 3
unit_time: h # the resulting "unit_of_measurement" will be °C/h if the sensor.temperate has set °C as its unit
time_window: "00:01:00" # we look at the change over the last minute
First attempt
So my first idea was to smooth out the data before applying the derivative.
These are the settings I used for the filter integration:
- platform: filter
name: "Température ECS filtrée sans helper"
entity_id: sensor.espaltherma_temp_ecs
filters:
- filter: outlier
window_size: 4
radius: 1.0
- filter: lowpass
time_constant: 15
precision: 6
- filter: time_simple_moving_average
window_size: "00:02:00"
precision: 6
The filtering itself was disappointing. The filtered values were often quite far from the actual data, and when taking the derivative of the filtered data, the values weren’t much more meaningful than previously.
This is when I started to believe that part of my problem is that I only have raw data points when there is a change of value of the DHW temperature. I THINK (?) that as a result, the filter integration had an inconsistent number of points to work with, yielding bad results.
Second attempt
This time I went completely overboard in my opinion. I told myself that since I couldn’t get good filtered values from sparse data, I would create a “continuous” version of my DHW data. With one point every 5 seconds, the filter integration could yield a smooth temperature curve, and then I could get a nice derivative of that.
Here is what I did:
- Create a “Number” Helper, for storing DHW data
- Create an automation that fires every 5 seconds, and that stores the current raw DHW value into the helper
- Filter the data stored in the helper
- Take the derivative of the filtered data
So this is super tedious, and also takes a huge hit on the volume of data stored in my recorder. However, the results are starting to look adequate.
Here’s a view of all three approaches at once:
My question(s)
What is the best way to get the derivative of a signal which is noisy and sparse?
I feel that what I’m doing here is terribly wrong.
Am I wrong in the method?
Am I wrong in the settings of the components I’m trying to use?
I feel that what I’m doing can’t possibly be the best way to do this.
Thank you for reading this far!
