I want to make a 24 hour rolling average of a sensor value. Currently, the sensor value comes in about every 1-2 minutes but it isn’t reliable. I’d like to take those values coming from it and rolling average them across 24 hours so I can see the trending change over time.
My brain keeps getting “wrapped around the axle” because I’m pretty sure I need to use the sliding_window_moving_average here but I’m not sure how to structure it. I think it would be cool if the sensor value updated every minute or every ten minutes to give me the last 24 hours of average data but I don’t want to get bogus (un-averaged data) either as the window is filling up with values.
I see that I can define a send_every and send_at_first value. Can I define the following to get minutely-updated data averaged over the last hour while avoiding an un-full window average for the first 59 measurements?
Also, I’m sure there are limits to how big we can make the window size but I don’t know what they are and how to tell if I’m close to the limit. If I can do what I’m asking about above, could I do this definition below and get a minutely-updated data averaged over the last 24 hours while avoiding an un-full window average for the first 1439 measurements?
Bah, ESPHome won’t even compile the YAML like that. It doesn’t allow send_first_at to be greater than send_every. Is there no way to avoid reporting partially filled moving average pipeline data?
I did a bunch of experiments and determined that the best way to do it is to update the sliding_window_moving_average as often as possible and let the window perform its magic. That way you get a very smooth transition to your new state:
These are test results of me creating a step function with an ultrasonic distance sensor. I’d point it at my bookshelf like half a meter away and then immediately at the ceiling ~1.5m away. You can see the various divisions I created for smoothing out that step function. I believe the 30 minute rolling average is a logarithmic decay because the window wasn’t full when I flipped it from 1.5m to 0.5m.
I think with this thread (so long ago) I was also trying to figure out a good way to report AQI data. I got a few of these PMSX003 Particulate Matter Sensor — ESPHome sensors. What I found was that normal professionally calculated/reported AQI data is a 24-hour mean and is sampled hourly so the data your looking at is really slow… like 12 hours ago. I wanted something that’s much more rapid like 30-minutes ago – aligning more to what I’m experiencing. So, I used my experimentation data to come up with this:
Ditto!!! my ESPHome file to control the stove is 633 lines of code! Im amazed at how versatile ESPHome is and just how much you can do with it. your PMSX003 code is interesting. I made one as well when I lived in Sonoma County CA and we had fires every frigging year. I moved to Idaho a few yrs ago so now we now have clean air AND freedom, so I no longer need it.
Im using your filter idea to smooth some sensors in my pellet stove ultrasonic level sensor and temp sensors that fluctuate all over the place. works great
What a coincidence, I’ve been living in Idaho since 2001!! Nice to find a “local” on here! Funny you say that the air is “clean”. It has gotten substantially worse since we moved here. When I look at the last few months there were quite a few days where the AQI was RED (including a big one over Christmas)!
I guess this might be “clean” when compared to California’s air.
Anyway, this is why I invested in building a house air cleaning system…
I’ve put these little AQI sensors all over my house so I can monitor the quality of the air inside and out. I also got several "smart"ish air purifiers that I can control with HA so I could kick them on if that room’s AQI goes south. My house was built in 1982 so it isn’t very airtight construction. When we have prolonged periods of smoky weather outside, it eventually seeps in and the purifiers kick on to clean it out. Highly recommended!
ok, mabe you can help. I want to turn this rocking Lambda filter use in all my Plotly graphs into something I can use in ESPHome’s sensor filters. It works better than any filter I can find and when using it w/ Plotly, you can immediately see the results of changes made to the values:
- entity: sensor.esph_house_raw_pellet_level_in_percent
name: Level Lambda Smoothing applied
show_value:
right_margin: 40
line:
color: yellow
shape: spline
width: 1
smoothing: 1
lambda: |-
(ys, xs) => {
const MINUTES = 8 // <-- change here
const window_size_ms = 1000*60*MINUTES
const result = {x:[], y:[]}
let x_next = +xs[0] + window_size_ms
let y_accum = 0
let y_count = 0
for (let i = 0; i < xs.length; i++) {
const x = +xs[i]
const y = +ys[i]
if (x>x_next){
result.y.push(y_accum/y_count)
result.x.push(new Date(x_next))
y_accum = 0
y_count = 0
x_next = +x + window_size_ms
}
if (y){
// ignore unavailable/unknown state points
y_accum += y
y_count ++
}
}
return result;
}
Maybe you can quote what exactly you’re talking about? I can’t tell by your description. If it’s the my_window_size that you’re referring to, I’m just using the window moving average in ESPHome to smooth out the signal. Take a look here: Sensor Component — ESPHome
Otherwise, please send the code you’re asking about and I’ll try to explain.
