First time posting. Have been a little active on the subreddit, but I need more specifics before trying to dive into a thing and it being not the wisest choice for what I’m trying to attempt make use of the collected interior and exterior environmental data. I’m using tons of Xiaomi/Aqara WSDCGQ11LM’s which have the SHT30 temperature sensor. However, since the firmware sends out the temperature once there has been a significant enough change, I have to deal with quantization noise. This is somewhat mitigated by taking the collective mean of them, but the derivative of the resulting curve is messy showing that I need to shape the data to make use of it.
This is where my problem lives. I’ve tried the built-in low pass filter, 30 samples, 6 for the time constant, but I got a temporal phase shift like what I’d expect with a moving average. Speaking of those, I have to “double tap” an MA filter to get reasonable performance out of it, but the noise I’m dealing with is over longer time frames, 1-2 hrs, that appears to be resulting of the “random” temporal updates from the Zigbee sensors. Some would say, “okay, fine, just use a larger window for the moving average” but if I do that I have to deal with the temporal phase shift that’s half of the duration of the window, thus negating any “live” nature of the data.
In past when dealing with old noisy sensors like this, I could log the data and shape it with some special, strong algorithms like Savitzky-Golay filtering. And then there is polynomial fitting. In order to implement these to “get” “clean data” that I can then manipulate for rate of change or more comprehensive analysis, It seems like I’ll have to build something myself. I’m new to HA’s ecosystem so I don’t know where to start per se. I’m a hardware and math guy, not so much software development. I’ve done my share in BASH, can read C with some light experience in it, haven’t messed with python much (where I’ve been pointed) and I’m not quick, but I’ll make due. Where to I get this started in HA? How’d you go about it?
Edit: here’s a graphed sample of what I’m dealing with
