i am using an ultrasonic sensor in our rain water tank.
The water level changes very slowly but I sometimes have measurement errors that I cannot explain.
I use a median of 30 values to try to limit the effect of these errors. Would it be possible to simply remove the extreme values from this sample?
I would say median is a perfectly valid and good way to handle them.
For single spikes you should only need a window of 3, for double spikes a window of 5.
A window of ~10 should be enough to both handle outliers and provide a good centrepoint if your sensor measurements have some variability. This is a reason why a median filter is one of my favourites.
You can also construct a lambda filter (if > Y then Z, else x).
I still have a problem which in my opinion is related to the treatment via the quantiles.
Look at the graph I get
The trend looks good to me but the sharp drops are amazing. Could it be due to the rounding function I use?
My idea was precisely to provide more reliable values for the quantiles.
[15:05:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.22 m
[15:05:15][D][sensor:127]: 'cuve_distance': Sending state 0.22415 m with 5 decimals of accuracy
[15:06:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:06:15][D][sensor:127]: 'cuve_distance': Sending state 0.55995 m with 5 decimals of accuracy
[15:07:37][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.22 m
[15:07:37][D][sensor:127]: 'cuve_distance': Sending state 0.21952 m with 5 decimals of accuracy
[15:08:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:08:15][D][sensor:127]: 'cuve_distance': Sending state 0.55995 m with 5 decimals of accuracy
[15:09:13][D][sensor:127]: 'cuve_wifi': Sending state 20.00000 % with 0 decimals of accuracy
[15:09:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:09:15][D][sensor:127]: 'cuve_distance': Sending state 0.55995 m with 5 decimals of accuracy
[15:10:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.22 m
[15:10:15][D][sensor:127]: 'cuve_distance': Sending state 0.22398 m with 5 decimals of accuracy
[15:11:16][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:11:16][D][sensor:127]: 'cuve_distance': Sending state 0.56012 m with 5 decimals of accuracy
[15:12:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.22 m
[15:12:15][D][sensor:127]: 'cuve_distance': Sending state 0.22415 m with 5 decimals of accuracy
[15:13:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:13:15][D][sensor:127]: 'cuve_distance': Sending state 0.55995 m with 5 decimals of accuracy
[15:14:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:14:15][D][sensor:127]: 'cuve_distance': Sending state 0.55978 m with 5 decimals of accuracy
[15:15:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.22 m
[15:15:15][D][sensor:127]: 'cuve_distance': Sending state 0.22398 m with 5 decimals of accuracy
[15:16:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:16:16][D][sensor:127]: 'cuve_distance': Sending state 0.55995 m with 5 decimals of accuracy
[15:17:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.22 m
[15:17:15][D][sensor:127]: 'cuve_distance': Sending state 0.22415 m with 5 decimals of accuracy
[15:18:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:18:15][D][sensor:127]: 'cuve_distance': Sending state 0.55995 m with 5 decimals of accuracy
[15:19:11][D][sensor:127]: 'cuve_wifi': Sending state 24.00000 % with 0 decimals of accuracy
[15:19:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:19:15][D][sensor:127]: 'cuve_distance': Sending state 0.55978 m with 5 decimals of accuracy
[15:20:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:20:15][D][sensor:127]: 'cuve_distance': Sending state 0.55995 m with 5 decimals of accuracy
[15:21:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:21:15][D][sensor:127]: 'cuve_distance': Sending state 0.55995 m with 5 decimals of accuracy
[15:22:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:22:15][D][sensor:127]: 'cuve_distance': Sending state 0.56029 m with 5 decimals of accuracy
[15:23:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:23:15][D][sensor:127]: 'cuve_distance': Sending state 0.55978 m with 5 decimals of accuracy
[15:24:15][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:24:15][D][sensor:127]: 'cuve_distance': Sending state 0.55978 m with 5 decimals of accuracy
[15:25:16][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:25:16][D][sensor:127]: 'cuve_distance': Sending state 0.55978 m with 5 decimals of accuracy
[15:26:16][D][ultrasonic.sensor:040]: 'cuve_distance' - Got distance: 0.56 m
[15:26:16][D][sensor:127]: 'cuve_distance': Sending state 0.55995 m with 5 decimals of accuracy
for the moment I do not know if I should multiply the measurements or increase the size of the sample
For the moment I make a measurement per minute and a sample of 10 measurements.
In the end I have a value recorded every 10 minutes