How to get rid of outliers?

Third party integrations Node-RED
1

image
image681×808 148 KB

I have problems in understanding what causes ghost values/outliers (?) reported from my custom made sensor.

image
image1410×466 131 KB

First of all, I have many temperature sensors, constantly reporting temperatures over MQTT. In HA/Node Red, I use a standard “MQTT In”-node for receivement.

filter temp diff > 1 °C-node:

const oldMeanTempC = parseFloat(context.get("oldMeanTempC")) || 0;
const currentMeanTempC = parseFloat(msg.payload);
context.set("oldMeanTempC", currentMeanTempC);

const tempDiffTooBig = (currentTempC, previousTemp) => {
    const MAX_TEMP_DIFFERENCE_C = 1;
    const tooBig = Math.abs(currentTempC - previousTemp) > MAX_TEMP_DIFFERENCE_C;

    node.status({ fill: "yellow", shape: "dot", text: `Math.abs(${currentTempC}-${previousTemp}) > ${MAX_TEMP_DIFFERENCE_C} -> ${tooBig}`});

    return tooBig;
}

if (tempDiffTooBig(currentMeanTempC, oldMeanTempC)) {
    node.status({ fill: "red", shape: "dot", text: `outlier temp value (${msg.payload} °C)` });
    return null;
}

node.status({ fill: "green", shape: "dot", text: `${msg.payload} °C` });

return msg;

What on earth may cause all those strange bottom peaks in my first illustration? As you can see, I first calculate mean value based on 10 previous value, then I even actively discard values (return null;) if a value difference is > 1 from earlier reported.

What can I do to get rid of these outliers?

2

Hi, you can try the following node (not mine): node-red-contrib-cistern (node) - Node-RED

It filters out outliers or erroneous peaks of sensor values. It does a great job if there are peaks in your values, however if all the incoming values are within range the node generates an error. So in your case if all 10 incoming values are 10° there is an error, if 9 are 10° and one -20° it will work.

I opened a ticket in the makers GitHub, I hope he can correct this as it works great if there are a lot of peaks in the measurement.

1 Like