Trying to adjust template sensor to provide "Modal" number

[pcal]

HI Guys, I’m using a pressure sensor in a water tank to indicate the level of the water.

The sensor outputs a voltage that equates to the water level, which is connected to the otherwise spare Analog input on a Shelly Uni that’s installed in the door control on the other side of the shed, to provide a sensor with a voltage reading.

I’ve created a template sensor which takes that voltage as its input and calculates a basic % number between 1 and 100 to indicate the level of water in the tank. So far, so good.

Problem is, there is some interference coming from goodness knows where which causes the voltage reading to jump up by a volt or two for several seconds, quite often. I have no idea if its the pressure transducer, or the Analog input on the Shelly, or some random radio signal hitting the wire between the two. But it throws my measurement of the water level off, depending on exactly what moment you look at it - as per the sample graph below…

I"m taking the baseline of the graph as the accurate reading, and all the spikes as interference.

I don’t want to use an average (mean) - as this will still give a false high reading most of the time, but it seems that a “modal” reading for the previous say 5 minutes, would be more often than not about right.

I have been round in circles for some time and not found anything that looks like making this work. My templating skills are rudimentary at best - but I’m hoping there is just some stat function I don’t know about that can adjust the template to read the modal number…

{{ (states('sensor.shed_door_control_adc') | float / 5 * 100 ) }}

I’m guessing this shouldn’t be too hard, but it has managed to allude me so far. Any clues?

Cheers,

Pcal

[tom_l]

You should really see if you can eliminate that interference at the source.

Are you using screened cable between the sensor and Shelly?

Is the screen only earthed at one end (good) or both (bad)?

Could you move the Shelly closer to the sensor and use long leads on the door sensor?

The binary door sensor should be less susceptible to noise. Especially if you use a normally closed sensor as it will be very low impedance to GND most of the time.

If none of that is possible you could use the value_min characteristic of the statisitcs sensor and a 5 minute max_age:

Statistics

[Hellis81]

You could make sure it only allows a certain percentage in difference or it will skip that value.

Here I have v as the current voltage and old_v which is the previous voltage.

{% set v = 2.1 %}
{% set old_v = 1.3 %}

{% if ((v-old_v)/v) | abs > 0.1 %}
  {{ old_v }}
{% else %}
  {{ v }}
{% endif %}

This will not allow changes greater than 10%.

I guess for you that means you need:

{% set v = (states('sensor.shed_door_control_adc') | float / 5 * 100 ) | int %}
{% set old_v = states('sensor.tank_fill') | int %}

{% if ((v-old_v)/v) | abs > 0.1 %}
  {{ old_v }}
{% else %}
  {{ v }}
{% endif %}

Edit:
Just realized that 10% is probably too much.
Your values are sometimes almost in the 10% range.
Perhaps make it 5 or 7% to start with and see what that does.

[PeteRage]

What pressure sensor device are you using? I’d like to do the same for my welll. I have a dozen current sensors that output voltage going to analog inputs and have not seen this issue.

I’d try grounding one of the sensor legs on the Shelly side. Post the device manual links and your current wiring. You may also need a resistor to dampen the signal. If you have access to a DC multimeter put that on the sensor and see if it jumps around. Disconnect the sensor and attach to a 1.5v battery and see if the signal is consistent.

[pcal]

Thanks Tom.

I agree fixing the issue is better than masking its impact - but there are some logistical challenges in this case. The pressure transducer I bought came with a 10 meter cable attached which from my recollection is not shielded. It is mostly run inside a steel C section, and covers about 2/3 of the distance to the Shelly. I did use a shielded cable as the extension, but most of its length is un-shielded and there isn’t a lot I can do about that. The cable is molded to the sensor and includes a breather tube so a sheilded replacement is not readily available and I couldn’t fit it without compromising the waterproof seal anyway.

[pcal]

I like this approach @Hellis81. Will give it a shot and see what happens…

[pcal]

@PeteRage The sensor I used was this one …

The AliExpress listing gives more information than the printed manual - which essentially just identifies V+, V-, and Signal wires. Mine is the 2m range unit with a 0-5V output.

[pcal]

@Hellis81

I guess for you that means you need:

{% set v = (states('sensor.shed_door_control_adc') | float / 5 * 100 ) | int %}
{% set old_v = states('sensor.tank_fill') | int %}

{% if ((v-old_v)/v) | abs > 0.1 %}
  {{ old_v }}
{% else %}
  {{ v }}
{% endif %}

@Hellis81 I tried to add the suggested code to my template sensor but it threw some errors which went away once I swapped “int” for “float” in the set instructions.

Once the errors were gone (per the Template editor), I added it to my sensor code, but the sensor became Unavailable. Working on the assumption there was a circular reference issue, I created a new template sensor using this code, and took the previous template sensor as its input.

It did work, but I still got the spikes on the graph - even after adjusting the 0.1 threshold all the way down to 0.02.

So I changed tack and tried the Statistics sensor as suggested by @tom_l

Added this to the configuration.yaml to filter the voltage reading…

sensor:
  - platform: statistics
    name: "Min Tank Pressure"
    entity_id: sensor.shed_door_control_adc
    state_characteristic: value_min
    max_age:
      minutes: 5

And adjusted my original template sensor to get its data from the statistics sensor.

{{ (states('sensor.min_tank_pressure') | float / 5 * 100 ) }}

Again, it sort of worked. The number of spikes has reduced (but not gone completely), and I now have a bunch of pseudo-random periods where the sensor goes Unavailable as well.

Interestingly, it seems to be most stable while the rain was actually falling this afternoon (but for the 5 minute steps in what otherwise should have been a smooth upward slope), and only went haywire while the level reading should have been static.

From previous analysis, I’ve never seen a spike that lasted more than about 30 seconds - so a 5 minute interval should absolutely cover them all. Why would there still be spikes, and why would the sensor be going unavailable all the time now? - to the best of my knowledge, that has never happened before. The sensor has been in place for quite a few months now and while its only just made it to the top of my “things to fix” list, has been exhibiting the graph spike since the day it was installed.

Many thanks

[pcal]

For the sake of completeness, this is the history chart of the raw voltage reading for the same time period as the filtered output above. If nothing else, it shows that the physical sensor input was still present when the calculated number became unavailable…

[Hellis81]

I don’t understand why the code produce any errors at all.
Can you paste it in developer tools.
What does it say there?
It should not be a circular reference error, this method is common to use when you need to filter values.

[pcal]

It was in Developer Tools where the error was being shown - but it is making a fool of me now - because the error is not being shown there any more. I played around with a great many things since then, so may have altered something else that was impacting it? I can’t remember exactly what the message was - but it was complaining about parsing(?) a string and an integer together? Either way the first thing I tried (changing the int to float) made the error go away.

However, when I copy the template code into the tank.fill sensor - the entire sensor goes perpetually unavailable (as per my lovelace screen that reads this data below)…

When I switch the code back to using the Statistics filter I (usually) get…

This too sometimes goes unavailable, but only for a few minutes at a time until it comes back again by itself.

Using my original “unfiltered” template sensor, I could sit and watch the water level rise and fall in the tank, which would have seemed cool if I didn’t know it was just a glitch…

[Hellis81]

You are correct. Unavailable makes sense.

We need to change it slightly to make sure it gets a first value.

{% set v = (states('sensor.shed_door_control_adc') | float / 5 * 100 ) | int %}
{% set old_v = states('sensor.tank_fill') | int %}

{% if old_v == "unavailable" %}
  {{ v }}
{% else %}
{% if ((v-old_v)/v) | abs > 0.1 %}
  {{ old_v }}
{% else %}
  {{ v }}
{% endif %}
{% endif %}

Pardon the indentation, I’m trying on my phone.
This should make sure it sets the tank fil the first time and then allows 10% changes.

The only issue is that if the first value is a false high value then it will have a hard time getting down to the correct value

[pcal]

Thanks @Hellis81,

Pasted your adjusted code into my template sensor - and it immediately went Unavailable again, and stayed that way for three hours while I gave it a chance to settle in as I was doing something else.

I’ve changed tack again - and gone all Old School. Way back when I was an Apprentice Electrical Technician I used to carry an Analogue multi-meter as well as my fancy new Digital one. The physical damping of the needle in the analogue meter made it easier to read voltages that bounced around a lot.

I’ve applied that principle here. Set up a new Number Helper with a range of 0 to 5 (to match the voltage output of the pressure transducer) and gave it a step size of 0.001

I had an unrelated Automation that triggers once every minute to check if certain appliances are running, and increment a counter if they are. (The counters are the input for Running Hours meters associated with those appliances). Just added a couple of extra lines to that automation so if the value of sensor.shed_door_control_adc is more than input_number.tank_report, then the Tank Report number in incremented by one step. If the value is less then it decrements.

Then swapped my original template sensor.tank_fill to read from input_number.tank_report instead - and I now have a very smooth and stable number representing the water level in my tank which is effectively dampened to not change by more than 0.001v (0.02%) per minute. All the spikes are gone, and I have only the occasional “ripple” in the reading to show where they were. If there is legitimately a big change, it takes several minutes for the correct value to flow through in multiple small steps, but in my application that isn’t a problem…

It’s nowhere near as elegant as the template code you were suggesting - but it is stable and it works.

Thanks heaps for all your suggestions and helping me to ponder different ways of looking at the problem.