Calibrate_linear - more suitable alternative?

My setup for my Ultrasonic tank level sensor is based upon ESPHome and the Ultrasonic platform.
I am measuring the distance from the sensor to the top of the fluid in the tank.

Originally I had just one sensor set up in the ESPHome config, measuring distance.

sensor:
 - platform: ultrasonic
   trigger_pin:
     number: GPIO4 #Yellow sensor cable from A02YYUW
   echo_pin:
     number: GPIO5 #White sensor cable from A02YYUW
   name: "Distance to Oil"
   id: oil_distance
   unit_of_measurement: "m"
   accuracy_decimals: 2
   update_interval: 1s
   timeout: 10.0m
   filters:
    - sliding_window_moving_average:
        window_size: 20
        send_every: 20

From this, I created 2 template sensors in HA in order to convert this distance into a % of tank remaining as well as a reading for the number of Litres of fluid remaining.

# Converting calculated % of tank remaining into Litres. Tank holds 1300L when full 
      - name: "Oil Remaining"
        unit_of_measurement: "Litres"
        state: "{{ (13 * (states('sensor.oil_level')|float(0))) | round(1) }}"
# Converting measured distance from Ultrasonic sensor into % of tank remaining. 
# 1.12m from sensor to bottom of tank, 0.2m air gap from oil when full to sensor
      - name: "Oil Level"
        unit_of_measurement: "%"
        state: "{{ (100 * (1.12 - (states('sensor.distance_to_sensor')|float(0)) + 0.2) / 1.12) | round(1) }}"

As the ‘oil remaining’ value is based on a % of tank being full, it is only an approximation. I have subsequently found a chart for my tank, which lists EXACT volume levels for several given heights.

image

I thought about changing my setup and creating an extra sensor in my ESPHome config to give me the Litres remaining, based on the measured distance form the sensor, and using the calibrate_linear filter, including all the known data points.

 - platform: ultrasonic
   trigger_pin:
     number: GPIO4 #Yellow sensor cable from A02YYUW
   echo_pin:
     number: GPIO5 #White sensor cable from A02YYUW
   name: "Oil Remaining"
   id: oil_remaining
   unit_of_measurement: "L"
   accuracy_decimals: 0
   update_interval: 1s
   timeout: 10.0m
   filters:
    - calibrate_linear:
    # Map 0.0 (from sensor) to 0.0 (true value)
        - 1.20 -> 93
        - 1.10 -> 217
        - 1.00 -> 361
        - 0.90 -> 516
        - 0.8 -> 678
        - 0.7 -> 843
        - 0.6 -> 1002
        - 0.5 -> 1147 
        - 0.4 -> 1269
        - 0.3 -> 1356
        - 0.2 -> 1378

Unfortunately this is now giving me incorrect readings. For exapmle, setting the distance at 0.20m, I am getting a reading of 1507 Litres, rather than the 1378 Litres that I entered in the filter for this distance.
After re-reading the ESPHome calibrate_linear documentation, it seems that the entered data point values can actually change if you enter more than 2, as it has to create the linear relationship that fits best.

Can anyone suggest an alternative way of plugging these known data points in, so that only the readings between these known data points are estimated based on a linear scale?

Perhaps Sensor Component — ESPHome

1 Like

Yeah plotting your points in Excel the linear equation is a pretty good fit except for the lower ADC readings.

A polynomial with degree 3 is an almost exact fit. Use that degree in Nick’s suggestion.

2 Likes

Thanks so much @tom_l and @nickrout - that has resolved the problem and my sensor is now giving me the desired readings. I did read the polynomial filter information, but at the time, didn’t think it would do anything better than what I was doing with the calibrate_linear filter. I can see now that it is quite different in how it works, and the results show that. New ESPHome sensor config with tweaked data points also:

 - platform: ultrasonic
   trigger_pin:
     number: GPIO4 #Yellow sensor cable from A02YYUW
   echo_pin:
     number: GPIO5 #White sensor cable from A02YYUW
   name: "Oil Remaining"
   id: oil_remaining
   unit_of_measurement: "L"
   accuracy_decimals: 0
   update_interval: 1s
   timeout: 10.0m
   filters:
    - calibrate_polynomial:
       degree: 3
       datapoints:
    # Map 0.0 (from sensor) to 0.0 (true value)
        - 1.02 -> 93
        - 0.92 -> 217
        - 0.82 -> 361
        - 0.72 -> 516
        - 0.62 -> 678
        - 0.52 -> 843
        - 0.42 -> 1002
        - 0.32 -> 1147 
        - 0.22 -> 1269
        - 0.12 -> 1356
        - 0.05 -> 1378

Now when my sensor distance reading is 0.09m, I am seeing a volume in Litres of 1365, which sits nicely between the entered data point either side of that distance. Thanks again.

Just for my own learning - how does the sensor output values that are outside of the scope of the entered data points? I would have thought that it would maintain the gradient of the calibrated readings, but perhaps it works differently? The reason I ask is that my sensor currently reads 1.52m distance. This is resulting in an output of 147 litres. As the highest distance I specified as a data point in the calibrate_polynomial was 1.02m, which is calibrated to a value of 93 litres, I expected values above 1.02 to result in a lower number of Litres than 93?

When you fit a polynomial to a section of a curve like this the match is only valid for the short interval you have defined. Then it can diverge quite widely.

You will need more widely spaced points for a wider prediction range. Then the polynomial equation, and possibly degree required will change.

Look what happens if you change the degree of the polynomial for the points you have:

1 Like

I see what you mean - thanks for taking the time to show the various polynomial degree lines, very interesting!

Based on using the suggested polynomial with degree 3, and looking at that line graph which represents this, the value for a distance of 1.52m looks like it should be a small negative number, around -50ish, however I am getting a positive value of 147. This would seem to fit more between the blue and the green lines in the graph above, so somewhere between degree 3 and 5?

I understand about the requirement for more and wider spread data points to cater for this sort of distance, I’m just curious how the reading I am getting doesn’t seem to follow the projected line for degree 3?

There’s no guarantee that ESPHome fits the same polynomial that Excel does.

Ah I see, the curve must provide different values from the ESPHome polynomial outside of the entered data points. Thanks again for your help with this.

Just curious/thinking/warning, because the tank you mentioned contains oil, and the sensor you are using seems to be not (explicit) suitable for chemical and explosion/flammable/gas environments, did you took that in consideration? :thinking:

Yes, a good point. I spent some time before embarking on this project identifying potential risks, given the environment where the sensor will be located.

Sounds to me you made your personal ATEX Risk Assessment :+1: :grinning_face_with_smiling_eyes:

EDIT: Changed (wrong) original URL, sorry

Hi! Thanks to all, your discussion was very helpful to me!
I’ll use a pressure sensor at the bottom of the tank instead of the ultrasonic sensor.