Question regarding derivative sensor

Configuration
1

Hi guys,

I have created a derivative sensor based on the humidity sensor of my bathroom. I want to use this in an automation, starting my exhaust fan if i detect a rapid rise in humidity.

However, i am not getting any sense out of the value of the derivative sensor.

Example (derivative in top graph):

derivative
derivative808×654 34.3 KB

At the time of the red line, it is clear that the relative humidity rises from 51% to 64%, a leap of 13%. However, the derivative sensor only reports a change of 4,2%/min.

This is the configuration of my derivative sensor:

sensor:
  - platform: derivative
    source: sensor.luftfoler_bad_humidity
    unit_time: min
    max_sub_interval: "00:01:00"

I suspect there might be some issues regarding averaging/sampling -interval, but i am not able to make any sense of it...

Any ideas? :slightly_smiling_face:

/Teigmo

2

It looks like there was a bit over 3 minutes between the 51% reading and the 64% reading... and 13% / 3 minutes is pretty close to 4.2%/min

3

Aha - so the reporting interval of the source sensor is also relevant?

I think my humidity sensor reports based on change. It can be up to an hour between readings, if the humidity is stable. And then there can be a sudden jump of several tens of percent for the next reading, if someone is taking a shower.

So you are telling me that in this case, this sudden jump will be averaged over the last hour (based on when the last sample was)?

Also when max_sub_interval: "00:01:00" is set?

This will lead to very unpredictable behavior...

4

The docs state "By default the derivative is calculated between two consecutive updates without any smoothing." So, yes, I'm pretty sure that's what would happen.

But the existence the max_sub_interval config variable makes that a bit confusing. What purpose does it actually serve if it is only going to report 0? On the other hand, what else could it report if it has no evidence of change? Maybe it would be helpful if you were using a long time_window and had a source that had irregular updates...? :man_shrugging:

You may want to see if there is any way to set a minimum reporting time or change the hysteresis on your actual humidity sensor.

5

I may be able to force my sensor to report every minute. However, this will probably kill my battery within days...

I have set the time_window config variable to 1 minute, let's see if this changes anything.

The time window in which to calculate the derivative. Derivatives in this window will be averaged with a Simple Moving Average algorithm weighted by time. This is for instance useful for a sensor that outputs discrete values, or to filter out short duration noise. By default the derivative is calculated between two consecutive updates without any smoothing.

Let's see if this calculates the derivative every minute based on the held value of the source sensor, regardless of update intervals...

EDIT:
Nope - did not solve the problem. Same behavior. :confused:

6

If you only want the last minute of time to be considered, then you should set a time window.

max_sub_interval is mainly used for driving the sensor back to 0 when it stops updating. Prior to that option existing, a derivative could often never be 0, because when the source sensor stopped changing, the derivative would not be updated and just hold its old value.

time_window controls what period of time is used for the calculation of the derivative
max_sub_interval controls when the derivative updates, if you want it to update more often than on a source sensor change.

7

Thank you all for your kind suggestions. :slightly_smiling_face:

I have tried bothtime_window and max_sub_interval set to 1 minute. This is the result i get (derivative on top):

result
result813×648 31.5 KB

It looks like the behavior is the same. First positive change gets averaged back to the last sample. The next negative change is considered larger, since there is a shorter time back to the last sample of the source sensor.

I wish the derivative sensor would have reported +49% and then -50% for a minute, since that is the actual change of value of the source sensor, within the set interval (1 minute).

8

I've tried this. It does not work any better than using a template trigger to trigger on:

(bathroom humidity) > (house average humidity) + (some offset)

9

Great idea, Tom! Thank you. However, i currently have no other humidity sensors to compare against... :confused: I might do this in the future.

I made a rather ugly workaround, creating a template sensor that updates every minute:

- trigger:
    - platform: time_pattern
      minutes: "/1"
  sensor:
    - name: "Bad luftfuktighet"
      unit_of_measurement: "%"
      state: "{{ states('sensor.luftfoler_bad_humidity') }}"
      attributes:
        last_updated: "{{ now() }}"

Then calculating the derivative based on this template sensor, yields the expected result. Not pretty, but it works...

If anyone has a better solution to the derivative problem, i am listening. :slightly_smiling_face:

10

I have to agree with @tom_l. I initially set up a derivative sensor and didn't have much luck.

My thought was the fan must pull dry air from somewhere else (or what's the poin?). So, I decided to turn on the fan when the humidity difference from where the air is coming from is some value. That's what I do now. I've also considered looking at hot water pipe temperature as the trigger.

There's a bit of delay, so maybe not ideal. I now have a flow meter on my house water and probably could just turn the fan on when some flow value is detected at shower time of day.

In the guest bath I have a door sensor and when the door is closed for about 30 seconds turn on the hot water pump and also turn on the fan. Just in case...

11

I am not sure it will be better, but another option to look into could be using a statistics sensor, with change_second as state_characteristic:

12

Just to throw the simple idea out there:

Is there a humidity threshold (say 60%) where:

  • It never gets that high when no one is in the shower.
  • It always exceeds that value if someone is in the shower.

If so, you can simply trigger when the value is above that value and turn off say 5 minutes after the value drops below the threshold.

13

Thank you all for suggesting solutions to the actual problem - starting the bathroom fan. However, i am mainly looking for a solution/explanation for the behavior of the derivative sensor.

If the derivative value is dependent on the reporting interval of the source sensor, it will (in my opinion) lead to unpredictable behavior. I would love if anyone has a solution to this problem, or an explanation of why it is designed like this - in case i am missing something.

14

I would turn the question around.

Assume the primary sensor reports intermittently - on average 2 or 3 times per hour, but the reporting interval is based on fluctuations of the data (i.e. it only reports on major changes).

If you were creating the derivative sensor how you would design it?

15

I would have it respect the time unit that is configured in the sensor options. If that is set to seconds, the sensor should calculate the derivative every second, looking at how much the value have changed since one second ago.

If the value is set to minutes, the calculation should run every minute, reporting how much the sensor have changed since one minute ago. If for some reason your sensor have changed up/down several times during that minute, this information will be lost due to not sampling often enough, and you should change to seconds if catching smaller changes is important for your application.

As another example, i am using a derivative sensor to calculate the momentary bandwidth utilization of my internet connection. This is done by fetching the total data amount Mbit using SNMP, and doing a seconds based derivative of this value. I then expect this to return Mbit/s. It is now my understanding that this is not always the case, if the source sensor does not report consistently every single second - also when there is no change in value. I would imagine the same issue to be relevant for e.g. water meters if you want Liters -> Liters/second or other accumulated consumption measurements that you would like to derive in to momentary consumption.

Now, this is my expectation/assumption of how this should work. There may be good reasons for it to behave differently, and i would love if anybody could explain that to me. :slightly_smiling_face:

16

So if we apply the design you outlined to to the example I gave, logically we might choose minutes or hours as the resolution.

If we choose minutes we will get a brief spike immediately after one of the updates from the sensor then for the rest of the time that value will hold steady, so the derivative value will be zero.

If we choose hours then we will effectively sample the value every hour and loose any resolution that occurs within the hour.

I have no idea about the actual design implemented by HA, however the graph you presented in the original post looks very similar to my prediction about what would happen with 1 minute resolution.

17

So, to conclude this thread:

Having a template sensor that updates regularly, works around the problem for me. This has now functioned flawlessly for a week.

Also, i am not the only one finding the behavior of the derivative sensor to be a little unintuitive. After working a bit more on my problem, i found these threads discussing the same issues: