How Bayes Sensors work, from a Statistics Professor (with working Google Sheets!)

[uezezs]

Can someone tell me the table cell giving 0.33 as shown in the yaml example? The -in my understanding- appropriate cell in the table returns 0.29

[josemsubcn]

Dear # JudoWill thanks to share, but I have a question for your excel file… For the Probalility formula, I think that have an error, the formula say:

=INDEX(L:L,COUNTA(L:L),1)

In my opinion this formula don’t take into account the last row, any value for this row show the same result. I think that the correct formula should be:

=INDEX(L:L,COUNTA(L:L)+1,1)

What do you think?.

Jose

[uezezs]

@josemsubcn Yes, you’re right and it has been confirmed in one of the posts above

[uezezs]

Am I the only one who gets confused about the differences between the explanation and the table?

e.g. 4 hours of 10 hours are 0.29 not 0.33 (Cell F16).

[cearum]

@uezezs It seems in his example above he put house is occupied at 12 hours and not 14 hours (as in the spreadsheet). So it’s 4/12=~0.33 whereas 4/14=0.29. I’m guessing it’s a typo/mistake in the yaml saying 10 hours (as it’s 10 hours away not home). Example said 12 hours, but spreadsheet has 14 (to make 10+14=24)

I’d go with the spreadsheet as it seems the most accurate (with the fix mentioned above for the Index).

[monotux (Oscar)]

Another thanks for the original post - I’m still don’t really understand this but the spreadsheet was really helpful together with the explanation. Thanks!

I created a few boolean input selectors and made a bayesian sensor based on these. It helped me better understand the explanation above when I could easily toggle the selectors and see how that affected the probability, instead of waiting for real life sensors to trigger, cool down and so forth.

I’m using most of my sensors (motion + a custom for my media players, to see if anything is playing) and two template sensors with delay_off setup like in one of the entries in this thread. Works great so far!

[JohannCR (Johann)]

Hi, can you share how you use time of day sensors ?
Thanks a lot for your clear explanations, that helped a lot !

[jjross]

Oh! Time of day sensors would be really helpful here. I would love to see how to get that working with bayes. Anyone have an example?

[JohannCR (Johann)]

Also, how can we handle multiple device_trackers, particularly to ignore them if they are not on the same page ?

Example : I’ve got two gps device_trackers but both can occasionnaly be thrown off… I would like the bayesian sensor to make them “cancel each other” when they disagree, and rely on other signals until they agree. Is that doable ?

Edit:
Well, I found a way to do that but the probabilities cannot be defined by hours true/false. Here is a minimum working example :

• platform: bayesian
  name: home
  prior: 0.8
  probability_threshold: 0.8
  observations:
  • platform: state
    entity_id: device_tracker.gps1
    to_state: “home”
    prob_given_true: 0.6
    prob_given_false: 0.4
  • platform: state
    entity_id: device_tracker.gps1
    to_state: “not_home”
    prob_given_true: 0.4
    prob_given_false: 0.6
  • platform: state
    entity_id: device_tracker.gps2
    to_state: “home”
    prob_given_true: 0.6
    prob_given_false: 0.4
  • platform: state
    entity_id: device_tracker.gps2
    to_state: “not_home”
    prob_given_true: 0.4
    prob_given_false: 0.6
  • platform: state
    entity_id: light.xxx
    to_state: “on”
    prob_given_true: 0.6
    prob_given_false: 0.4
  • platform: state
    entity_id: light.xxx
    to_state: “off”
    prob_given_true: 0.4
    prob_given_false: 0.6

non-contradictory GPS takes priority over light status:
home with light → 0.93 HOME
home without light → 0.86 HOME
not home with light → 0.73 NOT_HOME
not home without light → 0.54 NOT_HOME

light status takes priority over contradictory GPS:
contradictory gps with light → 0.86 HOME
contradictory gps without light → 0.73 NOT_HOME

I’m going to try adding other observations and see if it holds

[abhi.ko (Abhi)]

You sir are a legend and definitely a great professor, no doubt. That was very clear. I was looking for more information on how this is implemented in HA and your write-up was excellent. I am trying to set up a sensor to estimate whether all of the household is in bed or not, could you please check if the below code makes sense for that.

- platform: bayesian
  prior: 0.32
  name: 'Bedtime'
  probability_threshold: 0.9
  observations:
      - entity_id: group.alarmo_device_tracker
        prob_given_true: 0.99  # If I'm in bed then I have to be home - hence 100% probability
        prob_given_false: 0.7 # I could be home and not in bed 70% of the time during any given day
        platform: 'state'
        to_state: 'home'
      - entity_id: group.alarmo_device_tracker
        prob_given_true: 0.009 # If I'm in bed, then I can't be not home
        prob_given_false: 0.5 # I'm not in bed and not home (i.e. at work or school etc.) 
        platform: 'state'
        to_state: 'not_home'        
      - entity_id: 'sensor.sun'
        prob_given_true: 0.97
        prob_given_false: 0.33
        platform: 'state'
        to_state: 'below_horizon'
      - entity_id: 'sensor.sun'
        prob_given_true: 0.03
        prob_given_false: 0.99
        platform: 'state'
        to_state: 'above_horizon'        
      - entity_id: group.interior_lights
        prob_given_true: 0.99 # I never go to bed with lights on
        prob_given_false: 0.8 # Lights off when I'm not in bed, which is true for most of the DAY, but excluding some lights (e.g.  bathroom, garage, pantry) 
        platform: 'state'
        to_state: 'off'
      - entity_id: group.interior_lights
        prob_given_true: 0.001 # reverse of the above
        prob_given_false: 0.2 # reverse of the above
        platform: 'state'
        to_state: 'on'        
      - entity_id: binary_sensor.master_door_contact
        prob_given_true: 0.98
        prob_given_false: 0.3
        platform: 'state'
        to_state: 'off'
      - entity_id: binary_sensor.master_door_contact
        prob_given_true: 0.02
        prob_given_false: 0.7
        platform: 'state'
        to_state: 'on'        
      - entity_id: 'variable.last_motion'
        prob_given_true: 0.95
        prob_given_false: 0.15
        platform: 'state'
        to_state: 'Bedroom Zooz Sensor motion'
      - entity_id: binary_sensor.abhi_pixel_is_charging
        prob_given_true: 0.9
        prob_given_false: 0.4
        platform: 'state'
        to_state: 'on'
      - entity_id: binary_sensor.abhi_pixel_is_charging
        prob_given_true: 0.1
        prob_given_false: 0.6
        platform: 'state'
        to_state: 'off'        
      - entity_id: alarm_control_panel.alarmo
        prob_given_true: 0.99
        prob_given_false: 0.01
        platform: 'state'
        to_state: 'armed_night'
      - entity_id: alarm_control_panel.alarmo
        prob_given_true: 0.01
        prob_given_false: 0.99
        platform: 'state'
        to_state: 'disarmed'  

[HarvsG]

@JudoWill

The code only triggers (and adds to the “current_obs”) when the state is in the to_state. When the TV leaves the on state, it will fall out of the current_obs list (that’s the else in the last two lines). This leads to an important consequence, the not “to_state” observations are not seen by the bayesian sensor.

I have created a pull request to fix this issue: https://github.com/home-assistant/core/pull/67631

@JudoWill if you felt like reviewing my code and/or reasoning that would be amazing and I would be very greatful

[HarvsG]

These should work

    - platform: "template" 
      prob_given_true: 0.95
      prob_given_false: 0.1
      value_template: >-
        {% if is_state('binary_sensor.device_tracker.gps1', 'home')
           and is_state('binary_sensor.device_tracker.gps1', 'home') %}
           true
        {% elif is_state('binary_sensor.device_tracker.gps1', 'not_home')
           and is_state('binary_sensor.device_tracker.gps1', 'not_home') %}
           false
        {% endif %}

And because you have to give the inverse until my PR is merged

    - platform: "template" 
      prob_given_true: 0.9
      prob_given_false: 0.05
      value_template: >-
        {% if is_state('binary_sensor.device_tracker.gps1', 'home')
           and is_state('binary_sensor.device_tracker.gps1', 'home') %}
           false
        {% elif is_state('binary_sensor.device_tracker.gps1', 'not_home')
           and is_state('binary_sensor.device_tracker.gps1', 'not_home') %}
           true
        {% endif %}

If they disagree they will evaluate to null and so should be ignored by Bayesian.

P.S your probabilities look too conservative - so I’ve tweaked them in my example. Assumes they will both accidentally read ‘home’ when you are away 5% of the time and that they will both read “not_home” when you are home 10% of the time, which is probably still too conservative

[HarvsG]

@JudoWill

If we want the TV being OFF to indicate we are likely to be away, we need to adjust the sensor.

prior: 0.58 # home roughly 14/24 hours per day
observations:
  - platform: state
    entity_id: sensor.tv
    prob_given_true: 0.33  #4 hours of TV per 10 hours I’m home
    prob_given_false: 0.017 #15 minutes of TV per 14 hours away, I don’t like zeros here
    to_state: on
  - platform: state
    entity_id: sensor.tv
    prob_given_true: 0.66 # 8/12 Hours off while home
    prob_given_false: 0.946 # 13.75/14 hours off while I’m away
    to_state: off # this is the “TRUE” state now

Now, when the TV is on. It will have p=0.96
When the TV is off. It will have p=0.49

Why do the inverse situations in prob_given_false not sum to 1?
13.75/14 = 0.982?

[RKor (Rani)]

Hey guys,

What do we do if the home occupiers don’t really have a routine?
We work from home and the house MIGHT be empty a couple hours a week

[HarvsG]

Looks good, but mathematically the inverse probabilities should sum to 1.

      - entity_id: group.alarmo_device_tracker
        prob_given_true: 0.99  # If I'm in bed then I have to be home - hence 100% probability
        prob_given_false: 0.7 # I could be home and not in bed 70% of the time during any given day
        platform: 'state'
        to_state: 'home'
      - entity_id: group.alarmo_device_tracker
        prob_given_true: 0.01 # If I'm in bed, then I can't be not home
        prob_given_false: 0.3 # I'm not in bed and not home (i.e. at work or school etc.) 
        platform: 'state'
        to_state: 'not_home' 

[erika.mason24]

@JudoWill

You’re right, as far as “added sensors” I probably only have 4 or 5. But I use a lot of time-of-day sensors. In the pre-COVID times my wife and I had regular hours of the day when we left the house for work (maybe that will come back some day). Same with evening and night times when we were asleep. I also integrate my calendar and the workday sensor.

Time of day sensors are SUPER useful for occupancy and sleep sensors. While tod itself doesn’t provide much information they can help distinguish between an asleep house, an empty house, or a house where someone is quietly reading by the window.

I’d love to see some of yours as further examples.

I’m attempting to do room occupancy based on tod, motion, room power usage and other room occupancy and power usage but feel I’m missing the mark in having it only move a few points rather than the current drastic changes I see.

[dvbit]

@JudoWill

I can estimate this by saying: “I work ~8 hours 5 days a week, have other ~2 hours of non-house activities per day”. So, I estimate that, knowing nothing, my house is empty ~10 hours per day, which means it is occupied 14 hours per day. So, I’d set my prior at ~14/24 ~= 0.58.

that is not actually knowing nothing

[teskanoo]

I think I would like to see a bit of groundswell behind this proposal - the negated state should have consequence for the calculation. Implementing this would encourage my Bayes sensors to “turn off” / “dial down” the probability when their contributing inputs are false

[HarvsG]

I have a PR:

Fix Bayesian sensor to use negative observations

I just need to improve the tests, unfortunately I am a bit pressed for time at the moment

Edit: PR is reviewer approved and awaiting merge.
Edit2: @teskanoo the PR is now merged, not sure what release it will be in. Hopefully 2022.10
Edit3: Released in 2022.10 - this is a breaking change but I also included some repairs which should detect and notify for most broken configs.

[silvio]

This explanation is top! I wish all teachers would be like this. Keep up the good work mate.