[monitor] Reliable, Multi-User, Distributed Bluetooth Occupancy/Presence Detection

I tried the above, and got to the following conclusions:

Faster leave state detection
I’ve set PREF_BEACON_EXPIRATION to 150 seconds. I could go lower, but then it starts to report false positives (on leaving). Just around 50% confidence it hits 100% again. This could still work with HA, but I’d like to keep MQTT traffic as low as possible considering I’ll probably be using this protocol a lot in the future.

Reducing noise
I’ve set PREF_FAIL_FILTER_MANUFACTURER_ARRIVE="Apple|Samsung|Google" and blacklisted a couple of my devices that do not meet this criteria. I feel the noise is a lot less indeed. But strangely it still allows 2 Samsung TV’s from my neighbour with manufacturer Samsung Electronics Co. Ltd.. The type is GENERIC_BEACON_PUBLIC, so shouldn’t these devices be ignored as well?

I’m curious how you did this. Are you running a script on your raspberry pi’s that listens to a MQTT topic, and you publish to that topic when HA restarts?

1 Like

Now I’m noticing some odd behaviour. When I restart HA (which I do a lot) my home state defaults to not_home, though the confidence of both floors is 100. HA sees this as well since all messages are retained.

Restarting both monitor services it reports 100 confidence again, but HA isn’t changing the home state. When I actually move my Nut out of reach, as soon as it drops below 100 (and above 10) confidence HA succesfully changes the state to home.

As far as I know HA always changed the home state according to the confidence retreived, even directly after restarting. What am I messing up here?

edit: I can solve it through this automation, but I seriously doubt this was necessary before:

automation:
  - alias: Get presence on HA restart
    hide_entity: true
    trigger:
      - platform: homeassistant
        event: start
    action:
      - service: device_tracker.see
        data_template:
          dev_id: rourke
          location_name: >
            {% if states('sensor.home_occupancy_confidence_rourke') | int > 10 %}
              home
            {% else %}
              not_home
            {% endif %}
          source_type: bluetooth

I’m on my way out to work, so not a lot of time to look at this right now. Can you put the MAC addresses for your neighbor’s TVs in the blacklist file?

As far as the automation goes…I’m using the “not so binary presence detection” from here:

and so I have several automations on HA restart, for example, one of them is:

alias: 'Mark Anwen as home when HA starts'
trigger:
  - platform: homeassistant
    event: start
condition:
  - condition: state
    entity_id: device_tracker.anwen
    state: 'home'
action:
  - service: input_select.select_option
    data:
      entity_id: input_select.anwen_status_dropdown
      option: Home
  - service: input_boolean.turn_off
    data:
      entity_id: input_boolean.anwen_ext_away

Side note: I’m only running one instance of Monitor, because I don’t need to cover the whole house, just where we put our keys when we are home.

Did you fix this? My husband got a new iPhone as well and it doesn’t show up.

I did that, and it works. I hope it will ignore future devices from those manufacturers though. :slight_smile:

That is quite interesting, although I’m not sure if I’m gonna need this. I have to think about it. But with the automation I’ve posted earlier I expect this runs the way I expect it to run. Although I’m not very happy with the not_home to home transition on every HA boot, since this sets some other automations in motion. :confused: I swear this wasn’t necessary before.

Now that I’m not tracking my phones anymore this is a valid point for me too. I might relocate my Pi Zero W to the garage. That way it detects coming home faster.

Thanks again by the way. Appreciate all your help.

1 Like

Glad things are working somewhat better for you. I forgot to give you this automation for just restarting the Monitor service on HA restart:

alias: 'Restart Monitor on HA startup'
trigger:
  - platform: homeassistant
    event: start
action:
  - delay: '00:00:30'
  - service: mqtt.publish
    data:
      topic: "monitor/scan/restart"

Have a look through the not-so-binary stuff when you have time - it might help with the not_home to home transitions.

Glad my posts have been useful for you - I’m really grateful for the help others have given me, and I’m always happy when I can help someone else!

2 Likes

I wasn’t aware of the restart function in monitor. How good of the creator to have this built in! But just to be sure: if you use -x all MQTT messages are retained. So doesn’t HA automatically get the latest correct data on a restart?

By the way, what do you mean with the ‘not-so-binary stuff’? I’m eager to learn.

By the “not so binary stuff” I meant the link I gave you earlier to Phil Hawthorne’s blog. There’s a section about “Stopping the Just Arrived State” that may or may not be helpful for your issue. I’m not sure why you are getting not_home on a restart - mine shows as unavailable in the history for a very brief time and then has the same status it had earlier. You can watch the MQTT messages in the MQTT section of Developer Tools in HA; maybe that would be helpful to diagnose the issue.

Regarding testing…when I was figuring all this out initially, I’d put my tracker in a small metal tin with a lid to make it be “not home”. It worked great - the tin blocked the BT signal, and I didn’t have to keep getting up from my desk to move my tracker out of range. As soon as I’d open the tin…boom! Home, confidence 100! :grinning:

Nope. Tried once more & still could not see it. unfortunately I don’t have the time nor technical nohow to resolve this & for now have connected my work phone (Android).

Alright, thanks anyway. I will try to look into it when I have the time and make sure to write the solution here if I find it.

1 Like

That is a nice way of testing it. Very clever.

With the persons I configured it kept resetting the state on HA reboots, so after a while working in HA and triggering some automations I got annoyed. I switched back to input_booleans since they remember states on HA reboot. It’s working as it should now.

Another problem arised. After 1 day I checked my MQTT sniffer for devices that have been detected (and so the messages retained) and for every floor that was more than 300! In just 1 day. So I think the solution is to somehow only scan for my preset Nut trackers, just haven’t discovered how yet. If anyone has a solution I’d love to hear it.

Did you get this solved? I have the same problem where script works for some hours and then just doesn’t detect my Samsung S7 anymore. I need to restart script to get it working again.

I’m a little confused when it comes to the regular addresses and the beacons. My wife and I have iPhones. They do respond to name requests so I have those addresses in the known static addresses, but do they also report as a beacon? If they do is there any advantage to figure out their beacon addresses and adding those as well?

Beacons are things like the Tile trackers, not phones.

Ok thanks, I knew the tiles were beacons, but wasnt sure if the phone had the ability to mimic a beacon and also reported as a beacon.

When I run dmesg, the log is totally flooded with

Bluetooth: hci0: advertising data len corrected

is this something I can do anything about?
Thanks!

Any idea what is wrong with my setup? I’ve got Pi Zero W and it’s detecting my Samsung S7 correctly for couple of hours. After a while it stops detecting the phone but the script is still running fine (I get OK when sending “monitor/scan/echo”). Sending arrival scan request from HA goes through to Pi but it still doesn’t find my phone. Even restarting the script by sending “monitor/scan/restart” doesn’t help, I need to reboot the whole Pi (power cycle) before it starts working again. I have also tried reinstalling the whole thing. Below is a part of log messages through ssh.

logs
pi@raspberrypi:~/monitor $ sudo bash monitor.sh -u -x -td -V
> starting monitor.sh (v. 0.2.197)...
> removing web request caches
> updating monitor.service
> retaining mqtt status reports
> warning: verbose logging is enabled. this setting is only for informational and debugging purposes
> monitor.service updated with arguments:  -x -td
> preference: minimum time between the same type of scan = 15
> preference: regex filter for flags to accept = .*
> preference: regex filter for flags to reject = NONE
> preference: regex filter for manufacturers to accept = .*
> preference: regex filter for manufacturers to reject = NONE
> preference: maximum sequential arrive scan attempts = 1
> preference: maximum sequential depart scan attempts = 2
> preference: selected HCI device = hci0
> mqtt trigger: monitor/scan/ARRIVE
> mqtt trigger: monitor/scan/DEPART
> 5C:XX:XX:XX:F8:44 confidence topic: monitor/1/sts7 (has not previously connected to hci0)
> log listener pid = 25913
> btle scan pid = 25914
> btle text pid = 25915
> btle listener pid = 25917
> mqtt listener pid = 25920
> packet listener pid = 25922
> beacon database time trigger pid = 25925
[+] 0.2.197 04-11-2019 05:51:52 PM [CMD-INFO]	**** started arrival scan [x1 max rep] **** 
[+] 0.2.197 04-11-2019 05:51:55 PM [CMD-SCAN]	(No. 1) 5C:XX:XX:XX:F8:44 arrival? 
[+] 0.2.197 04-11-2019 05:51:55 PM [CMD-RAND]	[passed filter] data: 6C:XX:XX:XX:98:4D pdu: ADV_IND rssi: -70 dBm flags: 0x1a man: Apple, Inc. delay: 0
[+] 0.2.197 04-11-2019 05:51:56 PM [CMD-RAND]	[passed filter] data: 44:XX:XX:XX:C2:C0 pdu: ADV_IND rssi: -47 dBm flags: 0x06 man: Apple, Inc. delay: 0
[+] 0.2.197 04-11-2019 05:52:00 PM [CMD-MQTT]	monitor/1/sts7
 {
    "id":"5C:XX:XX:XX:F8:44",
    "confidence":"0",
    "name":"STS7",
    "manufacturer":"Samsung Electronics CoLtd",
    "type":"KNOWN_MAC",
    "retained":"true",
    "timestamp":"Mon Nov 04 2019 17:52:00 GMT+0000 (GMT)",
    "version":"0.2.197"
 }
[+] 0.2.197 04-11-2019 05:52:01 PM [CMD-MQTT]	monitor/1/sts7/rssi
[+] 0.2.197 04-11-2019 05:52:01 PM [CMD-INFO]	**** completed arrival scan **** 
[+] 0.2.197 04-11-2019 05:52:01 PM [CMD-NAME]	5C:XX:XX:XX:F8:44 STS7  Samsung Electronics CoLtd
[+] 0.2.197 04-11-2019 05:52:02 PM [CMD-RAND]	[passed filter] data: FB:XX:XX:XX:26:91 pdu: ADV_IND rssi: -53 dBm flags: 0x06 man: Tile, Inc. delay: 0
[+] 0.2.197 04-11-2019 05:53:00 PM [CMD-RAND]	[passed filter] data: 40:XX:XX:XX:71:8D pdu: ADV_SCAN_IND rssi: -67 dBm flags: none man: Google delay: 3
[+] 0.2.197 04-11-2019 05:53:00 PM [CMD-INFO]	**** started arrival scan [x1 max rep] **** 
[+] 0.2.197 04-11-2019 05:53:01 PM [CMD-SCAN]	(No. 1) 5C:XX:XX:XX:F8:44 arrival? 
[+] 0.2.197 04-11-2019 05:53:09 PM [CMD-INFO]	**** completed arrival scan **** 
[+] 0.2.197 04-11-2019 05:53:09 PM [CMD-NAME]	5C:XX:XX:XX:F8:44 STS7  Samsung Electronics CoLtd
[+] 0.2.197 04-11-2019 05:53:29 PM [CMD-RAND]	[passed filter] data: 5B:XX:XX:XX:41:FA pdu: ADV_SCAN_IND rssi: -61 dBm flags: none man: Google delay: 2
[+] 0.2.197 04-11-2019 05:53:29 PM [CMD-INFO]	**** started arrival scan [x1 max rep] **** 
[+] 0.2.197 04-11-2019 05:53:29 PM [CMD-SCAN]	(No. 1) 5C:XX:XX:XX:F8:44 arrival? 
[+] 0.2.197 04-11-2019 05:53:34 PM [CMD-RAND]	[passed filter] data: 70:XX:XX:XX:17:59 pdu: ADV_SCAN_IND rssi: -60 dBm flags: none man: Google delay: 0
[+] 0.2.197 04-11-2019 05:53:37 PM [CMD-INFO]	**** completed arrival scan **** 
[+] 0.2.197 04-11-2019 05:53:37 PM [CMD-NAME]	5C:XX:XX:XX:F8:44 STS7  Samsung Electronics CoLtd
[+] 0.2.197 04-11-2019 05:54:27 PM [CMD-INST]	[pass mqtt] arrive scan requested 
[+] 0.2.197 04-11-2019 05:54:28 PM [CMD-INFO]	**** started arrival scan [x1 max rep] **** 
[+] 0.2.197 04-11-2019 05:54:28 PM [CMD-INST]	[pass mqtt] arrive scan requested 
[+] 0.2.197 04-11-2019 05:54:29 PM [CMD-SCAN]	(No. 1) 5C:XX:XX:XX:F8:44 arrival? 
[+] 0.2.197 04-11-2019 05:54:33 PM [DEL-RAND]	RAND 40:XX:XX:XX:71:8D expired after 66 seconds 
[+] 0.2.197 04-11-2019 05:54:37 PM [CMD-INFO]	**** completed arrival scan **** 
[+] 0.2.197 04-11-2019 05:54:38 PM [CMD-NAME]	5C:XX:XX:XX:F8:44 STS7  Samsung Electronics CoLtd
[+] 0.2.197 04-11-2019 05:55:22 PM [CMD-INST]	[pass mqtt] echo 
[+] 0.2.197 04-11-2019 05:55:24 PM [CMD-INST]	[pass mqtt] echo 
[+] 0.2.197 04-11-2019 05:55:37 PM [CMD-INST]	[pass mqtt] service restart requested 
Terminated

I’m having problems with the arrive/depart message, can someone please help me figure out what’s going on? I feel like I went through the entire thread twice by now :wink:

I’ve setup monitor on a Pi ZeroW (tried a 3B+, same behavior) and most of it seems to work fine. I can see messages through MQTT and I’m also able to add new MACs through monitor/setup/ADD STATIC DEVICE so MQTT is working both ways. Still, whatever I try, publishing to monitor/scan/depart and monitor/scan/arrive doesn’t seem to do anything! I’m looking at the monitor.sh bash script for logs and I’m aware that arrive and depart don’t get triggered if all known devices are home or away respectively. I have two phones added and one with BT on, the other off.

I don’t know where to look to further investigate. Other people seem to have had the some problem but the solutions posted in this thread haven’t helped me so far. I really hope this can work out, I had an early version of monitor working a year ago and it was just great for my situation. But because of happy family circumstances my home automation setup suffered from neglect and I’m basically starting from scratch again. Thx for any help!

Any luck figuring this out without automation? I’m running into the same problem.

Unfortunately no. Whatever I did it still resets to not_home on HA boot. In the end I reverted back to input_booleans which remember their state. It’s doesn’t feel like ‘the proper way’, but it works quite well.