[monitor] Reliable, Multi-User, Distributed Bluetooth Occupancy/Presence Detection

A lot of cars turn their bluetooth radios off when the ignition is off - the answer will depend on your car.

You probably caught this version between updates that I pushed. Iā€™d recommend updating again. Current beta version is 0.1.948 (~430 commits ahead of master).

For your testing purposes with non-Apple devices (docs will be updated on beta eventuallyā€¦ sorry), replace the PASS filters with .* and replace each FAIL filter with NONE or some other garbage non-null string.

Then run from the command line and watch how your phone presents itself as a random advertisement. Some phones will say ā€œGoogleā€ while others say ā€œunknownā€ or another chipset manufacturer. When in doubt on the filters, use .* to match anything.

ok updated

so if my phones are reporting manufacturer as ā€œUnknownā€ and ā€œSamsung Electronics CoLtdā€ I should add those to the pass regex as ā€œUnknown|Samsungā€ ?

whatā€™s adv_flags ?
edit: nevermind, i see youā€™ve updated the documentation for beta. iā€™ll read that first.

1 Like

Yes, this format is generally correct.

However, for ā€œunknownā€ device, the best option is to disable the filter. Unknown devices report in inconsistent ways and might not always match the string ā€œunknownā€

so if my manufacturer is ā€œUnknownā€ and might possibly change will my flag change?
where do i find my flag values?

you are going to want to do this if your device is sending unknown.

#ARRIVE TRIGGER FILTER(S)
PREF_PASS_FILTER_ADV_FLAGS_ARRIVE=".*"
PREF_PASS_FILTER_MANUFACTURER_ARRIVE=".*"

#ARRIVE TRIGGER NEGATIVE FILTER(S)
PREF_FAIL_FILTER_ADV_FLAGS_ARRIVE="NONE"
PREF_FAIL_FILTER_MANUFACTURER_ARRIVE="NONE"
1 Like

Yes, thatā€™s what Iā€™ve done. (That should probably be the default when this is merged with master).

My question was where does one find the flag value?
Or maybe Iā€™m not seeing it because my phones arenā€™t randomly broadcasting, and none of this matters anyways.

1 Like

Yes, it will be. Beta is still in testing. :slight_smile:

Iā€™m fairly sure I also have an Android phone that doesnā€™t do BLE advertisements. Mine is a OnePlus 5. I have however found a ā€œworkaroundā€ that seems to be reliable. I have installed the app Beacon Simulator. If I create a beacon in the app and contiously have it running it seems I also get the ADV_RAND messages in the monitor.sh log and presence detection is working very reliable.

2 Likes

so you have to have that app running all the time?

iā€™ve a partial workaround for my phones not broadcasting. its partial since I only have the one pi running monitor currently, and it doesnā€™t adequately cover my house.

anyways, its basically a modified version of the automation sequence that @benjimatt posted here

  • when the door is opened I stop the depart sequence from running.
  • when my door is closed i check whether both phones are away (0 confidence)
    ā€” if so then I run an arrive scan.
    ā€” if not then i run the depart sequence from the linked post above. I donā€™t do the restart sequence (because I only have the one pi running monitor, so it canā€™t be out of sync with itself). instead at the end of the depart sequence I do an arrival scan.

the obvious problem is that one of the phones can be in the house but outside of the piā€™s bluetooth range. so it comes back as not home. not a huge problem for me since iā€™m also tracking based on WiFi and a bunch of logic based stuff surrounding my doors and motion sensors. I might add some additional trigger for an arrival scan like triggering a certain motion sensor that is close to pi.

hereā€™s my node red flow for the door open trigger depart/arrive scans.

[{"id":"200d9ab3.10c406","type":"server-state-changed","z":"46af5846.433258","name":"Front Door Closed","server":"37567a14.bdb636","entityidfilter":"binary_sensor.door_front","entityidfiltertype":"exact","outputinitially":false,"state_type":"str","haltifstate":"","halt_if_type":"str","halt_if_compare":"is","outputs":1,"x":110,"y":400,"wires":[["68fcd0e0.206c2"]]},{"id":"68fcd0e0.206c2","type":"rbe","z":"46af5846.433258","name":"","func":"rbe","gap":"","start":"","inout":"out","property":"payload","x":270,"y":400,"wires":[["1172a172.e3f3cf"]]},{"id":"1172a172.e3f3cf","type":"switch","z":"46af5846.433258","name":"","property":"payload","propertyType":"msg","rules":[{"t":"eq","v":"off","vt":"str"},{"t":"else"}],"checkall":"false","repair":false,"outputs":2,"x":310,"y":480,"wires":[["63924c3.0141db4"],["a47dbf0d.a0485"]]},{"id":"7d37fa8f.716e94","type":"mqtt out","z":"46af5846.433258","name":"","topic":"monitor/scan/depart","qos":"","retain":"","broker":"51a5b2b3.56d4ec","x":1180,"y":300,"wires":[]},{"id":"dfd41ded.0d2aa","type":"looptimer","z":"46af5846.433258","duration":"10","units":"Second","maxloops":"3","maxtimeout":"1","maxtimeoutunits":"Minute","name":"","x":960,"y":360,"wires":[["7d37fa8f.716e94"],["81071387.0d62c"]]},{"id":"fd0bd0bf.d3c16","type":"delay","z":"46af5846.433258","name":"","pauseType":"delay","timeout":"10","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":720,"y":360,"wires":[["dfd41ded.0d2aa"]]},{"id":"a47dbf0d.a0485","type":"change","z":"46af5846.433258","name":"Clear Loop","rules":[{"t":"set","p":"payload","pt":"msg","to":"stop","tot":"str"},{"t":"set","p":"reset","pt":"msg","to":"reset","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":490,"y":500,"wires":[["fd0bd0bf.d3c16","dfd41ded.0d2aa","81071387.0d62c"]]},{"id":"e63db97e.b1b6f8","type":"mqtt out","z":"46af5846.433258","name":"","topic":"monitor/scan/arrive","qos":"","retain":"","broker":"51a5b2b3.56d4ec","x":1330,"y":420,"wires":[]},{"id":"81071387.0d62c","type":"delay","z":"46af5846.433258","name":"","pauseType":"delay","timeout":"10","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":1160,"y":420,"wires":[["e63db97e.b1b6f8"]]},{"id":"63924c3.0141db4","type":"function","z":"46af5846.433258","name":"BT Monitor Values","func":"const haCtx = global.get('homeassistant');\nconst configCtx = haCtx.homeAssistant;\nconst jBTState = configCtx.states['sensor.rpibt_jphone'];\nif ( jBTState != \"0\" ) { return [null, msg]; }\nconst eBTState = configCtx.states['sensor.rpibt_ephone'];\nif ( eBTState != \"0\" ) { return [null, msg]; }\n\nreturn [msg, null];","outputs":2,"noerr":0,"x":510,"y":340,"wires":[["544f6fa7.8de78"],["fd0bd0bf.d3c16"]]},{"id":"544f6fa7.8de78","type":"mqtt out","z":"46af5846.433258","name":"","topic":"monitor/scan/arrive","qos":"","retain":"","broker":"51a5b2b3.56d4ec","x":750,"y":260,"wires":[]},{"id":"37567a14.bdb636","type":"server","z":"","name":"Home Assistant","legacy":false,"hassio":false,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open"},{"id":"51a5b2b3.56d4ec","type":"mqtt-broker","z":"","name":"","broker":"","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]
2 Likes

Yes, I have the app running at all times. I have three Pi Zero W covering my house using sample config provided by andrewjfreyer with one mqtt sensor per Pi and a min_max sensor. Itā€™s been running reliable for me for a week now

1 Like

Iā€™ve been using this for a week now and quite honestly, Iā€™m still in awe :). The Nmap scanner which I still have running is grossly inaccurate by comparison.

Monitor is getting it right 99% of the timeā€¦that 1% is because my RPi is not able to see my whole house, but Iā€™ll be solving that soon. One thing I do notice, my iphone 8 is measured at being consistent on the confidence scale more than the wifeā€™s iphone6sā€¦but monitor still handles even that :slight_smile:

My apple watch never shows up (not sure why, maybe because its permanently connected to my phone).

Just want to say thank you for this. Is there a beer fund?

1 Like

lol, no beer fund haha - Iā€™m just glad this is working well for so many folks.

2 Likes

Wondering if anyone else running master build with 2 rPi Zero W is going offline? Once in a while my send pi will show offline and within HA it will say unknown. I will need to restart the pi or service in order for this to cone back online.

I tried using beta as well but i find it is not fast at detecting my devices. With pi1 i am using -tr and with my second pi upstairs i am using -tad.

What are other using? and what are others behave file look like that have more than 1 pi running monitor?

I cant think of any trigger i can use for our front door without a sensor, i do have a cam in the front and can use the motion for the triggers but i would get some false once in a while as well.

Thank you!

1 Like

Yes, this will happen on master if you are in a noisy BT environment. Suggest the beta!

Massive merge into master! Version 0.1.956 merges hundreds of bug fixes and optimizations into the master branch from beta. Please read the new readme and comments in this thread related to beta.

Instead of posting all the many, many changes and updates - Iā€™ll ask everyone read the updated readme. Post questions here, and Iā€™ll do my best to answer if theyā€™re not already addressed in the readmeā€™s FAQs.

GitHub Here

5 Likes

Thanks for all your work on this.

In your most recent update of the install walk through it looks step #9 got dropped (initial run with sudo bash monitor.sh)

Also, add me to the list of people with android phones that donā€™t appear to be sending random advertisements. My wife and I both have samsung galaxy s9s that arenā€™t detected using your suggested hcitool lescan and hcidump method.

I have paired my phone with both regular bluetooth devices and a BLE device (a 2017 nvidia shield remote), and the phone seems to be connected and working just fine with the bluetooth devices. Is it possible that hcitool isnā€™t detecting the advertisements? Or would it definitely be a problem with my phone?

#9 is fixed. Thanks for the note.

For the Samsung Galaxy devices, I think itā€™d be interesting to conduct a few tests off this thread. Can you DM me when itā€™s convenient?

Before I start, just wanted to thank @andrewjfreyer, awesome work sir! Iā€™ve been watching this thread and decided to add a Pi Zero W to my collection so I can join in on the fun!

However, upon setting everything up it doesnā€™t appear to be finding any random advertisements from any of my Android phones, Galaxy S5, J2 Pro or Note 9.

Would be happy to assist debug in anyway I can.

Yes, almost all of the reports of no advertisement have been from Android phones. Iā€™m going to borrow/buy a few for testing.

Thanks for the report!

1 Like