[Monitor] Reliable, Multi-User, Distributed BT Occupancy/Presence Detection

automation
mqtt
Tags: #<Tag:0x00007f2054da3330> #<Tag:0x00007f2054da3178>

#829

A lot of cars turn their bluetooth radios off when the ignition is off - the answer will depend on your car.


#830

You probably caught this version between updates that I pushed. I’d recommend updating again. Current beta version is 0.1.948 (~430 commits ahead of master).

For your testing purposes with non-Apple devices (docs will be updated on beta eventually… sorry), replace the PASS filters with .* and replace each FAIL filter with NONE or some other garbage non-null string.

Then run from the command line and watch how your phone presents itself as a random advertisement. Some phones will say “Google” while others say “unknown” or another chipset manufacturer. When in doubt on the filters, use .* to match anything.


#831

ok updated

so if my phones are reporting manufacturer as “Unknown” and “Samsung Electronics CoLtd” I should add those to the pass regex as “Unknown|Samsung” ?

what’s adv_flags ?
edit: nevermind, i see you’ve updated the documentation for beta. i’ll read that first.


#832

Yes, this format is generally correct.

However, for “unknown” device, the best option is to disable the filter. Unknown devices report in inconsistent ways and might not always match the string “unknown”


#833

so if my manufacturer is “Unknown” and might possibly change will my flag change?
where do i find my flag values?


#834

you are going to want to do this if your device is sending unknown.

#ARRIVE TRIGGER FILTER(S)
PREF_PASS_FILTER_ADV_FLAGS_ARRIVE=".*"
PREF_PASS_FILTER_MANUFACTURER_ARRIVE=".*"

#ARRIVE TRIGGER NEGATIVE FILTER(S)
PREF_FAIL_FILTER_ADV_FLAGS_ARRIVE="NONE"
PREF_FAIL_FILTER_MANUFACTURER_ARRIVE="NONE"

#835

Yes, that’s what I’ve done. (That should probably be the default when this is merged with master).

My question was where does one find the flag value?
Or maybe I’m not seeing it because my phones aren’t randomly broadcasting, and none of this matters anyways.


#836

Yes, it will be. Beta is still in testing. :slight_smile:


#838

I’m fairly sure I also have an Android phone that doesn’t do BLE advertisements. Mine is a OnePlus 5. I have however found a “workaround” that seems to be reliable. I have installed the app Beacon Simulator. If I create a beacon in the app and contiously have it running it seems I also get the ADV_RAND messages in the monitor.sh log and presence detection is working very reliable.


#839

so you have to have that app running all the time?

i’ve a partial workaround for my phones not broadcasting. its partial since I only have the one pi running monitor currently, and it doesn’t adequately cover my house.

anyways, its basically a modified version of the automation sequence that @benjimatt posted here

  • when the door is opened I stop the depart sequence from running.
  • when my door is closed i check whether both phones are away (0 confidence)
    — if so then I run an arrive scan.
    — if not then i run the depart sequence from the linked post above. I don’t do the restart sequence (because I only have the one pi running monitor, so it can’t be out of sync with itself). instead at the end of the depart sequence I do an arrival scan.

the obvious problem is that one of the phones can be in the house but outside of the pi’s bluetooth range. so it comes back as not home. not a huge problem for me since i’m also tracking based on WiFi and a bunch of logic based stuff surrounding my doors and motion sensors. I might add some additional trigger for an arrival scan like triggering a certain motion sensor that is close to pi.

here’s my node red flow for the door open trigger depart/arrive scans.

[{"id":"200d9ab3.10c406","type":"server-state-changed","z":"46af5846.433258","name":"Front Door Closed","server":"37567a14.bdb636","entityidfilter":"binary_sensor.door_front","entityidfiltertype":"exact","outputinitially":false,"state_type":"str","haltifstate":"","halt_if_type":"str","halt_if_compare":"is","outputs":1,"x":110,"y":400,"wires":[["68fcd0e0.206c2"]]},{"id":"68fcd0e0.206c2","type":"rbe","z":"46af5846.433258","name":"","func":"rbe","gap":"","start":"","inout":"out","property":"payload","x":270,"y":400,"wires":[["1172a172.e3f3cf"]]},{"id":"1172a172.e3f3cf","type":"switch","z":"46af5846.433258","name":"","property":"payload","propertyType":"msg","rules":[{"t":"eq","v":"off","vt":"str"},{"t":"else"}],"checkall":"false","repair":false,"outputs":2,"x":310,"y":480,"wires":[["63924c3.0141db4"],["a47dbf0d.a0485"]]},{"id":"7d37fa8f.716e94","type":"mqtt out","z":"46af5846.433258","name":"","topic":"monitor/scan/depart","qos":"","retain":"","broker":"51a5b2b3.56d4ec","x":1180,"y":300,"wires":[]},{"id":"dfd41ded.0d2aa","type":"looptimer","z":"46af5846.433258","duration":"10","units":"Second","maxloops":"3","maxtimeout":"1","maxtimeoutunits":"Minute","name":"","x":960,"y":360,"wires":[["7d37fa8f.716e94"],["81071387.0d62c"]]},{"id":"fd0bd0bf.d3c16","type":"delay","z":"46af5846.433258","name":"","pauseType":"delay","timeout":"10","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":720,"y":360,"wires":[["dfd41ded.0d2aa"]]},{"id":"a47dbf0d.a0485","type":"change","z":"46af5846.433258","name":"Clear Loop","rules":[{"t":"set","p":"payload","pt":"msg","to":"stop","tot":"str"},{"t":"set","p":"reset","pt":"msg","to":"reset","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":490,"y":500,"wires":[["fd0bd0bf.d3c16","dfd41ded.0d2aa","81071387.0d62c"]]},{"id":"e63db97e.b1b6f8","type":"mqtt out","z":"46af5846.433258","name":"","topic":"monitor/scan/arrive","qos":"","retain":"","broker":"51a5b2b3.56d4ec","x":1330,"y":420,"wires":[]},{"id":"81071387.0d62c","type":"delay","z":"46af5846.433258","name":"","pauseType":"delay","timeout":"10","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":1160,"y":420,"wires":[["e63db97e.b1b6f8"]]},{"id":"63924c3.0141db4","type":"function","z":"46af5846.433258","name":"BT Monitor Values","func":"const haCtx = global.get('homeassistant');\nconst configCtx = haCtx.homeAssistant;\nconst jBTState = configCtx.states['sensor.rpibt_jphone'];\nif ( jBTState != \"0\" ) { return [null, msg]; }\nconst eBTState = configCtx.states['sensor.rpibt_ephone'];\nif ( eBTState != \"0\" ) { return [null, msg]; }\n\nreturn [msg, null];","outputs":2,"noerr":0,"x":510,"y":340,"wires":[["544f6fa7.8de78"],["fd0bd0bf.d3c16"]]},{"id":"544f6fa7.8de78","type":"mqtt out","z":"46af5846.433258","name":"","topic":"monitor/scan/arrive","qos":"","retain":"","broker":"51a5b2b3.56d4ec","x":750,"y":260,"wires":[]},{"id":"37567a14.bdb636","type":"server","z":"","name":"Home Assistant","legacy":false,"hassio":false,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open"},{"id":"51a5b2b3.56d4ec","type":"mqtt-broker","z":"","name":"","broker":"","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

#840

Yes, I have the app running at all times. I have three Pi Zero W covering my house using sample config provided by andrewjfreyer with one mqtt sensor per Pi and a min_max sensor. It’s been running reliable for me for a week now


#841

I’ve been using this for a week now and quite honestly, I’m still in awe :). The Nmap scanner which I still have running is grossly inaccurate by comparison.

Monitor is getting it right 99% of the time…that 1% is because my RPi is not able to see my whole house, but I’ll be solving that soon. One thing I do notice, my iphone 8 is measured at being consistent on the confidence scale more than the wife’s iphone6s…but monitor still handles even that :slight_smile:

My apple watch never shows up (not sure why, maybe because its permanently connected to my phone).

Just want to say thank you for this. Is there a beer fund?


#842

lol, no beer fund haha - I’m just glad this is working well for so many folks.


#843

Wondering if anyone else running master build with 2 rPi Zero W is going offline? Once in a while my send pi will show offline and within HA it will say unknown. I will need to restart the pi or service in order for this to cone back online.

I tried using beta as well but i find it is not fast at detecting my devices. With pi1 i am using -tr and with my second pi upstairs i am using -tad.

What are other using? and what are others behave file look like that have more than 1 pi running monitor?

I cant think of any trigger i can use for our front door without a sensor, i do have a cam in the front and can use the motion for the triggers but i would get some false once in a while as well.

Thank you!


#844

Yes, this will happen on master if you are in a noisy BT environment. Suggest the beta!


#845

Massive merge into master! Version 0.1.956 merges hundreds of bug fixes and optimizations into the master branch from beta. Please read the new readme and comments in this thread related to beta.

Instead of posting all the many, many changes and updates - I’ll ask everyone read the updated readme. Post questions here, and I’ll do my best to answer if they’re not already addressed in the readme’s FAQs.

GitHub Here


#846

Thanks for all your work on this.

In your most recent update of the install walk through it looks step #9 got dropped (initial run with sudo bash monitor.sh)

Also, add me to the list of people with android phones that don’t appear to be sending random advertisements. My wife and I both have samsung galaxy s9s that aren’t detected using your suggested hcitool lescan and hcidump method.

I have paired my phone with both regular bluetooth devices and a BLE device (a 2017 nvidia shield remote), and the phone seems to be connected and working just fine with the bluetooth devices. Is it possible that hcitool isn’t detecting the advertisements? Or would it definitely be a problem with my phone?


#847

#9 is fixed. Thanks for the note.

For the Samsung Galaxy devices, I think it’d be interesting to conduct a few tests off this thread. Can you DM me when it’s convenient?


#848

Before I start, just wanted to thank @andrewjfreyer, awesome work sir! I’ve been watching this thread and decided to add a Pi Zero W to my collection so I can join in on the fun!

However, upon setting everything up it doesn’t appear to be finding any random advertisements from any of my Android phones, Galaxy S5, J2 Pro or Note 9.

Would be happy to assist debug in anyway I can.


#849

Yes, almost all of the reports of no advertisement have been from Android phones. I’m going to borrow/buy a few for testing.

Thanks for the report!