[monitor] Reliable, Multi-User, Distributed Bluetooth Occupancy/Presence Detection

Thanks! I’ll experiment with these settings tomorrow evening.

Okey, sorry if this is a noobish kinda question. I really do understand that you already thought of this but I wanna learn and understand :slight_smile:

Why do monitor care about other devices outside known_static_addresses?
Couldn’t the noise be eliminated with having monitor listening for just those known devices?
And how can the noise of all the “guests” phones make noise in my HA sensor? The only sensor I have within home assistant is state_topic: 'rpizero/monitor/MY:PHONES:MAC:ADDRESS'?

Thanks for your answer, patience and of course wonderful contribution with monitor!

1 Like

@fillilutten, these are common questions that aren’t sufficiently explained in the documentation. Sorry! Above, I wrote an analogy to try to highlight the difference between presence and monitor and why each functions in the way it functions. The following analogy is a bit different.

Blackout Gymnasium

You’re in a pitch black gymnasium along with some other people.

Some of the people in the gym periodically make sounds (e.g., eating a chip/crisp, dribble a basketball, cough, etc.), others sit quietly and don’t make a sound unless you specifically ask them and still others periodically say their own name out loud at regular intervals whether or not you want them to.

For privacy reasons, we can’t just shout “WHO’S HERE” because then everyone would say their name at once and we couldn’t hear anything. So, all have agreed that anyone in the gym who might respond will only respond if their own name is called, like taking attendance in a classroom.

You’d like to know if your friends are in the gym too. How can you find out if your friends are there with you if you can’t see them? How can you find out if your friends have left the gym if you can’t see them?

Well, if our friends say their name out loud it’s easy to know if they’re present or absent. All we have to do is listen. Otherwise, we don’t know if our friends are present without asking for them one at a time since none of the sounds we hear are clearly associated with any particular person and our other friends won’t respond unless we specifically ask for them.

So, let’s come up with a strategy to figure out which of our friends are in the gym.

  • Strategy 1: Ask your friends if they are there at regular intervals. If a friend doesn’t respond, you’ll presume they’ve left.

  • Strategy 2: Wait until you hear a sound, then ask our friends if they’re present.

Strategy 1

Here’s how Strategy 1 might play out if my friends are John and Jane:

Me:     Is John here?
(WAIT FOR 5 SECONDS FOR JOHN TO RESPOND)
Me (to self) - JOHN is not here.  
Me:     Is Jane here? 
Anon:   Jane is here!
Me (to self) - JANE is here. 

(WAIT 30 SECONDS)

Me:     Is John here?
(WAIT FOR 5 SECONDS FOR JOHN TO RESPOND)
Me (to self) - JOHN is not here.  
Me:     Is Jane here?
Anon:   Jane is here!
Me (to self) - JANE is here. 

(WAIT 30 SECONDS)

Me:     Is John here?
Anon:   John is here!
Me (to self) - JOHN is here. 
Me:     Is Jane here?
(WAIT FOR 5 SECONDS FOR JANE TO RESPOND)
Me (to self) - JANE is not here.
...  

After asking for John and not hearing a response, we presume that John is not in the gym. We also presume that Jane is in the gym with us because she answered. Later, we determine that John has arrived and Jane has left.

The only problem with this strategy is that we’re constantly shouting into the gym. That makes it difficult for other people in the gym to hear.

Strategy 2

Here’s how Strategy 2 might play out if my friends are John and Jane:

...
(--- SOUND ---)
Me:     Is John here? 
(WAIT FOR 5 SECONDS FOR JOHN TO RESPOND)
Me (to self) - JOHN is not here.  
Me:     Is Jane here? 
Anon:   Jane is here!
Me (to self) - JANE is here.  

(FIVE MINUTES PASS)

(--- SOUND ---)
Me:     Is John here?
(WAIT FOR 5 SECONDS FOR JOHN TO RESPOND)
Me (to self) - JOHN is not here.  

(THIRTY SECONDS PASS)

(--- SOUND ---)
Me:     Is John here?
Anon:   John is here!
Me (to self) - JOHN is here.  

Again, after asking for John and not hearing a response we can presume that John is not in the gym with us. We also presume that Jane is in the gym with us because she answered.

Later, we hear another sound and ask for John (we don’t need to ask for Jane since we know she’s already there). With this strategy, we’ll have to periodically ask whether Jane and John are still present, but we’ll get to that a bit later.


What does this have to do with bluetooth?

The blackout gym is analogous to the bluetooth environment around a raspberry pi.

Some bluetooth devices in the environment periodically broadcast a random address (i.e., make an untraceable/unidentifiable sound), some bluetooth devices sit quietly and don’t broadcast unless you specifically ask them, and still others periodically broadcast their own name out loud at regular intervals whether or not you want them to.

  • Modern phones and some beacons are the first category of devices. They advertise randomly, but they do not advertise themselves.

  • Older phones and other bluetooth devices do not advertise at all, but will respond to a request if you know their MAC address.

  • Some beacon bluetooth devices will advertise their name (and/or other information) publicly whether or not you want them to.

presence employs Strategy 1 above and monitor employs Strategy 2 above to check for the presence of your phones. Since presence only asks known devices whether or not they are in the nearby bluetooth environment, it cannot detect beacons while monitor can.

More simply:

  • presence periodically asks every device in your configuration whether it is present. This adds noise to the 2.4GHz environment (sound in the blackout gym) and may interfere with other devices.

  • monitor - with default settings - waits for a random advertisement (unidentifiable noise) and then asks every device in your configuration that is absent whether it is present. Since it is already listening for random advertisements, monitor also reports beacons/ibeacons that advertise their own name.


So, @fillilutten , in answer to your question, monitor is “interested” in the entire bluetooth environment beacuse your “known devices” only make unidentifiable advertisements/sounds. Your phone does not say “ANDREW’S PHONE” or “PHONE 12345” - it only says “THERE IS A DEVICE HERE”. We still have to figure out which device it is.

That’s why we need to “monitor” the bluetooth environment instead of just waiting for your phone to advertise itself (since your phone doesn’t do that!).

I hope that helps!

10 Likes

Do the service parameters get updated based on the parameters I last used when running the script from the terminal?

When I run it through terminal everything works normally, but when I run sudo systemctl restart monitor.service, the heartbeat isn’t sent and journalctl -u monitor.service seems to show the monitor only running arrival scans and not departures.

I tried turning off my phone to test if departure scans were running, but it hasn’t updated for over 5 minutes.

I last ran the script with sudo bash monitor.sh -m -e -x -g -a -b -E

1 Like

If you include -u when running from terminal, then the systemctl script will be updated. Otherwise, the script parameters are just runtime options.

The default trigger for departure is an expiration of a random advertisement. For testing, make sure that bluetooth is fully off. In iOS 12, tapping “bluetooth” in the control center does not disable the bluetooth radio. You need to disable from settings.

1 Like

Ahh of course, I read that at one stage and used the -u flag, but it fell out of my brain.

Thank you for the awesome script and your dedicated support in this thread!

1 Like

@andrewjfreyer Im on 0.1.674 and it seems to be working just fine. Decimal is now gone from the message sent to the mqtt server.

btw thanks for the new update trigger. Going to be nice in the future.

location/scan/update

Can I recommend you do something like this. (just a suggestion) (at least separating a different restart trigger)

		elif [[ $mqtt_topic_branch =~ .*RESTART.* ]]; then 
			log "${GREEN}[INSTRUCT] ${NC}mqtt restart  ${NC}"
			
			#RESTART SYSTEM
			systemctl restart monitor.service		

		elif [[ $mqtt_topic_branch =~ .*UPDATEBETA.* ]]; then 
			log "${GREEN}[INSTRUCT] ${NC}mqtt update beta  ${NC}"
			
			#GIT FETCH
			git fetch
			
			#GIT FETCH
			git checkout beta				
			
			#GIT PULL
			git pull

			#RESTART SYSTEM
			systemctl restart monitor.service	
			
		elif [[ $mqtt_topic_branch =~ .*UPDATE.* ]]; then 
			log "${GREEN}[INSTRUCT] ${NC}mqtt update master  ${NC}"
			
			#GIT FETCH
			git fetch
			
			#GIT FETCH
			git checkout master				
			
			#GIT PULL
			git pull

			#RESTART SYSTEM
			systemctl restart monitor.service				
		fi
1 Like

Any idea how I can avoid creating a sensor per device per location in HA?

(4 PI’s on two floors) and 5-6 occupants. is there a way to make the mqtt topic sensor friendly?

@andrewjfreyer, I have the -g flag set and the mac address of the nut in the known_beacons file. The problem is that confidence never comes above 0 (and hence 0 is reported over mqtt to HASS). The two Kontakt iBeacons and a few iPhones we have works, but the nut and Apple Watch 4 I haven’t been able to get to work yet (confidence stays at 0).

Any other ideas ? :slight_smile: (the nut is being detected by room assistant if I start that, so I’m sure the bluetooth mac address is correct).

Thanks!

1 Like

Can anyone recommend a bluetooth dongle I can plug into my Ubuntu server laptop to use with this? Are there any particular requirements?

Is it fair to say that using the option -r on its own, effectively replicates the functionality of presence i.e. only looks for known devices and does so regularly (strategy 1)? Adding in -a will report the result of every scan (not just changes). It also only works for devices that respond to a ‘call out’.

The always listening function (strategy 2) is enabled by the use of -b and or -g options? This also works with devices that only make a noise or shout out.

So if you are in a relatively small location (one Monitor) and have no means of triggering scans (no door sensors), using -r -a will reliably report presence of known devices that respond to calls. The repeat frequency can be set to minimise wireless interference (even as often as 1 minute should not make a difference I suggest). The granularity of reporting presence is an individual choice with obvious trade-offs.

1 Like

I tried NUT Mini but no luck. Having the same error logs as you (confidence: 0).
Did you manage to get it working?
Working perfectly with:
-iphone 5
-iphone 8
-old samsung (android)
-BlackBerry

I was so wrong! I’ve done some testing today and the Pi is definitely the cause of my WiFi interference. So I absolutely need to try to minimise the bluetooth scanning. Any tips on that are most welcome!
It’s only my Particle Core devices which seem to get knocked offline by the Pi, they use a CC3000 Wifi chip. The newer Particle Photon devices seem more tolerant of the interference and can maintain the connection. Well, I think they can maintain the connection. They might be dropping crucial MQTT packets and I don’t know about it.

Yes, that’s what -r does.

Nice. I’ll add this.

I think I need to pick one up to see what’s going on with these devices.

OK, so I need to cut down drastically the bluetooth interference my WiFi is suffering at the moment. So I’m going to try “trigger only”. I have two Pi Zeros. Thing is, I’m a bit confused. A door opening or closing could be somebody leaving or somebody arriving. So do I send an MQTT "monitor/scan/ARRIVE" or "monitor/scan/DEPART" when a door is opened?
And I guess in this scenario I set both Pis up wholly identically (just different hostnames and my router will dish out different IPs) and then I make an average of their scans in Home Assistant? THANKS

1 Like

If using a door sensor (Front door / backdoor) I would make these trigger a depart scan
monitor/scan/DEPART
If you are wanting to use an arrive scan I think you would need a different trigger that isnt a door. Maybe a garage trigger or a Gate trigger.

1 Like

Thanks for this explanation - it is very hard to find such detailed BT related info on the web.
You seem to be quite knowledgeable regarding BT. Maybe you could help me with some questions.
For the context: I live in an apartment on 4th floor next to the elevator. There is only one enterence to my home.

  • If I understand correctly “monitor” requires exclusive access to BT device. Since I’m using my PI zeros for MiFloras I would need to get another BT adapter. Is that right ?
  • Is there any major advantage in using “monitor” instead of plain "hcitool name " triggered on main door being open/closed. I’ve already have appdaemon installed on my slave PI zeros so I have easy access to HA state. The most appealing thing about “monitor” is BT domain knowledge embedded in the script, however if I could achieve mostly the same with “hcitool name” and replicating confidence algo then I could adapt the solution more to my setup/needs/domain knowledge.
  • Is “monitor” susceptible to BT MAC spoofing ?
  • I use BT presence detection to open my front door, however IIUC "hcitool name " can be MAC spoofed, hence in addition I use "hcitool auth " (device is already paired). Is this considered safe (I’m aware of BT range extension attack) ?
  • I have dedicated BT devices for each phone that constantly scan its BT address to detect arrival. I guess triggering scans on “random advertisement” from a phone would help with 2.4Ghz interference. How can I figure out whether interference is a problem ? Won’t these BT devices that sit next to each other interfere with each other ?
1 Like

Your questions in order:

  1. yes, I’d recommend another dedicated pi.

  2. monitor uses hcitool name, so you can do the same. Some devices do not respond immediately, so monitor has in-built error checking, repetition, and other logic. Additionally, BT hardware has a hard time repeatedly scanning, so in-built delays are crucial too. Monitor also has logic to intelligently determine when to scan. Replicating hcitool name in a loop will replicate presence.

  3. Yes. If you’re worried about spoofing and auto-unlock of your door, I suggest an actionable iOS alert triggered when you arrive home that gives you an option to unlock the door. This is what I do.

  4. Yes, an auth is more secure, and not susceptible to spoofing.

  5. It’s probably not necessary to have three adjacent dedicated devices. Yes, they’ll interfere with each other and reduce your effective detection range. If they’re constantly searching for your devices, you’ll be flooding your home and your neighbors with 2.4GHz noise.

And you yourself DON’T use Monitor this way, do you, Andrew? You scan on trigger only on all of your nodes? Is that right? :slight_smile:

1 Like