[monitor] Reliable, Multi-User, Distributed Bluetooth Occupancy/Presence Detection

monitor

TL;DR: Bluetooth-based passive presence detection of beacons, cell phones, and any other bluetooth device. The system is useful for mqtt-based home automation.

Script here. Read the README here.

Comparing presence.sh and monitor.sh

Thank you to everyone for helping test and provide input in the original thread. I am very pleased to announce a major major update that many of you already know about… monitor.sh as an alternative to, and in many ways a dramatic improvement over, presence.sh

Here’s a quick comparison chart:

Feature presence monitor
Phones/watches/laptops detection :heavy_check_mark: :heavy_check_mark:
Generic Beacon Detection detection :x: :heavy_check_mark:
iBeacons detection :x: :heavy_check_mark:
RSSI (when available) reporting :x: :heavy_check_mark:
Raw data bytes of BTLE advertisement reporting :x: :heavy_check_mark:
Reduced 2.4GHz Wi-Fi Interference :x: :heavy_check_mark:
Interval-triggered name scanning :heavy_check_mark: :heavy_check_mark:
Advertisement-triggered name scanning :x: :heavy_check_mark:
MQTT-triggered name scanning :x: :heavy_check_mark:
MQTT-triggered name scanning ONLY :x: :heavy_check_mark:
Bash script of ridiculous length :heavy_check_mark: :heavy_check_mark:
HA Forum Discussion Link Link

Why do you care?

Here’s the problem I faced - all presence detection options I tried didn’t work reliably for my family.

The various GPS-based options (e.g., OwnTracks, iOS App, HomeKit, etc.) periodically would stop working if an app on our phones crashed. Plus the minimum radius for ‘home’ or ‘away’ was often way larger than our house. I would register as ‘home’ when I was on a dog walk a few blocks away. Not great.

Similarly, the Wi-Fi options (e.g., nmap, ubiquiti, etc.) were neither fast nor reliable because we have iOS devices that disconnect to save battery. Plus, our ubiquiti APs did not report a disconnect for quite some time - in some cases, fifteen minutes or more. Yes, I know we can configure timeouts, but that was not workable either. Still too slow. Also, the phones would disconnect when charging at night. Bad.

Also, Bluetooth LE options (e.g., Happy Bubbles, built-in bluetooth presence sensor, Radius Networks Script, room-assisant, etc.) either required BTLE tags ($$$ + regular battery changes), were limited to detection within bluetooth’s range, or became expensive to implement (five happy bubbles + btle tags = ~$180). Don’t get me wrong, I love the happy bubbles stuff and that project is awesome. But for us, it did not work reliably enough to justify the cost. Plus, the server spammed the mqtt broker and slowed Home Assistant down. Not good.

In sum, these were the problems I had when implementing the current presence solutions:

  • Too slow to respond (wifi)
  • Too unreliable (GPS, wifi, btle)
  • Too expensive (happy bubbles + btle tags)
  • Too big of a radius defining my ‘home’ (GPS, wifi)
  • Too small of a radius (happy bubbles)
  • False positives and false negatives. Switching from home to away when we are at the house is not ok. Similarly, a false positive when we are not home is a huge problem. (GPS)

I wanted a cheaper solution that would meet these specs:

  • Fast to respond. Arrival should be recognized in no more than 5-10 seconds when I’m on the curb or in the garage.
  • Reliable without adding anything to my wife’s phone or keychain or purses. Should work without her caring or knowing that it’s working.
  • Cheap. Less than $50 for the house would be awesome.
  • Tight radius around the house. When I’m at at the curb walking the dog, I should be marked as ‘away’.
  • Respond to BTLE devices too

Summary

A JSON-formatted MQTT message is reported to a specified broker whenever a specified bluetooth device responds to a name query. Optionally, a JSON-formatted MQTT message is reported to the broker whenever a public device or an iBeacon advertises its presence. A configuration file defines ‘known_static_addresses’. These should be devices that do not advertise public addresses, such as cell phones, tablets, and cars. You do not need to add iBeacon mac addresses into this file, since iBeacons broadcast their own UUIDs consistently.


Oversimplified Analogy of the Bluetooth Presence Problem

Imagine you’re blindfolded in a large room with other people. We want to find out who of your friends is there and who of your friends isn’t there:

Some of the people in the room periodically make sounds (e.g., eating a chip, sneeze, cough, etc.), others sit quietly and don’t make a sound unless you specifically ask for them by name, and still others periodically announce their own name out loud at regular intervals whether or not you want them to do that:

Here’s the problem. You can’t just shout “WHO’S HERE” because then everyone would say their name at the same time and you couldn’t tell anything apart. That won’t work. Obviously, you also can’t simply ask “WHO ISN’T HERE.” That’s a pretty foolish suggestion…

So, what about taking attendance like in a classroom? Great idea! Everyone in the room agrees beforehand to respond to an attendance call only when their own name is shouted. That way you can actually hear whether someone responds. Neat!

Since you’re blindfolded, you don’t know how big the room is. That means you have take attendance loudly because we don’t want our friends to not hear their name being called - taking attendance quietly simply won’t do.

Ok, now how can you figure out which of your friends are in the room? If your friends say a name out loud randomly (“Alan!”) it’s easy to know if they’re present or absent - all you have to do is listen for them in the din of the crowd. For most of your friends though, you need to take attendance from a list shouting names one at a time. All other sounds you hear in the room are totally anonymous … you have no idea who made what sound.

So, one way to take attendance is to shout for each friend on a list by name, one at a time, repeatedly. Shout, get a response, wait for a moment, and ask again.

Once a friend stops responding (for some period of time), you presume that he or she has left:

This technique should work just fine, but there’s a problem. You’re constantly shouting into the room, which means that it’s difficult for you to hear quiet responses and it’s difficult for other people to carry on conversations. What else can we do?

A smarter approach is to wait for an anonymous sound, then start asking whether your friend is there:

This technique is a very simplified description of how montior works for devices like cell phones (friends on a list) and beacons (announce a name out loud). This also gives an idea of how monitor uses anonymous sounds to reduce the number of times that it has to send inquiries into the Bluetooth environment.


Background on BTLE

The BTLE 4.0 spec was designed to make connecting bluetooth devices simpler for the user. No more pin codes, no more code verifications, no more “discovery mode” - for the most part.

It was also designed to be much more private than previous bluetooth specs. But it’s hard to maintain privacy when you want to be able to connect to an unknown device without substantive user intervention, so a compromise was made.

The following is oversimplified and not technically accurate in most cases, but should give the reader a gist of how monitor determines presence.

Connectable Devices

BTLE devices that can exchange information with other devices advertise their availability to connect to other devices, but the “mac” address they use to refer to themselves will be random, and will periodically change. This prevents bad actors from tracking your phone via passive bluetooth monitoring. A bad actor could track the random mac that your phone broadcasts, but that mac will change every few moments to a completely different address, making the tracking data useless.

Also part of the BTLE spec (and pre-LE spec) is a feature called a name request. A device can request a human-readable name of a device without connecting to that device but only if the requesting device affirmatively knows the hardware mac address of the target device. To see an example of this, try to connect to a new bluetooth device using your Android Phone or iPhone - you’ll see a human-readable name of devices around you, some of which are not under your control. All of these devices have responded to a name request sent by your phone (or, alternatively, they have included their name in an advertisement received by your phone).

Beacon Devices

The BTLE spec was used by Apple, Google, and others to create additional standards (e.g., iBeacon, Eddystone, and so on). These “beacon” standards didn’t care that the MAC addresses that were periodically broadcast were random. Instead, these devices would encode additional information into the broadcast data. Importantly, most devices that conform to these protocols will consistenly broadcast a UUID that conforms to the 8-4-4-4-12 format defined by IETC RFC4122.

Using Advertisements to Trigger "Name" Scans

So, we now know a little bit more about ‘name’ requests which need a previously-known hardware mac address and ‘random advertisements’ which include ostensibly useless mac addresses that change every few moments.

To see these things in action, you can use hcitool and hcidump. For example, to see your phone respond to a ‘name’ request, you can type this command:

hcitool name 00:00:00:00:00:00

Of course, replace your hardware bluetooth mac address with the 00:00… above.

To see your phone’s random advertisements (along with other random advertisements), you will need to launch a bluetooth LE scan and, separately, monitor the output from your bluetooth radio using hcidump

sudo hcitool lescan &
sudo hcidump --raw

This command will output the raw BTLE data that is received by your bluetooth hardware while the hcidump process is operating. Included in this data are ADV_RAND responses.

Knowing this, we can explain to the differences between presence and monitor.

The presence script, with default settings, requests a name from your owner devices at regular intervals. You can adjust those intervals, but your pi will be regularly ‘pinging’ for each owner device. The longer your intervals, the slower the system will be to respond. The shorter the intervals, the more quickly the system will respond, but the more 2.4GHz bandwidth is used. In other words, the more often presence scans, the more likely it is that presence will interfere with 2.4GHz Wi-Fi. The more devices running presence, the more interference. This is a huge bummer for home with 2.4GHz Wi-Fi and other Bluetooth devices (especially older or cheaper devices). Below is a simplified flowchart showing the operation of presence:

On the other hand, the monitor script, with default settings, will only request a name from your owner device after a new random advertisement is detected. If there are no devices that randomly advertise, monitor will never scan for new devices, clearing 2.4GHz spectrum for Wi-Fi use. The monitor script will also detect and report the UUID of nearby iBeacons. Below is a simplified flowchart showing the operation of monitor, showing the flows for both detection of arrival and detection of departure:

Example with Home Assistant

I have three raspberry pi zero w throughout the house. We spend most of our time on the first floor, so our main ‘sensor’ is the first floor. Our other ‘sensors’ on the second and third floor are set up to trigger only, with option -t.

The first floor constantly monitors for advertisements from generic bluetooth devices and ibeacons. The first floor also monitors for random advertisements from other bluetooth devices, which include our phones. When a “new” random device is seen (i.e., an advertisement is received), the first floor pi then scans for the fixed address of our cell phones. These addresses are are stored in the known_static_addresses file. If one of those devices is seen, an mqtt message is sent to Home Assistant reporting that the scanned phone is “home” with a confidence of 100%. In addition, an mqtt message is sent to the second and third floor to trigger a scan on those floors as well.

When we leave the house, we use either the front door or the garage door to trigger an mqtt trigger of [topic_path]/scan/depart after a ten second delay to trigger a departure scan of our devices. The ten second delay gives us a chance to get out of bluetooth range before a “departure” scan is triggered. Different houses/apartments will probably need different delays.

Home Assistant receives mqtt messages and stores the values as input to a number of mqtt sensors. Output from these sensors can be averaged to give an accurate numerical occupancy confidence.

For example (note that 00:00:00:00:00:00 is an example address - this should be your phone’s address):

- platform: mqtt
  state_topic: 'monitor/first floor/00:00:00:00:00:00'
  value_template: '{{ value_json.confidence }}'
  unit_of_measurement: '%'
  name: 'First Floor'

- platform: mqtt
  state_topic: 'monitor/second floor/00:00:00:00:00:00'
  value_template: '{{ value_json.confidence }}'
  unit_of_measurement: '%'
  name: 'Second Floor'

- platform: mqtt
  state_topic: 'monitor/third floor/00:00:00:00:00:00'
  value_template: '{{ value_json.confidence }}'
  unit_of_measurement: '%'
  name: 'Third Floor'

These sensors can be combined/averaged using a min_max. I use a max sensor, but mean may also be appropriate for certain implementations:

- platform: min_max
  name: "Home Occupancy Confidence of 00:00:00:00:00:00"
  type: max
  round_digits: 0
  entity_ids:
    - sensor.third_floor
    - sensor.second_floor
    - sensor.first_floor

Then I use the entity sensor.home_occupancy_confidence in automations to control the state of an input_boolean that represents a very high confidence of a user being home or not.

As an example:

- alias: Occupancy 
  hide_entity: true
  trigger:
    - platform: numeric_state
      entity_id: sensor.home_occupancy_confidence
      above: 10
  action:
    - service: input_boolean.turn_on
      data:
        entity_id: input_boolean.occupancy

Conclusion

Thank you all for your patience and support over the course of this project. Please report all issues on github, but feel free to post here as well.

Extras

Please also see the awesome appdaemon script by @Odianosen25 here and another by @PianSom here and yet another, a hassio add-on by @Limych.

138 Likes

Special thanks to all the wonderful folks who have helped in one way or another with this project by reporting issues, helping other users, alpha testing, and so on. These great folks include - and I’m sorry if I’ve forgotten someone; I’m very fortunate that so many people have helped me, I have lost track of the comprehensive list of helpers - in no particular order: @PianSom, @eboon, @daneboom, @ih8gates, @gumbo, @piotr, @mcfrojd, @jnvd3b, @LarsAC, @nicolasbisi

5 Likes

Apparently, I can only mention 10 users in a comment. Here are other great folks I need to thank: @robmarkcole, @vegard, @foraster, @apt, @keithh666, @caswell1000 @Odianosen25

There are many, many more. Thank you all!

5 Likes

Just put all my sensors back on Monitor. The only concern that I am really trying to test is the departure time. I need to know if I can lower 2 values. (some of these are picky but i figure id ask)

  1. my door is on an instant autolock so when I go to take out the trash I always take my phone with me. Its about a minute to 2 min walk to the trash. If I trigger a depart scan when my door opens how long should it take the nodes to see that im actually away and put my confidence to 0.

  2. is there a setting to change how many levels of confidence there are? id like 2-3 max. IE, scan once, goes down to 70, scan twice goes down to 40, scan 3 times down to 0. Is there a way I can set it to something like this.

  3. last thing. instead of a triggering a depart or arrival scan can I just trigger a name scan for all my devices (only 2) instead? reasoning… Lets say Im home and I go to take the trash out and my door does a trigger to scan for devices currently home. Eventually it will start bringing my confidence down to zero. But lets say my GF comes in a minute after me while the depart is still running. I assume this would mean she wouldnt be picked up because the devices are still doing a Depart scan which only check for devices currently not home. So if I could trigger just a SCAN instead of depart or arrive for lets say 5 minutes then go back to original settings of waiting for a new advertisement.

Im hoping you understand what I meant by the third one.

1 Like

Hi @benjimatt

  1. I use a 10 second delay before triggering a depart scan with my doors, which results in about a 20 - 30 second period before I’m “away.” If you need a 5 minute round trip to notice that you’re still home, I’d suggest a delay of 300 seconds. Note that in that time, other triggers might scan to determine whether you’re still home, such as the expiration of another recently-seen beacon. In my house, when I take the trash out, I will be noticed as away. The tight radius around my house is, somewhat, the motivating factor in this project.

  2. The confidence levels are based on how many repetitions of “no response” from your device are required before you are defined as “not home.” Confidence drops by half for each repetition. With default settings, there are four repetitions, so you’ll see six levels: 100, 50, 25, 12, 6, 0. I may add a setting to adjust the reporting behavior, depending on interest.

  3. Your assumption is correct - only one type of scan is performed at a time. The script is structured in this way to minimize scanning as much as possible. So, when you leave and a depart scan is triggered, it’s only scanning for your device (since she is not currently home). If she arrives while the depart scanning is ongoing, the arrive scan request will be rejected, but when you arrive back home, after having been marked as away, both of your devices will be scanned and she’ll be marked as home. I think your hypo is an edge case, but it may happen for a number of people. I’ll think on a solution to this potential issue.

Oh my. The depart mqtt message just clicked as to how useful it is.

2 Likes

Amazing work with this.

Have you considered packaging this up as a Docker image? If not, would you be open to collaboration to make that happen? I’d love to just have a docker-compose that I can toss on all of the Pis in my house as an easier way to manage and update them all.

6 Likes

One of the original motivations for using bash was to reduce dependencies. Certainly now, in retrospect, I should have written in a higher level language with more dependencies and packages, but here we are…

Anyway, containerizing/packaging the script isn’t something I have seriously considered because it would add installation requirements for the end user. Certainly open to others who’d like to help with that though.

1 Like

Updated to most recent version yesterday, continued to work excellent.

This is really the most reliable and fast method of presence detection I have tried. Thanks for the idea and making it available, @andrewjfreyer!

As for a Docker container: I can provide help. What might be useful is a -d flag (or whatever) to control the directory location for config files.

Lars

1 Like

If i want monitor to start automaticly at reboot with the -g & -b flags,
do i change this line in “/etc/systemd/system/monitor.service”
from

ExecStart=/bin/bash /home/pi/monitor/monitor.sh  &

to

ExecStart=/bin/bash /home/pi/monitor/monitor.sh -g -b  &

Or do i set the startflags someware else?

2 Likes

I asked the same question and @andrewjfreyer has made it easy. You simply add the -u flag and all other flags will be set as default for the service.

4 Likes

Out of interest how is everybody else running their monitor setup? I have 2 Tiles in the known beacon file and two iPhones in the known devices file.

A Pi Zero on each floor. First floor runs with -r -g -b flags and second with -b -g -t flags. I’m unsure if this is the best method? I also plan on implementing triggered scans from Home Assistant using my door status and conditions e.g. If marked home run depart scan after delay and if away run arrive scan. Has anybody else implemented similar?

Hopefully by Monday monitor will be my only method of presence detection! :slight_smile: Cheers @andrewjfreyer

2 Likes

I would also like to use it in docker. From what I read it is quite difficult to pass the hci device to the docker container.

I am currently using Monitor with the default settings. I trigger a depart scan every time my door opens. I only have 2 devices being monitored now and this is 2 cell phones, one IOS one Android.

In regards to HASS I am detecting presence by Ping and by the Monitor MQTT messages. Essentially whichever comes first will detect if im home but I do not use Ping to detect if im away because our phones do not respond to a ping if in standby for a while.

my Pi’s are complete overkill. Currently using 3 Pi zeros for my two bedroom apartment lol. I am honestly thinking about using 4 Pi zeros and having 2 of them have only a single devices. I know this would be a waste but it would speed up detection in theory. All of my Pi’s are connected through ethernet.

It would be pretty nice if we could combine Pi’s so that they can detect multiple devices at once. I wonder how hard this would be.

@J_IO_B, thanks for the kind words. I think your survey idea is great! Here we go:

For my main monitor node, I use:

  • There are too many options and I don’t understand which ones I should be using. Help!
  • -m send heartbeat signal
  • -e report bluetooth environment periodically via mqtt
  • -r repeatedly scan for arrival & departure of known devices
  • -f format MQTT topics with only letters and numbers
  • -b report ibeacon advertisements
  • -a report all scan results, not just changes
  • -x retain mqtt status messages
  • -g report generic bluetooth advertisements
  • -t scan only on mqtt trigger messages

0 voters

For my other monitor nodes, I use:

  • I don’t have more than one node.
  • -m send heartbeat signal
  • -e report bluetooth environment periodically via mqtt
  • -r repeatedly scan for arrival & departure of known devices
  • -f format MQTT topics with only letters and numbers
  • -b report ibeacon advertisements
  • -a report all scan results, not just changes
  • -x retain mqtt status messages
  • -g report generic bluetooth advertisements
  • -t scan only on mqtt trigger messages

0 voters

2 Likes

My detection time is pretty quick, usually less than 10 secs when I get in and the system sends me a welcome home text message over Telegram. When leaving same, though I have a 30 secs delay.

Many times when ppl use the door to activate a scan, the system might be busy and then reject it. I think this is the issue, because I have noticed that if a scan is sent when the system is scanning, it’s rejected and it ignores it. This is made worse when the scan frequency is more; kind of.

So if in code one can be checking it’s its busy, and then send the data when not busy, there is a huge speed in detection. @andrewjfreyer has made it easier to know when it’s busy be reporting start and end scan messages.

This method also helps me, as even if I have my door opened for a long time, like if talking to someone, once closed or opened, as it’s constantly checking if busy, it ensures no message is rejected

Hope this helps in making it faster for you, than spending more cash. Regards

2 Likes

I just switched from presence to monitor. I am aiming for a quite simple setup to start:

One raspi zero set up with all default options checking for presence of an iphone and an android phone (using the same mac addresses I had used with presence.sh).

I am running the service without modification (created automatically through sudo bash monitor.sh. Am I missing any additional flags? If so, where would I have to add them?

I also noticed that git clone created most files as user root. I changed them all to pi:pi and made monitor.sh executable for everyone. Was that a bad idea?

Also, I noticed that the instructions in this forum and on Github show the example for HA to listen for the topic location: state_topic: 'location/first floor/00:00:00:00:00:00'. This probably should be updated to something like <topic root>/<publisher identity>/MAC address

Thanks for your help to get me off the ground (btw. a docker version of this great tool would be awesome - and should actually simplify things for novice users).

1 Like

Hello @davosian,

Before running the service, did you run sudo bash monitor.sh first? After that made the required modifications to the newly created file, I think behavior_preferences.

Regards

1 Like

I actually dont have any real issues in regards to incoming detection. I have an august lock and have turned off the native auto unlock feature because if I dont go far enough from the home it doesnt autounlock.
Monitor has completely replaced this. My monitor detection unlocks my door and if there is an issue it may take an extra 3 seconds when i get to the door.

My outgoing detection seems to work just fine. The system detects if im gone within a minute or 2.

The issues is that BY DESIGN hcitool can only do a single name scan at a time. I dont think this is something that can actually be fixed. This is why I think I may just go with 4 Pi’s with 2 of them scanning for the same device. You realize Pi zeros are only 5 dollars… I can skip a couple trips to mcdonalds for bluetooth perfection lol

1 Like

Hello @Odianosen25,
thanks for the hint about behavior_preferences. I did run the intial sudo bash monitor.sh and have the default config for my setup. Are there any optimizations to be made for my setup (just two cellphones detected by one pi without door sensor).

Best,
Dennis

1 Like