[presence] Reliable, Multi-User, Distributed Bluetooth Occupancy/Presence Detection

Yes, this was breaking change from a few days ago.

1 Like

Thanks for the confirmation, I had to check several times until I saw the difference!

I’ll keep it running to fine tune the detection.

1 Like

Thanks for the reply. Ok I ran the command and on the first few attempts I got

Set scan parameters failed: Input/output error

Once it eventually ran I got a list of x512 Mac addresses followed by (unknown) x25 were followed by names and two of those were my Tiles example below.

C4:7C:00:00:00:00 (unknown)
C4:7C:00:00:00:00  Flower care 
D7:2A:00:00:00:00  Home36596
F9:99:00:00:00:00  Tile
D9:CF:00:00:00:00  N0076
C0:EB:00:00:00:00  N06WL
F4:D0:00:00:00:00  Tile

Was this what you wanted from me? If the full list would help further I don’t mind PM’ing it you?

1 Like

I’ve tried with -b :slight_smile: What i didn’t realise is that the beacon wouldn’t appear with its beacon id but it’s mac address (note AA…AA is my phone and BB…BB is another beacon reporting good too.

Next question - They like to depart and arrive quite often, I assume because they broadcast occasionally and not always when there’s a scan. How do I overcome that (when the items not moved)

Starting monitor.sh (v. 0.1.585)...
> ibeacon reporting mode enabled
> preference: delay between scans = 3
> preference: periodic arrive/depart check interval = 15
> preference: periodic arrive interval = 45
> preference: periodic depart interval = 90
> preference: database refresh interval = 35
> preference: max arrival scan attempts = 2
> preference: max depart scan attempts = 4
> preference: random advertisement expiration = 45
> preference: beacon rssi change required for reporting = 20
> preference: bluetooth environmental report frequency = 300
> preference: interval until beacon is considered expired = 145
> preference: preferred HCI device = hci0
0.1.585 05:09:24 PM monitor/scan/arrival/start
0.1.585 05:09:25 PM [CMD-INFO]	**** Started arrival scan. [x2 max rep] ****
0.1.585 05:09:25 PM [CMD-SCAN]	(No. 1) AA:AA:AA:AA:AA:AA arrival?
0.1.585 05:09:29 PM monitor/lounge/AA:AA:AA:AA:AA:AA
{
	retain: false
	version : 0.1.585
	confidence : 100
	name : Phill
	timestamp : Fri Sep 07 2018 17:09:28 GMT+0100 (BST)
	manufacturer : Apple, Inc.
	type : KNOWN_MAC
}
0.1.585 05:09:29 PM [CMD-SCAN]	(No. 1) BB:BB:BB:BB:BB:BB arrival?
0.1.585 05:09:29 PM monitor/scan/arrive
0.1.585 05:09:29 PM [CMD-NAME]	AA:AA:AA:AA:AA:AA Phillip's iPhone  Apple, Inc.
0.1.585 05:09:37 PM [CMD-SCAN]	(No. 2) BB:BB:BB:BB:BB:BB arrival?
0.1.585 05:09:46 PM [CMD-INFO]	**** Completed arrival scan. ****
0.1.585 05:09:48 PM [CMD-RSSI]	C1:00:E2:00:00:63 PUBL RSSI: -87 dBm (Moderate Motion Departing)
0.1.585 05:09:48 PM monitor/scan/arrival/end
0.1.585 05:09:50 PM monitor/lounge/BB:BB:BB:BB:BB:BB
{
	retain: false
	version : 0.1.585
	confidence : 0
	name : Backpack
	timestamp : Fri Sep 07 2018 17:09:50 GMT+0100 (BST)
	manufacturer : IVT corporation
	type : KNOWN_MAC
}
0.1.585 05:09:50 PM monitor/scan/depart
0.1.585 05:09:51 PM [CMD-NAME]	BB:BB:BB:BB:BB:BB Backpack  IVT corporation
...
0.1.585 05:10:01 PM monitor/lounge/C1:00:E2:00:00:63
{
	retain: false
	version : 0.1.585
	confidence : 93
	name : Unknown
	timestamp : Fri Sep 07 2018 17:10:01 GMT+0100 (BST)
	manufacturer : Unknown
	type : GENERIC_BEACON
	rssi : -87
}
...

I seem to be getting about a few mins delay before monitor is updating and telling me I’m home (a lot more than presence did). I also remember when we talked earlier you mentioned monitor shows either 0% or 100% confidence but looking at the confidence I can see fluctuation before it shows as away (decreases from 100% 62% 50% 30% 20% 5% 0% for example)

Is this right? And how can I decrease the detection time? I’m using -r flag.

After some tests, I have verified that if I disconnect the bluetooth, with the default behaviour preferences, the script takes 12 minutes to change the confidence to 0 so I’m away.

If I reconnect the bluetootht, it took around 13 minutes to return to confidence 100 = home.
Is there any way to shorten these times?

Thanks!

Wow - you have quite a few BTLE devices! I’ve made an update - please let me know if your BTLE devices report. I’d use both the -b and -g flags to make sure that you capture all that monitor is written to capture.

You’re using monitor yes? This is not at all expected behavior. Can you run the script from terminal and report the output here? If you run with -R, mac addresses will be hidden from the logs.

For monitor, the preferred use for detecting known devices is without the -r flag. If you are using the -r flag, default preferences may take a minute or more to determine whether you are home.

Are you putting the beacon’s mac address in the known_address file? That is not necessary for beacons, and won’t work for most beacons. For detection of beacons, only the -b (ibeacons) and -g (generic) flags are necessary.

I probably need to update the documentation - it seems like this is a common misunderstanding. My fault!

Ah ok, just going on a previous post where you advised using -r to reproduce the presence feature set.

I’ll remove and test.

I was unclear, my fault. The monitor -r will duplicate the pinging functionality, but to duplicate the behavior, you’ll have to change the default preferences dramatically.

Monitor is completely re-tooled from presence, and the default behavior of monitor is much better than presence, in my opinion. I’ll work on updating the documentation this weekend.

1 Like

Thanks for clarification.

1 Like

Indeed, I’m using monitor.
Below is the requested log (sudo bash monitor.sh -R).
To simplify logs I left only a single device on the list “device1” whose mac ends in :44
Initially phone’s bluetooth was on. Just after the arrival scan has been completed, bluetooth was turned off.
After 7 minutes it has been detected the departure (in previous tests this time was around 12 minutes)

Once the departure has been detected, I turned bluetooth on again. Meanwhile the only device detected has been a Suunto watch (the one whose mac ends in :E0)
Finally 12+ minutes later device1 is detected again.

The log: pastebin

Ok, I have a few recommendations and comments.

First - I’m not sure which Huawei device :44 is, but it is possible that the bluetooth radio isn’t fully disabled when “turned off” via software. The iPhone, for example, only disables connections if bluetooth is “turned off” from the control center. My point is that some devices may need to be physically removed (or placed in an antistatic bag or another faraday cage) for testing. Super irritating to me that some devices are not honest about “turning off” bluetooth radios. Probably not the case here, just food for thought.

Second - As I mention in the readme, I have door sensors that send an MQTT message in the format of [topic_path]/scan/depart after a 10 second delay. This forces a departure scan, and dramatically improves performance for detection of actual, physical, departure.

Third - if you did not experience any interference with presence on your Wi-Fi network, feel free to decrease these values:

PREF_DATABASE_REFRESH_INTERVAL

PREF_RANDOM_DEVICE_EXPIRATION_INTERVAL

PREF_BEACON_EXPIRATION

Fourth - it is also possible that the bluetooth devices in your home rotate their random addresses much more infrequently than average. If this is the case, then triggering a periodic departure scan via mqtt may be your best best.

Fifth - note that the monitor will intentionally ignore very rapid changes in state. If you immediately change your bluetooth state after a change is registered, monitor may, in fact, ignore it. Test with a faraday cage if you can, or by physically leaving your home.

1 Like

There are strange things going on with monitor when run as a service…

If I have my light btle bulbs on when I start the service they are always seen, however if the light is powered off the confidence never drops below 100. Also if I have the bulbs off when I start the service they are never seen when powered back on. This is with the latest version=0.1.586 - I’m not sure if the same apply’s to my plant monitors as I would have to take the battery out to achieve the same effect.

Just thought you might want to know :slight_smile:

1 Like

Have you retained your MQTT messages?

No I don’t retain any messages - retained:false is reported via mqtt box. BTW the light bulbs and the plant monitors are detected as generic beacons.

Edit: it looks like I didn’t wait long enough :stuck_out_tongue: it does eventually decide it’s not there :slight_smile:

1 Like

I am learning from user reports that a depart scan should probably be forced a bit more frequently. I’ll push that update tonight.

Indeed! I think ill find the excludable devices update particularly helpful. I have pulled the update and ran using both -b and -g flags and the Tiles still don’t show. I have tried with the Tiles both in and out of the known_devices file.