@okercho From the log, we can see that this output is from the presence script. The monitor script does not report this field anymore. Perhaps the presence process is still active on one or more of these devices?
I still have presence in two of the Raspberrys, is not running for sure in this one… but it had it before.
Can’t both of them be running at the same time? (in different Raspberrys).
Cheers
Yes, presence and monitor can coexist in the same network, but not on the same pi. That said, the ‘error’ you see above has nothing to do with monitor, and could only come from presence.
Is that pi physically close to the Samsung Galaxy such that you would expect that pi to register that device?
Monitor seemed to run fine from the command line. Now I run it as a service and see the following message in the log:
Jul 16 21:11:48 srv01 bash[13182]: 0.1.448 09:11:48 PM [REJECT] Arrive scan request denied. Hardware busy.
Any idea where that comes from ?
Lars
1 Like
That pi is in the entry of the house, so it should detect me when I arrive home… I’ll test it and come back.
Anyway, I’ve updated monitor in the others raspberrys (I didn’t update them because I was suffering more interference in the door one) and now I’m only using monitor.
Cheers!
That error appears when you try to scan to frequently. Only one scan can occur at a time.
I have set
PREF_INTERSCAN_DELAY=15
but still see the error. Which setting should I change ?
Lars
It is an ignorable error. Are you triggering via MQTT?
No, I just let the script run without any parameters.
Lars
I’ve seen this myself often just running the “vanilla” script with two devices and no triggering. Agreed it doesn’t seem to matter, but I did wonder…
Feel free to ignore the error. Likely means that you have a number of devices around you that are all broadcasting random advertisements. This error reflects intentional rate limiting.
Thats certainly true! So, which device is “busy” in that instance?
I added a bit of description to the -h file for monitor last night, but here’s a more detailed description that I hope is helpful to highlight the problem we’re trying to solve, the challenges with presence for some users, and how monitor and presence differ.
The BTLE 4.0 spec was designed to make connecting bluetooth devices simpler for the user. No more pin codes, no more code verifications, no more “discovery mode” - for the most part. It was also designed to be much more private than previous bluetooth specs. But it’s hard to maintain privacy when you want to be able to connect to an unknown device without substantive user intervention (think about pairing new AirPods to an iPhone), so a compromise was made. BTLE devices advertise their presence (and, thus, availability to connect to other devices), but the “mac” address they use to refer to themselves will be random, and will periodically change. This prevents bad actors from tracking your phone via passive bluetooth monitoring. A bad actor could track the random mac that your phone broadcasts, but that mac will change every few moments to a completely different address, making the tracking data useless.
Also part of the BTLE spec is a feature called “name” request. A device can request a human-readable name of a device without connecting to that device but only if the requesting device affirmatively knows the hardware mac address of the target device. Again, think about connecting your iPhone to a new pair of AirPods. The name you see on the popup isn’t “00:11:22:33:44” or similar, it’s “Andrew’s AirPods” or “New AirPods.”
So, we now know a little bit more about ‘name’ requests which need a previously-known hardware mac address and ‘random advertisements’ which include ostensibly useless mac addresses that change every few moments.
To see these things in action, you can use hcitool and hcidump. For example, to see your iPhone respond to a ‘name’ request, you can type this command:
hcitool name 00:00:00:00:00:00
Of course, replace your hardware bluetooth mac address with the 00:00… above.
To see your iPhone’s random advertisements (along with other random advertisements), you will need to launch a bluetooth LE scan and, separately, monitor the output from your bluetooth radio using hcidump
sudo hcitool lescan & ; sudo hcidump --duplicates
This command will output the BTLE data that is received by your bluetooth hardware. Included in this data are ADV_RAND responses.
Knowing this, we can jump to the differences between presence and monitor.
The presence script, with default settings, requests a name from your owner devices at regular intervals. You can adjust those intervals, but your pi will be regularly ‘pinging’ for each owner device. The longer your intervals, the slower the system will be to respond. The shorter the intervals, the more quickly the system will respond, but the more 2.4GHz bandwidth is used. Now, as @eboon originally pointed out, the more often presence scans, the more likely it is that presence will interfere with 2.4GHz Wi-Fi. If you add two or more pi’s in your house, its clear that 2.4GHz spectrum being used very frequently to scan for presence of bluetooth devices. For many users, this meant that 2.4GHz Wi-Fi became unusable or unresponsive. Many users never noticed this (including myself) because we use 5GHz Wi-Fi or have a router/access point able to handle the extra bandwidth used by the presence script.
On the other hand, the monitor script, with default settings, will only request a name from your owner device after a new random advertisement is detected. If there are no devices that randomly advertise, monitor will never scan for new devices, clearing 2.4GHz spectrum. For example, if you live in a house with no other bluetooth devices and no nearby neighbors, when you come home, a random advertisement (that probably came from your phone) will be detected almost immediately by monitor. Thereafter, a name request is submitted to the phone.
Alternatively, if you - like me - live with neighbors nearby, you may see at least 5 - 6 devices that broadcast random advertisements just like your phone. In order to prevent a huge number of repeated scans, I added a rate-limiter to the monitor script so that only one scan is triggered at once. After all, if your phone arrives while the script is already scanning, your phone is highly likely to be detected anyway. The neat thing about this setup is that after the script receives advertisements from all or most of the devices around you, those devices will continue broadcasting with the same random mac address for at least a few minutes. Since these are not new devices, the monitor script ignores these until they expire (i.e., have not been seen for at least XX seconds).
What’s the result of this? The result is that the monitor script needs a bit of time to collect information about all the devices that are in the local environment before it becomes truly efficient when scanning. If you run the script manually, you will/should see many scans rejected in the first few minutes. Thereafter, rejected scans should occur much less frequently.
I hope this makes sense. The structure of the new script is entirely different from presence, but the spirit is the same.
9 Likes
@andrewjfreyer Thanks for the detail! I’m now curious to know how it detects departure?
Second question: given that my home as only two ways to get in or out, and I can detect both of them opening/closing using HASS, I could in theory use the -t option, and just trigger a scan when either door opens or closes or some time period after to avoid catching me at the door on my way out?
MQTT triggers are always enabled. I’d avoid the -t option for now.
Departure scans are triggered based on expiration of random advertisements. The expiration period is controlled in the preferences.
I’m at work now and (remotely) updated my Home Assistant to the newest version.
After restarting HA, I see:
But I’m not home at the moment and before the HA restart, these where all 0%.
Do I have to change something in my HA config? (retain settings maybe?)
- platform: mqtt
state_topic: 'location/owner/bg-voor/84:98:XX:CB:XX:XX'
value_template: '{{ value_json.confidence }}'
unit_of_measurement: '%'
name: 'Lex beneden voor'Hi Andrew,
Thanks a lot for the detailed explanation, I think now makes more sense.
Yesterday I moved all the 3 raspberrys I’m currently using to monitor, using screen to be able to see the logs, and this is what happened:
- Presence detection worked fine at the beginning, it detected me at home, my wife at home, and it detected when my wife left out for a few minutes (it took like 4 minutes to detect her away). No alerts of the heartbeat came for a while.
- During the night, more than 100 alerts of the hearbeat arrived, now from all the raspberrys.
- This morning, I left home for work, and no detection occurred after around 2h later. I checked the screen, and for 2 raspberrys, the last message was like 2hours ago
- Right now all the raspberrys shows “movement” in the logs, but heartbeat is still pinging me.
My impression:
- The heartbeat alert occurs when nothing has been published to the MQTT queue in the last 10 minutes, I think that no message is being published if the status is not changed… if this is the case, the heartbeat flow from the first comment is not valid anymore if you’re using monitor.
- Regarding why it took almost 2h to detect my depart, I’m not quite sure, I see in the logs that a lot of devices are coming and going, so it should trigger scans if I understood it correctly, and I don’t know why the script was “stuck” for 2 hours, and “came back to life” when I opened the screen in all of them. I also see the hardware busy error sometimes in the logs.
I’m using monitor script with the default values, and executing it without arguments.
Cheers!
Value on restart has a bit to do with your configuration, so it’ll be hard to diagnose from here without more information. Do you have the trigger flags on? If there are no state changes, the pis running presence or monitor won’t report any changes when HA reboots.
Yes, this is correct. The heartbeat is no longer required or desirable.
I managed to get it working for my iPhone X - woo! 
Was scratching my head a lot, and then just did a git pull in the folder - and voila! Something must have changed I guess!
Anyone got any tips on getting Apple Watches to be found? If I search for the MAC Address of my Apple Watch, nothing is found:
hcitool name 98:00:C6:09:ED:BB