[monitor] Reliable, Multi-User, Distributed Bluetooth Occupancy/Presence Detection

Ok. Thanks for the report.

Got a chance to look and noticed that your filters are lowercase - filters are case sensitive. I’ll push an update to remove this requirement.

The RSSI reports are just log entries mostly just for information. I plan to cut them down in a future update.

Can you provide me some logs via DM or github?

If you can tell me where they’re located, I sure will.

journalctl -u monitor -r

will output monitors logs. Copying a number of lines before and after the surprise exit will be helpful to me. Thanks!

Trying to get this up and running on a rpi3 b+ I am having issues with devices not being found.
pi@raspberrypi:~/monitor $ sudo hciconfig hci0 up
Can’t get device info: No such device
I followed the directions on the github used for installing on a pi-zero. Not sure much about a pi-zero so I uninstalled the pi-bluetooth entry. Still having issues. any ideas?

Please disregard I was using wrong rpi

I’ve been busy trying to tune a filter sensor to get sensible readings for availability for my beacons but I’m not getting good results. The problem stems from the way that monitor.sh is reporting confidence.

I have three beacons sitting about a metre away from the Pi Zero W. It’s actually outside though sheltered so there should be minimal interference. Even though the beacons are just sitting there, the confidence reported by monitor.sh is varying widely from 0-100% pretty much at random. After applying a lowpass filter with time constant of 10 I get a sensor that fluctuates between about 15% and 50% (average is about 30%), even though the beacons are present and close by the whole time. With the beacons nearby continuously I would hope to see numbers up in the 90s to make the system reliable for tracking the proximity of the beacons.

I have turned up the power and broadcast rate and it doesn’t make any noticeable difference.

When I use the iBeacon app to configure the beacons I can see them constantly updating the app with RSSI readings so they are definitely broadcasting reliably.

Is there anything I can tweak to increase the sensitivity to better report the confidence for the beacons. Else is there any data I can give you to investigate the behaviour a little?

Many thanks again for your work on this.

EDIT: I said above that changing the broadcast rate didn’t change my readings. That was premature! My lowpass sensors have gone from about 30% to about 60%. The broadcast rate is down from 100ms to 10ms. I think that’s the fastest they can go. The downside is that at these rates the battery life is pretty short… Can the sensitivity of monitor.sh be tweaked up a bit?

1 Like

What behavior_preferences of monitor have you moditified?

Thanks for getting back to me. Not much that I recall. I dropped the beacon timeout to 120 from 180. Here’s the file.

#MAX RETRY ATTEMPTS FOR ARRIVAL
PREF_ARRIVAL_SCAN_ATTEMPTS=1

#MAX RETRY ATTEMPTS FOR DEPART
PREF_DEPART_SCAN_ATTEMPTS=2

#SECONDS UNTIL A BEACON IS CONSIDERED EXPIRED
PREF_BEACON_EXPIRATION=120

#MINIMUM TIME BEWTEEN THE SAME TYPE OF SCAN (ARRIVE SCAN, DEPART SCAN)
PREF_MINIMUM_TIME_BETWEEN_SCANS=20

#ARRIVE TRIGGER FILTER(S)
PREF_PASS_FILTER_ADV_FLAGS_ARRIVE=".*"
PREF_PASS_FILTER_MANUFACTURER_ARRIVE=".*"

#ARRIVE TRIGGER NEGATIVE FILTER(S)
PREF_FAIL_FILTER_ADV_FLAGS_ARRIVE="NONE"
PREF_FAIL_FILTER_MANUFACTURER_ARRIVE="NONE"

EDIT: FYI I have just now updated from .069 to .084
Thanks

I am still working to fine tune beacon departure, but reducing the expiration explains the behavior that you’re seeing. The expiration interval is the time after which your beacon is considered completely gone. We check for expiration at an interval equal to one twelfth of the beacon interval. So, for you, that means monitor is checking for expiration pretty close to every 10 seconds.

The reported drops in confidence are based on a simple formula where difference is the time difference from now and when we last received an advertisement from the beacon:

percent_confidence=$(( 100 - difference * 100 / PREF_BEACON_EXPIRATION )) 

Understanding this, we can see why you’re seeing jumps. Let’s say your beacon advertises every 90 seconds, and starts advertising perfectly synchronized with the 10-second clock. Here’s a chart to show what’ll happen:

Advertise Interval Time Confidence
TRUE 0 0 100
1 10 91
2 20 83
3 30 75
4 40 66
5 50 58
6 60 50
7 70 41
8 80 33
TRUE 9 90 100

I’m playing with a few more formulas and techniques, but that’s why you’re seeing what you’re seeing.

TL;DR beacon expiration is not a well-tuned machine quite yet. Getting there!

Thanks very much for the comprehensive explanation.

Re the interval that the beacons advertise, is there any way I can check this directly? It seems like the beacons are transmitting very rapidly given that when I look at them in the iBeacon app that I use to configure them, the RSSI value is varying second to second. Might that be misleading?

Is it maybe possible to inspect what the Pi Zero W is seeing?

Btw I’ll reset the timeout to 180 and see what effect that has.

Thanks again

How does one create a sensor to show monitor is up and running? I have the topic etc just not sure the template value to use.

  - platform: mqtt
    name: 'Monitor Status'
    state_topic: 'monitor/Monitor_pi/status'

Does this look right? trying to start it as a service…

Still no messages are going across for mqtt

i am confused i was able to see the /rssi values, but no longer can see monitor/downstairs/9C:E6:5E:0B:4B:C9

1 Like

I’m doing a major retooling of beacons right now. Continue to update if you’re on beta at least once per hour.

1 Like

so can i use my apple watch as an endpoint beacon?

i believe i am on the latest beta version (updated yesturday), why is it that i can see messages in mqtt.fx but not in home-assistant?

  - platform: mqtt
    state_topic: 'monitor/downstairs/9C:E6:5E:0B:4B:C9'
    value_template: '{{ value_json.confidence }}'
    unit_of_measurement: '%'
    name: 'First Floor'

  - platform: mqtt
    state_topic: 'monitor/downstairs/9C:E6:5E:0B:4B:C9/rssi'
    value_template: '{{ value_json.confidence }}'
    unit_of_measurement: '%'
    name: 'First Floor RSSI'

Does this still work?

  - platform: mqtt
    state_topic: 'monitor/downstairs/9C:E6:5E:0B:4B:C9'
    value_template: '{{ value_json.confidence }}'
    unit_of_measurement: '%'
    name: 'Apple Watch - First Floor'
    availability_topic: "monitor/downstairs/status"
    payload_available: "online"
    payload_not_available: "offline"
    json_attributes:
      - id
      - name 
      - rssi
      - version
      - manufacturer
      - type
      - timestamp
      - retained
      - confidence

I seem to have everything working but it always gives zero for confidence on all my devices. What do I need to check to get this going?

{“id”:“98:00:C6:79:8F:55”,“confidence”:“0”,“name”:“JDPhone”,“manufacturer”:“Apple Inc”,“type”:“KNOWN_MAC”,“retained”:“false”,“timestamp”:“Sun Mar 24 2019 20:54:34 GMT+0000 (GMT)”,“version”:“0.2.124”}

Here is my journalctl -u monitor -r

Mar 24 21:32:20 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:32:20 pm [DEL-RAND]        RAND 54:5A:88:F3:D9:15 expired after 189 seconds
Mar 24 21:32:20 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:32:20 pm [DEL-RAND]        RAND 1D:0F:9B:34:C1:00 expired after 187 seconds
Mar 24 21:31:07 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:31:07 pm [DEL-RAND]        RAND 58:30:CA:B6:A2:4F expired after 181 seconds
Mar 24 21:29:37 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:29:37 pm [CMD-NAME]        00:9A:CD:B8:44:FE nexus  HUAWEI TECHNOLOGIES COLTD
Mar 24 21:29:36 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:29:36 pm [CMD-NAME]        18:74:2E:31:1A:CD dot  Amazon Technologies Inc
Mar 24 21:29:36 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:29:36 pm [CMD-NAME]        98:00:C6:79:8F:55 JDPhone  Apple Inc
Mar 24 21:29:36 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:29:36 pm [CMD-INFO]        **** completed arrival scan ****
_____
Mar 24 21:07:33 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:07:33 pm [CMD-NAME]        00:9A:CD:B8:44:FE nexus  HUAWEI TECHNOLOGIES COLTD
Mar 24 21:07:33 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:07:33 pm [CMD-NAME]        18:74:2E:31:1A:CD dot  Amazon Technologies Inc
Mar 24 21:07:33 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:07:33 pm [CMD-NAME]        98:00:C6:79:8F:55 JDPhone  Apple Inc
Mar 24 21:07:33 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:07:33 pm [CMD-INFO]        **** completed arrival scan ****
Mar 24 21:07:07 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:07:07 pm [CMD-INFO]        **** started arrival scan [x1 max rep] ****
Mar 24 21:03:01 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:03:01 pm [DEL-RAND]        RAND 3E:4D:3D:7E:4D:1F expired after 189 seconds
Mar 24 21:01:06 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:01:06 pm [DEL-RAND]        RAND 5A:F7:21:A9:9B:F6 expired after 186 seconds
Mar 24 21:00:18 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:00:18 pm [CMD-NAME]        00:9A:CD:B8:44:FE nexus  HUAWEI TECHNOLOGIES COLTD
Mar 24 21:00:18 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:00:18 pm [CMD-NAME]        18:74:2E:31:1A:CD dot  Amazon Technologies Inc
Mar 24 21:00:18 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:00:18 pm [CMD-NAME]        98:00:C6:79:8F:55 JDPhone  Apple Inc
Mar 24 21:00:18 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:00:18 pm [CMD-INFO]        **** completed arrival scan ****
Mar 24 20:59:52 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 08:59:52 pm [CMD-INFO]        **** started arrival scan [x1 max rep] ****
Mar 24 20:58:41 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 08:58:41 pm [DEL-RAND]        RAND 58:30:CA:B6:A2:4F expired after 188 seconds
Mar 24 20:56:07 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 08:56:07 pm [CMD-NAME]        98:00:C6:79:8F:55 JDPhone  Apple Inc
Mar 24 20:56:07 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 08:56:07 pm [CMD-INFO]        **** completed arrival scan ****

I am on a Pi Zero W using the on board Bluetooth.

my mqtt works (although it shows 0 confidence) and I have the same as you.
did you copy and paste the MAC address, maybe a zero vs an O (number vs a letter).

- platform: mqtt
  state_topic: 'monitor/garage/00:9A:CD:B8:44:FE'
  value_template: '{{ value_json.confidence }}'
  unit_of_measurement: '%'
  name: 'nexus'

thanks for the response, i see some of that…, oddly i moved my iphone litterally 3-4 away from the pi zero and my rssi drops to almost -9, i would have expect it to be in the 70’s

When you’re on beta, make sure to update as often as possible. Updating is always best.

Monitor is meant to track mobile devices - generally one device per person is what I use, what is the reason you want to track the presence of your echo dot? The pi zero is best suited to track only a few devices. Simplifying will be best. For example, try one device first to be sure you understand how the script works and then add additional devices.

Next, try filtering your devices.

If you still don’t see any response from any of your named devices, please open an issue on github.