[monitor] Reliable, Multi-User, Distributed Bluetooth Occupancy/Presence Detection

johnflorin · March 18, 2019, 7:09pm

It worked fine for a while, then exited again today, with “monitor.sh: line 897: Apple, Inc.: syntax error: invalid arithmetic operator (error token is “.”)”

andrewjfreyer · March 18, 2019, 7:20pm

I’ll check when I get a chance. Can you open an issue on github?

andrewjfreyer · March 18, 2019, 7:21pm

Ok. Thanks for the report.

andrewjfreyer · March 18, 2019, 10:56pm

Got a chance to look and noticed that your filters are lowercase - filters are case sensitive. I’ll push an update to remove this requirement.

The RSSI reports are just log entries mostly just for information. I plan to cut them down in a future update.

andrewjfreyer · March 18, 2019, 11:24pm

Can you provide me some logs via DM or github?

johnflorin · March 19, 2019, 4:44am

If you can tell me where they’re located, I sure will.

andrewjfreyer · March 19, 2019, 3:33pm

journalctl -u monitor -r

will output monitors logs. Copying a number of lines before and after the surprise exit will be helpful to me. Thanks!

Charles_Brown · March 19, 2019, 9:10pm

Trying to get this up and running on a rpi3 b+ I am having issues with devices not being found.
pi@raspberrypi:~/monitor $ sudo hciconfig hci0 up
Can’t get device info: No such device
I followed the directions on the github used for installing on a pi-zero. Not sure much about a pi-zero so I uninstalled the pi-bluetooth entry. Still having issues. any ideas?

Please disregard I was using wrong rpi

Michaelrch · March 20, 2019, 7:23am

I’ve been busy trying to tune a filter sensor to get sensible readings for availability for my beacons but I’m not getting good results. The problem stems from the way that monitor.sh is reporting confidence.

I have three beacons sitting about a metre away from the Pi Zero W. It’s actually outside though sheltered so there should be minimal interference. Even though the beacons are just sitting there, the confidence reported by monitor.sh is varying widely from 0-100% pretty much at random. After applying a lowpass filter with time constant of 10 I get a sensor that fluctuates between about 15% and 50% (average is about 30%), even though the beacons are present and close by the whole time. With the beacons nearby continuously I would hope to see numbers up in the 90s to make the system reliable for tracking the proximity of the beacons.

I have turned up the power and broadcast rate and it doesn’t make any noticeable difference.

When I use the iBeacon app to configure the beacons I can see them constantly updating the app with RSSI readings so they are definitely broadcasting reliably.

Is there anything I can tweak to increase the sensitivity to better report the confidence for the beacons. Else is there any data I can give you to investigate the behaviour a little?

Many thanks again for your work on this.

EDIT: I said above that changing the broadcast rate didn’t change my readings. That was premature! My lowpass sensors have gone from about 30% to about 60%. The broadcast rate is down from 100ms to 10ms. I think that’s the fastest they can go. The downside is that at these rates the battery life is pretty short… Can the sensitivity of monitor.sh be tweaked up a bit?

andrewjfreyer · March 20, 2019, 11:52am

What behavior_preferences of monitor have you moditified?

Michaelrch · March 20, 2019, 12:06pm

Thanks for getting back to me. Not much that I recall. I dropped the beacon timeout to 120 from 180. Here’s the file.

#MAX RETRY ATTEMPTS FOR ARRIVAL
PREF_ARRIVAL_SCAN_ATTEMPTS=1

#MAX RETRY ATTEMPTS FOR DEPART
PREF_DEPART_SCAN_ATTEMPTS=2

#SECONDS UNTIL A BEACON IS CONSIDERED EXPIRED
PREF_BEACON_EXPIRATION=120

#MINIMUM TIME BEWTEEN THE SAME TYPE OF SCAN (ARRIVE SCAN, DEPART SCAN)
PREF_MINIMUM_TIME_BETWEEN_SCANS=20

#ARRIVE TRIGGER FILTER(S)
PREF_PASS_FILTER_ADV_FLAGS_ARRIVE=".*"
PREF_PASS_FILTER_MANUFACTURER_ARRIVE=".*"

#ARRIVE TRIGGER NEGATIVE FILTER(S)
PREF_FAIL_FILTER_ADV_FLAGS_ARRIVE="NONE"
PREF_FAIL_FILTER_MANUFACTURER_ARRIVE="NONE"

EDIT: FYI I have just now updated from .069 to .084
Thanks

andrewjfreyer · March 20, 2019, 5:09pm

I am still working to fine tune beacon departure, but reducing the expiration explains the behavior that you’re seeing. The expiration interval is the time after which your beacon is considered completely gone. We check for expiration at an interval equal to one twelfth of the beacon interval. So, for you, that means monitor is checking for expiration pretty close to every 10 seconds.

The reported drops in confidence are based on a simple formula where difference is the time difference from now and when we last received an advertisement from the beacon:

percent_confidence=$(( 100 - difference * 100 / PREF_BEACON_EXPIRATION ))

Understanding this, we can see why you’re seeing jumps. Let’s say your beacon advertises every 90 seconds, and starts advertising perfectly synchronized with the 10-second clock. Here’s a chart to show what’ll happen:

Advertise	Interval	Time	Confidence
TRUE	0	0	100
	1	10	91
	2	20	83
	3	30	75
	4	40	66
	5	50	58
	6	60	50
	7	70	41
	8	80	33
TRUE	9	90	100

I’m playing with a few more formulas and techniques, but that’s why you’re seeing what you’re seeing.

TL;DR beacon expiration is not a well-tuned machine quite yet. Getting there!

Michaelrch · March 20, 2019, 5:28pm

Thanks very much for the comprehensive explanation.

Re the interval that the beacons advertise, is there any way I can check this directly? It seems like the beacons are transmitting very rapidly given that when I look at them in the iBeacon app that I use to configure them, the RSSI value is varying second to second. Might that be misleading?

Is it maybe possible to inspect what the Pi Zero W is seeing?

Btw I’ll reset the timeout to 180 and see what effect that has.

Thanks again

Charles_Brown · March 22, 2019, 5:21pm

How does one create a sensor to show monitor is up and running? I have the topic etc just not sure the template value to use.

  - platform: mqtt
    name: 'Monitor Status'
    state_topic: 'monitor/Monitor_pi/status'

david1 · March 23, 2019, 5:38pm

Does this look right? trying to start it as a service…

Still no messages are going across for mqtt

i am confused i was able to see the /rssi values, but no longer can see monitor/downstairs/9C:E6:5E:0B:4B:C9

andrewjfreyer · March 23, 2019, 11:18pm

I’m doing a major retooling of beacons right now. Continue to update if you’re on beta at least once per hour.

david1 · March 23, 2019, 11:42pm

so can i use my apple watch as an endpoint beacon?

david1 · March 24, 2019, 3:58pm

i believe i am on the latest beta version (updated yesturday), why is it that i can see messages in mqtt.fx but not in home-assistant?

  - platform: mqtt
    state_topic: 'monitor/downstairs/9C:E6:5E:0B:4B:C9'
    value_template: '{{ value_json.confidence }}'
    unit_of_measurement: '%'
    name: 'First Floor'

  - platform: mqtt
    state_topic: 'monitor/downstairs/9C:E6:5E:0B:4B:C9/rssi'
    value_template: '{{ value_json.confidence }}'
    unit_of_measurement: '%'
    name: 'First Floor RSSI'

Does this still work?

  - platform: mqtt
    state_topic: 'monitor/downstairs/9C:E6:5E:0B:4B:C9'
    value_template: '{{ value_json.confidence }}'
    unit_of_measurement: '%'
    name: 'Apple Watch - First Floor'
    availability_topic: "monitor/downstairs/status"
    payload_available: "online"
    payload_not_available: "offline"
    json_attributes:
      - id
      - name 
      - rssi
      - version
      - manufacturer
      - type
      - timestamp
      - retained
      - confidence

jasondreher · March 24, 2019, 9:35pm

I seem to have everything working but it always gives zero for confidence on all my devices. What do I need to check to get this going?

{“id”:“98:00:C6:79:8F:55”,“confidence”:“0”,“name”:“JDPhone”,“manufacturer”:“Apple Inc”,“type”:“KNOWN_MAC”,“retained”:“false”,“timestamp”:“Sun Mar 24 2019 20:54:34 GMT+0000 (GMT)”,“version”:“0.2.124”}

Here is my journalctl -u monitor -r

Mar 24 21:32:20 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:32:20 pm [DEL-RAND]        RAND 54:5A:88:F3:D9:15 expired after 189 seconds
Mar 24 21:32:20 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:32:20 pm [DEL-RAND]        RAND 1D:0F:9B:34:C1:00 expired after 187 seconds
Mar 24 21:31:07 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:31:07 pm [DEL-RAND]        RAND 58:30:CA:B6:A2:4F expired after 181 seconds
Mar 24 21:29:37 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:29:37 pm [CMD-NAME]        00:9A:CD:B8:44:FE nexus  HUAWEI TECHNOLOGIES COLTD
Mar 24 21:29:36 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:29:36 pm [CMD-NAME]        18:74:2E:31:1A:CD dot  Amazon Technologies Inc
Mar 24 21:29:36 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:29:36 pm [CMD-NAME]        98:00:C6:79:8F:55 JDPhone  Apple Inc
Mar 24 21:29:36 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:29:36 pm [CMD-INFO]        **** completed arrival scan ****
_____
Mar 24 21:07:33 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:07:33 pm [CMD-NAME]        00:9A:CD:B8:44:FE nexus  HUAWEI TECHNOLOGIES COLTD
Mar 24 21:07:33 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:07:33 pm [CMD-NAME]        18:74:2E:31:1A:CD dot  Amazon Technologies Inc
Mar 24 21:07:33 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:07:33 pm [CMD-NAME]        98:00:C6:79:8F:55 JDPhone  Apple Inc
Mar 24 21:07:33 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:07:33 pm [CMD-INFO]        **** completed arrival scan ****
Mar 24 21:07:07 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:07:07 pm [CMD-INFO]        **** started arrival scan [x1 max rep] ****
Mar 24 21:03:01 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:03:01 pm [DEL-RAND]        RAND 3E:4D:3D:7E:4D:1F expired after 189 seconds
Mar 24 21:01:06 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:01:06 pm [DEL-RAND]        RAND 5A:F7:21:A9:9B:F6 expired after 186 seconds
Mar 24 21:00:18 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:00:18 pm [CMD-NAME]        00:9A:CD:B8:44:FE nexus  HUAWEI TECHNOLOGIES COLTD
Mar 24 21:00:18 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:00:18 pm [CMD-NAME]        18:74:2E:31:1A:CD dot  Amazon Technologies Inc
Mar 24 21:00:18 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:00:18 pm [CMD-NAME]        98:00:C6:79:8F:55 JDPhone  Apple Inc
Mar 24 21:00:18 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 09:00:18 pm [CMD-INFO]        **** completed arrival scan ****
Mar 24 20:59:52 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 08:59:52 pm [CMD-INFO]        **** started arrival scan [x1 max rep] ****
Mar 24 20:58:41 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 08:58:41 pm [DEL-RAND]        RAND 58:30:CA:B6:A2:4F expired after 188 seconds
Mar 24 20:56:07 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 08:56:07 pm [CMD-NAME]        98:00:C6:79:8F:55 JDPhone  Apple Inc
Mar 24 20:56:07 raspberrypi bash[6392]: [+] 0.2.124 24-03-2019 08:56:07 pm [CMD-INFO]        **** completed arrival scan ****

I am on a Pi Zero W using the on board Bluetooth.

jasondreher · March 24, 2019, 9:39pm

my mqtt works (although it shows 0 confidence) and I have the same as you.
did you copy and paste the MAC address, maybe a zero vs an O (number vs a letter).

- platform: mqtt
  state_topic: 'monitor/garage/00:9A:CD:B8:44:FE'
  value_template: '{{ value_json.confidence }}'
  unit_of_measurement: '%'
  name: 'nexus'