[presence] Reliable, Multi-User, Distributed Bluetooth Occupancy/Presence Detection

daneboom · August 23, 2018, 8:19pm

This is such an amazing, promising piece of software!
Just to say the known static devices/addresses thing is still possibly a stumbling block for some. Nearly caught me out today.

I’m just trying to debug now. One thing I’m struggling with (and it’s so basic) is how to see the “stream of events” from the script live in ssh? I achieved that once by running bash monitor.sh but everytime I’ve tried that since the script hasn’t been able to terminate the old process and I get an I/O error.

Second thing I’m wondering about is how to “increment” my wife’s WiFi MAC to get her Bluetooth MAC as her WiFi MAC ends in a letter! Do I just go up the alphabet?! Thanks!!

benjimatt · August 23, 2018, 8:34pm

I think once you have an ssh link you can do this
sudo journalctl -f -u monitor.service

that should allow you to see a live stream of what is actually happening.

daneboom · August 23, 2018, 8:37pm

Thanks, but it’s not the same as the output I got by starting the script from the shell (as opposed to letting the RaPi start the script)

Sample output:

Aug 23 20:35:44 raspberrypi sudo[3248]: pam_unix(sudo:session): session closed for user root
Aug 23 20:35:50 raspberrypi sudo[3281]:     root : TTY=unknown ; PWD=/home/pi/monitor ; USER=root ; COMMAND=/usr/bin/timeout --signal SIGINT 30 hcitool lescan
Aug 23 20:35:50 raspberrypi sudo[3281]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 23 20:36:20 raspberrypi sudo[3281]: pam_unix(sudo:session): session closed for user root
Aug 23 20:36:25 raspberrypi sudo[3318]:     root : TTY=unknown ; PWD=/home/pi/monitor ; USER=root ; COMMAND=/usr/bin/timeout --signal SIGINT 30 hcitool lescan
Aug 23 20:36:25 raspberrypi sudo[3318]: pam_unix(sudo:session): session opened for user root by (uid=0)
Aug 23 20:36:55 raspberrypi sudo[3318]: pam_unix(sudo:session): session closed for user root
Aug 23 20:37:00 raspberrypi sudo[3363]:     root : TTY=unknown ; PWD=/home/pi/monitor ; USER=root ; COMMAND=/usr/bin/timeout --signal SIGINT 30 hcitool lescan
Aug 23 20:37:00 raspberrypi sudo[3363]: pam_unix(sudo:session): session opened for user root by (uid=0)

daneboom · August 23, 2018, 8:42pm

If I try to start the script again I get this

pi@raspberrypi:~/monitor $ sudo bash monitor.sh 
Starting monitor.sh (v. 0.1.482)...
> preference: delay between scans = 5
> preference: periodic arrive/depart check interval = 90
> preference: periodic arrive interval = 90
> preference: periodic depart interval = 120 
> preference: database refresh interval = 35
> preference: max arrival scan attempts = 2
> preference: max depart scan attempts = 4
> preference: random advertisement expiration = 120
> stopping other instances of 'monitor.sh'

and then usually nothing at all but then sometimes an I/O error.

Going to try a reinstall.

EDIT: Reinstall did the trick. I’m now able to call the script and it will then show me what’s happening in real time. I’m now going to attempt to link this with Home Assistant. I’m already monitoring the MQTT with MQTTFX on OS X, it doesn’t ever seem to find a ‘depart’ topic even though it’s able to find the ‘arrive’ topic. Ho Hum. I’ll see how I get on.

foraster · August 23, 2018, 9:43pm

To debug, the first times I run the script I don’t start it as a service, instead I run it with the debug parameter:

# ./presence -d

After startup, it will show DEBUG MSG: followed by the topics and other useful information.

Verify that you create the mqtt sensor in HA listening for the exact same topic.

Regarding the MAC address, it is written in hexadecimal (each digit goes from 0 to 15), so after 9 it comes A,B,C,D,E and F

If the last byte is 8F, the adding 1 will result 90, because F+1 (16= 15+1 (10 = 16 (10 = 90 (16
Hexadecimal addition in detail: Hexadecimal - Wikipedia

daneboom · August 23, 2018, 10:09pm

ah! I’m running monitor

walaj · August 23, 2018, 11:17pm

Any reason that this would happen:

{“retained”:“true”, “version”:“0.1.482”, “confidence”:“100”,“name”:"WalajGalaxyS8 ",“timestamp”:“Fri Aug 24 2018 08:52:36 GMT+1000 (AEST)”,“manufacturer”:“Samsung Electronics Co.,Ltd”,“type”:“Known Static MAC”}
{“retained”:“false”, “version”:“0.1.482”, “confidence”:“50”,“name”:"WalajGalaxyS8 ",“timestamp”:“Fri Aug 24 2018 09:11:18 GMT+1000 (AEST)”,“manufacturer”:“Unknown”,“type”:“Known Static MAC”}
{“retained”:“false”, “version”:“0.1.482”, “confidence”:“25”,“name”:"WalajGalaxyS8 ",“timestamp”:“Fri Aug 24 2018 09:11:30 GMT+1000 (AEST)”,“manufacturer”:“Unknown”,“type”:“Known Static MAC”}
{“retained”:“false”, “version”:“0.1.482”, “confidence”:“12”,“name”:"WalajGalaxyS8 ",“timestamp”:“Fri Aug 24 2018 09:11:43 GMT+1000 (AEST)”,“manufacturer”:“Unknown”,“type”:“Known Static MAC”}
{“retained”:“false”, “version”:“0.1.482”, “confidence”:“6”,“name”:"WalajGalaxyS8 ",“timestamp”:“Fri Aug 24 2018 09:11:55 GMT+1000 (AEST)”,“manufacturer”:“Unknown”,“type”:“Known Static MAC”}
{“retained”:“false”, “version”:“0.1.482”, “confidence”:“0”,“name”:"WalajGalaxyS8 ",“timestamp”:“Fri Aug 24 2018 09:12:33 GMT+1000 (AEST)”,“manufacturer”:“Samsung Electronics Co.,Ltd”,“type”:“Known Static MAC”}

Specifically why the manufacturer is unknown for the non-0/100 confidence entries?

daneboom · August 24, 2018, 7:50am

Calling the script from the ssh command line seems to break the script, necessitating a reinstall.

Odianosen25 · August 24, 2018, 8:51am

Hello @andrewjfreyer or @PianSom, anyone that can help really.

When using the monitor -t, I am a bit confused how its supposed to work; though honestly haven’t tried it, but I plan on doing this weekend when I get my kits in, and from the docs its confusing.

Am I supposed to run it with a -t flag and send it messages using the topics [topic path]/scan/ARRIVE or [topic path]/scan/DEPART?

What messages am I supposed to send to it? A blank payload? Or something like ARRIVE so it scans for arrive or DEPART or it scans for depart?

Can it do either or both depending on what I send to it? Looking into a situation where one could send it mqtt messages, depending on if all/some/no users are in the home and modifying how the monitor scans on the go, based on the readings it gets.

Its kind of similar to what you have in your documentation, but you didn’t state how you triggered it or what kind of message was sent to it to trigger.

Apologies if I am missing the obvious.

Thanks and regards

PianSom · August 24, 2018, 9:39am

Hi @Odianosen25

I ran a quick test. On one of my Pi Zeros I stopped the monitor service and started it manually with a -t.

Slightly to my surprise it then started a DEPART scan [4 repetitions] (and didn’t find anything, as no devices were near). It then posted zero confidence messages to MQTT for each device. I do not know why this happened.

I then posted a [topic path]/scan/ARRIVE with a blank message. I was using mosquitto_pub, which requires a message; my understanding is that no message is actually needed. It did a scan [2 repetitions] - found nothing, posted nothing.

I moved my phone to within range. On the log I saw

0.1.482 10:17:07 am [INSTRUCT] MQTT Trigger ARRIVE 
0.1.482 10:18:54 am [INSTRUCT] MQTT Trigger DEPART

but no other action (ie no MQTT posting) occurred.

I posted [topic path]/scan/ARRIVE with a blank message. A scan started, the phone was spotted and a MQTT message posted.

I posted [topic path]/scan/DEPART with a blank message. A scan for only the phone started, the phone was spotted, and no MQTT message posted.

I removed the phone from within range, and re-posted the ARRIVE topic. It did a 2 repetition scan, found nothing and posted nothing.

I then re-posted the DEPART topic. It checked for my phone and went in to the declining confidence loop, posting each time (50, 25, 12, 6) until it reached zero and stopped.

I stopped the monitor -t instance, moved my phone to within range, and started it again. No scans occurred this time.

I stopped the monitor -t instance, moved my phone out of range, and started it again. No scans again, leading me to believe that the first time it happened was due to a glitch in the Matrix.

Hope this helps!

Odianosen25 · August 24, 2018, 10:09am

Hello @PianSom,

Thanks for the quick feedback. So if I get what your readings say, its the below:

Using the -t flag, it can be used for both arrive and depart scans
I can use any payload to start it, as long as I send it to the right topic
The time interval with you triggering the scan (arrive/depart), and it detecting is pretty quick
So one has to basically trigger it to check, and it wouldn’t just scan for Arrive/depart on its own even if the phone is within/out of range respectively

So in all, the possible best way to run this is as thus:

One declares a range of sensors like door/garage to be used to trigger arrive/depart
If no one is home and the door/garage is triggered, it will only scan for arrive
If everyone is home and the door/garage is triggered, it will only scan for depart
If some in/out and the door is triggered, it first does an arrive scan, then does a depart (as it wouldn’t know if someone going out or coming in)

Do you think that’s a good assessment for speed check?

Odianosen25 · August 24, 2018, 10:41am

Hello @PianSom,

Please can you also do one more check for me, if running the service with anything outside the -t flag, like the -a or -r, can one still activate the scan in that mode?

Regards

PianSom · August 24, 2018, 10:41am

Hey @Odianosen25

Your points:

Yes
I imagine so, certainly blank works, so I guess it’s ignored
Yes - here’s the log. Less than 10 secs in this example (though this was the first device in the scan list)

0.1.482 10:20:55 am [INSTRUCT] MQTT Trigger ARRIVE 
0.1.482 10:20:58 am [CMD-INFO]	**** Started group scan. [x2 max rep] **** 
0.1.482 10:21:01 am [CMD-SCAN]	(No. 1) 40:98:xx;xx:xx arrived? 
0.1.482 10:21:04 am presence/owner/pi0kit/40:98:xx:xx:xx { version : 0.1.482, confidence : 100, name : xxx , timestamp : Fri Aug 24 2018 10:21:03 GMT+0100 (BST), manufacturer : Apple, Inc. , type : Known Static MAC ,"type":"Known Static MAC" }

Yes

For your algorithm, I’d think some fine tuning may be needed. For example, if I leave and close the back door then it may be a minute or two before my phone is out of range of the kitchen Pi. So an immediate DEPART scan may fail to notice.

Odianosen25 · August 24, 2018, 10:44am

Oh yeah sure, that’s configurable based on how the user has their system setup. Also I think only one pi location should be used to scan for arrive/depart when its done based on the door/garage trigger. But that will be based on the answer you give me above

Regards

PianSom · August 24, 2018, 11:10am

Hi @Odianosen25

Note that the topic is not device dependent, so a ARRIVE/DEPARTposting is heard by every monitor instance. However, I believe it may not be acted upon.

For example, here is the log entry from another Pi (running the default service with no flags) when a DEPART is published:

Aug 24 12:04:40 pi0ups bash[3525]: 0.1.482 12:04:40 pm [INSTRUCT] MQTT Trigger DEPART 
Aug 24 12:04:43 pi0ups bash[3525]: 0.1.482 12:04:43 pm [REJECT]        Departure scan request denied. Hardware busy.

and yet, after another identical publication:

Aug 24 12:06:54 pi0ups bash[3525]: 0.1.482 12:06:54 pm [INSTRUCT] MQTT Trigger DEPART 
Aug 24 12:06:57 pi0ups bash[3525]: 0.1.482 12:06:57 pm [CMD-INFO]        **** Started group scan. [x4 max rep] ****

So the answer is - not reliably, I guess.

PianSom · August 24, 2018, 11:24am

Hmmm.

When I arrive home by car it is usually first spotted because my car has a Wemos D1 in it which connects to the home network (car sensor). However, I then enter the house via the front door or the back door (both with sensors). The nearest Pi being the Kitchen (back door) or Upstairs (front door),

My point is that it is not always easy to know which pi to set scanning after a sensor trigger, so it may be better to allow for several.

Odianosen25 · August 24, 2018, 11:39am

Thanks for the responses and tests. This are my thoughts based on what you have provided so far

There is only one running with the default or -a or -r depending on user flag so it reads everything constantly. Just as how @andrewjfreyer has it, it should be in a location where most users are in the day.
When it sends a confidence of 0, meaning it can’t see anyone, it then triggers the rest to scan for arrival to confirm if the user is in the house. The other locations are declared in the app.yaml file.
Since it takes about 10 secs as you said, which is way less than the timeout to declare the user not home in the app (which won’t be less than 30 secs), the others would have reported so no false positives will exist
If door/garage sensors are available, when this is detected by the system, it sends a depart/arrive scan depending on if everyone is available or not.

Do you think that will be a fair assessment of how it should work? This way one doesn’t constantly scan and I believe detection will be way quicker as suggested by @benjimatt. @benjimatt, is this how you envisioned it based on your request?

Regards

Odianosen25 · August 24, 2018, 9:06pm

@andrewjfreyer,

Please is it possible to add some feedback for when the system has completed a scan? For example if I was to instruct the monitor with a -t flag to scan for Arrival or Departure, when its done, it will be nice to get some message back over mqtt so I can move unto the next thing.

Right now, there is no way for one to know, as the time its done scanning varies depending on the change that takes place.

Thanks and regards

foraster · August 25, 2018, 5:42pm

After some days of use, I confirm that I have been suffering the bluetooth-wifi interference reported above.

It was even harder when none of the devices was at home (because the script increases the scan frequency), making lots of network disconnect alerts

I’ll give the new monitor script a try!

daneboom · August 25, 2018, 8:28pm

One thought, and possibly this is something you’re all doing already and I’m not…

Is there a way the monitor script contacting the HA MQTT server could be taken into account in terms of a heartbeat signal? As in, if the MQTT server doesn’t hear from the Pi Zero W for an hour or so, then Home Assistant is alerted that there might be a connectivity problem?
Thanks!