[presence] Reliable, Multi-User, Distributed Bluetooth Occupancy/Presence Detection

finally i have fixed the problem by changing the “mqtt_preferences” direct in the presence.sh
thank you for your help

I had been watching your repo, and saw the massive amount of commits going through, so was eager to give this a try. But my knowledge of linux is pretty low, so I will probably have to wait for some basic instructions to follow before I do.

But on a side note, I installed the presence script on two pi zero’s, and don’t seem to have problems with wifi interference anymore, it certainly seems to be showing less on my unifi controller, it only seem’d to get wifi interference when I was running it on the same Pi as HA was on.

The basic installation instructions are the same for monitor as presence. Here’s the highlights:

  • ssh to your PI
  • git clone git://github.com/andrewjfreyer/monitor
  • cd monitor
  • sudo bash monitor.sh

The first time your run it, it’ll tell you that it needs more packages installed (e.g., bc, hcidump, etc). Install these with sudo apt-get install [package].

The next time your run the script, it’ll tell you that your preference files have been created for you. These are “known_static_addresses” and “behavior_preferences” and “mqtt_preferences.” Edit these in a similar manner to presence.

Then, you should be ready to go.

I know that isn’t much information, but I hope its enough to get you started. It has been a lot of work to get this far, and I’m almost satisfied with the performance. Going to be writing documentation here soon.

4 Likes

Ohh dam, I hope I didn’t come across rude, and meant I wanted you to write instructions about it right away, if that was how you read it, so sorry, that wasn’t my intention, just meant I would wait until you get a chance to write it up. But many many thanks for a quick run down on it all. I will give it a shot later on, or tomorrow when I get some free time to play around, and will report back for you. Thank you again for the effort you have put into this project :+1:

1 Like

@andrewjfreyer, how can I improve arrival detection time, it takes a while to pick up, minutes instead of seconds, same when leaving.

Hi Andrew

Thanks for all the work. I have installed and just started testing.

One very minor suggestion - it would be useful for a “hello, world” message to be published to the broker on start, just so it’s easy to see that all is good. The LWT message is working just fine.

1 Like

Hi!

I have installed presence.sh on 3 devices now (rpi2, rpi3 and rpi zero w), and for my Android devices it works perfectly. Great work Andrew!
For the iOS-devices however, which are 1 iPhone SE and 1 iPad 3, it works from time to time. Most of the time confidence = 100 on all 3 rpi’s, but there also are some moments that my Apple devices are completely off-radar for more than an hour (and did not leave the building). Does this sound familiar to any of you?

It looks like bluetooth goes to sleep or something, because when I move the device very close to one of the rpi’s confidence still is 0%. Will try to find out in which situations this occurs exactly.

I have the same problem with a “extreme” case where the device is constantly “in-out” in 5min cycle.

I also confirm that on IOS devices is much more probable to get this type of issues, but i’m assuming isn’t not the script fault, it’s more a IOS power management issue.

@andrewjfreyer Is it possible to get verbouse output logging with maybe a day of turnaround? So that I/we can debug any problems with either the script or the broker client/server?

Thanks

1 Like

@jpm good idea. @andrewjfreyer so far Monitor does a perfect job. I have no more interference for 2 days, so all looking good for now.

Hi there!

I’ve been following the thread for a while, until today that I got everything ready and I was able to install it. Kudos to @andrewjfreyer for this! awesome work :slight_smile:

I tried the presence script, and works perfect so far, but I had the same issues where the interferences are killing the network. In my case, I’ve a wired Rpi3 acting as hostAP, and even though the Rpi is working perfect (as its wired), the devices connected to it through wireless has frequent disconnections… I’ve also a RpiZero wireless connected, and I’ve severe lag due to the interferences.

I’ve moved to monitor script then, and in the Rpi3 is working perfect, it detects my phone and the devices connected to it just failed to answer ping a 0.3% of the times (3 pings in 700), however, the PiZero doesn’t work at all, it’s returning this errors:

0.1.434 08:11:22 [ERROR] Attempting to correct HCI error: Set scan parameters failed: Input/output error
0.1.434 08:11:38 location/owner/PAlta/NOT_MY_MAC { confidence : 100, name : [Error], timestamp : mié jul 11 2018 20:11:38 GMT+0200 (CEST), manufacturer : IEEE Registration Authority ,“type”:“Known Static MAC”}
0.1.434 08:11:41 [CMD-PUBL] NOT_MY_MAC_2 ADV_IND [Error] IEEE Registration Authority PUBL_NUM: 1
0.1.434 08:11:47 location/owner/PAlta/NOT_MY_MAC_2 { confidence : 100, name : [Error], timestamp : mié jul 11 2018 20:11:46 GMT+0200 (CEST), manufacturer : CC&C Technologies, Inc ,“type”:“Known Static MAC”}
0.1.434 08:11:50 [CMD-PUBL] NOT_MY_MAC_2 ADV_IND [Error] CC&C Technologies, Inc PUBL_NUM: 2
0.1.434 08:11:53 [ERROR] Attempting to correct HCI error: Disable scan failed: Input/output error
0.1.434 08:14:05 [ERROR] Attempting to correct HCI error: Set scan parameters failed: Input/output error (x14)
0.1.434 08:14:14 [INSTRUCT] MQTT Trigger ARRIVE
0.1.434 08:14:17 [ERROR] Attempting to correct HCI error: Set scan parameters failed: Input/output error
0.1.434 08:14:17 [CMD-INFO] **** Started group scan. [x2 max rep] ****
Device is not available.
0.1.434 08:14:20 [CMD-SCAN] (No. 1) MY_MAC arrived?
0.1.434 08:14:23 [CMD-SCAN] (No. 1) WIFE_MAC arrived?
0.1.434 08:14:26 [INSTRUCT] MQTT Trigger DEPART
0.1.434 08:14:29 [REJECT] Departure scan request denied. Hardware busy.
0.1.434 08:14:32 [CMD-SCAN] (No. 2) MY_MAC arrived?
0.1.434 08:14:41 [ERROR] Attempting to correct HCI error: Set scan parameters failed: Input/output error (x2)
0.1.434 08:14:50 location/owner/PAlta/NOT_MY_MAC { confidence : 100, name : Unknown, timestamp : mié jul 11 2018 20:14:49 GMT+0200 (CEST), manufacturer : IEEE Registration Authority ,“type”:“Known Static MAC”}
0.1.434 08:14:53 [CMD-PUBL] NOT_MY_MAC ADV_IND IEEE Registration Authority PUBL_NUM: 3
0.1.434 08:14:58 [INSTRUCT] MQTT Trigger DEPART
0.1.434 08:15:01 [REJECT] Departure scan request denied. Hardware busy.
0.1.434 08:15:07 [INSTRUCT] MQTT Trigger DEPART
0.1.434 08:15:10 [REJECT] Departure scan request denied. Hardware busy.
0.1.434 08:15:13 [ERROR] Attempting to correct HCI error: Set scan parameters failed: Input/output error
0.1.434 08:15:22 [CLEARED] NOT_MY_MAC expired after 184 seconds PUBL_NUM: 2
0.1.434 08:15:34 [ERROR] Attempting to correct HCI error: Set scan parameters failed: Input/output error (x2)
Device is not available.
0.1.434 08:15:37 [CMD-SCAN] (No. 2) WIFE_MAC arrived?
0.1.434 08:15:40 [CMD-INFO] **** Completed scan. ****
0.1.434 08:15:44 location/owner/PAlta/MY_MAC { confidence : 0, name : Aitor, timestamp : mié jul 11 2018 20:15:44 GMT+0200 (CEST), manufacturer : Samsung Electronics Co.,Ltd ,“type”:“Known Static MAC”}
0.1.434 08:15:47 location/scan/depart
0.1.434 08:15:50 [CMD-NAME] MY_MAC [Error] Aitor Samsung Electronics Co.,Ltd
0.1.434 08:15:53 location/owner/PAlta/70:28:8B:62:C4:5D { confidence : 0, name : Lucia, timestamp : mié jul 11 2018 20:15:51 GMT+0200 (CEST), manufacturer : Samsung Electronics Co.,Ltd ,“type”:“Known Static MAC”}
0.1.434 08:15:56 location/scan/depart
0.1.434 08:15:59 [CMD-NAME] WIFE_MAC [Error] Lucia Samsung Electronics Co.,Ltd
0.1.434 08:16:11 [ERROR] Attempting to correct HCI error: Set scan parameters failed: Input/output error (x2)
^CExited.

However, with presence script it worked perfect:
DEBUG MSG: Scan result: [Galaxy S8]
location/owner/PAlta/MY_MAC { confidence : 100, name : Galaxy S8, scan_duration_ms: 1141, timestamp : mié jul 11 2018 20:16:34 GMT+0200 (CEST)}
DEBUG MSG: Scan result: []
location/owner/PAlta/WIFE_MAC { confidence : 0, name : Unknown, scan_duration_ms: 5084, timestamp : mié jul 11 2018 20:16:43 GMT+0200 (CEST)}
DEBUG MSG: Scanning for 0 guest devices between owner scans (at least one device is present).
DEBUG MSG: Remainder of delay before next owner scan: 30
DEBUG MSG: Scan result: [Galaxy S8]
location/owner/PAlta/MY_MAC { confidence : 100, name : Galaxy S8, scan_duration_ms: 1134, timestamp : mié jul 11 2018 20:17:14 GMT+0200 (CEST)}
DEBUG MSG: Scan result: []
location/owner/PAlta/WIFE_MAC { confidence : 0, name : Unknown, scan_duration_ms: 5084, timestamp : mié jul 11 2018 20:17:22 GMT+0200 (CEST)}
DEBUG MSG: Scanning for 0 guest devices between owner scans (at least one device is present).

I’ll probably be mixing both versions, monitor on the Pi3 and presence in the PiZero and keep providing feedback :slight_smile:

Regards

1 Like

@okercho thank you so much for testing this script. Still rough, but it’s definitely improving. Glad to hear that interference is gone.

The repeated error correction you see is a hardware issue for bluetooth. The system continues to go down. Do you have other bluetooth services operating on this pi? To scan for devices, the script needs exclusive use of the bluetooth hardware.

@jpm You can always pipe the debug output to a file. Depending on the version of the script that you’re using, you should also pipe STD_ERR. For example:

sudo bash presence.sh -d 1>&2 >> logging_output.log

2 Likes

I’ve noticed on the presence script that if you have a high number of devices, the last devices on the list are not reliably detected. Not sure the cause. How many devices are you trying to detect?

No, no! Nothing like that. The instructions are easy to type out! :slight_smile: I’m very appreciative that so many people are interested in this solution.

1 Like

I have a total of 4 devices in my owner_devices file. The one with troubles is the 3rd.
I will try with less devices, see if that makes a difference

Hi Andrew, thanks for replying.

Yes, I’m using that Pi to check the temperature from a Xiaomi BLE sensor :frowning:, I may need to setup another Pi for this…

Regards

Hey Andrew and all!
Firstly, love the idea of this project. Exactly what I was looking to try and do! Great work!

I’ve set it up on my Zero W, and I’m getting just “0” for my device in Home-Assistant. When I do sudo systemctl status presence.service I get the following:

● presence.service - Presence service
   Loaded: loaded (/etc/systemd/system/presence.service; enabled; vendor preset:
   Active: active (running) since Wed 2018-07-11 22:42:57 UTC; 3min 44s ago
 Main PID: 222 (bash)
   CGroup: /system.slice/presence.service
       ├─222 /bin/bash /home/pi/presence/presence.sh &
       └─750 sleep 30

Jul 11 22:42:57 raspberrypi systemd[1]: Started Presence service.
Jul 11 22:42:58 raspberrypi bash[222]: cat: /home/pi/presence/guest_devices: No 
Jul 11 22:42:59 raspberrypi bash[222]: Error: Network is unreachable
Jul 11 22:43:01 raspberrypi bash[222]: Error: Network is unreachable
Jul 11 22:43:02 raspberrypi bash[222]: Error: Network is unreachable
Jul 11 22:43:02 raspberrypi bash[222]: Error: Network is unreachable

My understanding was that this was it not being able to connect or authenticate to the mosquitto server. However, when I look at the mosquitto logs on Hassio, I can see:

1531349187: New client connected from 192.168.1.118 as mosqpub|720-raspberrypi (c1, k60, u'homeassistant').
1531349187: Client mosqpub|720-raspberrypi disconnected.
1 Like

Odd. You’re right the only network service that’s being used is the MQTT. I’ll investigate.

1 Like

Thanks! I didn’t notice anything else out of the ordinary (yet) but I’ll keep an eye out.
I’m wondering if there’s something going on with the bluetooth on my Pi - I tried to do the [BLUEZ UPGRADE] part of your instructions as well, just in case, but it didn’t seem to make any difference.

What’s the best way to do a test bluetooth scan from the command line? hciutil scan ? When I run that it just says “Scanning…” then nothing else. Which makes me think something bluetooth related isn’t working.

I switched to monitor.sh this morning and so far so good. Will try it out for the next few days and
will report back to you

1 Like