This seems interesting: Location Awareness of BLE Connections Using RSSI Monitoring | Novel Bits
Wow, nice find. Yes it’s a very cool idea, and would be useful for both Bermuda and for the HA bluetooth stack generally, as it would also allow connected devices to roam between proxies.
The downside is that I think what they’re describing is basically an API for their SoC firmware, so as a solution it would only apply to Silicon Labs chips.
Supporting this on an esp32 (or any chip) would require either that it was built-in already, or that the radio firmware allowed listening to all traffic, not just traffic destined for that device (ie, analogous to “promiscuous mode” for ethernet adaptors). I don’t know if the current bluetooth stack used in esphome (or for that matter the low-level firmware for the espressif chips) supports that. To support roaming it would also have to support switching the effective MAC address in order to do transparent hand-over (unless the BLE spec has some built-in method for that). I feel like the esp32 as it is already feels a bit “stretched” doing just the esphome ble_proxy work, but I don’t have enough knowledge to know either way.
It might be worth raising in the esphome camp, or tagging @bdraco to see what they think. I do think it would be a pretty major undertaking, but I like the concept.
1 Like
Debugging is really a nightmare.
Haha, yep! When I get the individual sensors per-proxy set up this sort of troubleshooting should be a lot easier and more intuitive.
One FR that would be on the structure side in ha and nothing actual tech related:
Re-use the existing device in home assistant when adding entities. I mentioned it before but am failing to find the right wording.
Found the dev docs on it: Device Registry | Home Assistant Developer Docs
Doing this would also allow for autodiscovery of compatible to-be-tracked devices and in future the device’s area could be automatically updated by Bermuda
Is there a current “best” bluetooth tag that anyone has found to work better than others with this integration and Bluetooth proxy in general?My current use case is trying to add inside/outside tracking for the dogs, so looking for something I could attach to a collar or similar. I have a couple of tile devices laying around, but they don’t seem to advertise too often.
I have some super-generic tag thingies, and they seem to reliably advertise at 2 second intervals.
The ones I bought look exactly like these but for some reason I can’t find the exact listing in my aliexpress history. I know that the precise chip used in these products will sometimes change from batch to batch, so I can’t guarantee how any given ones will work, but mine still seems to be broadcasting reliably, and without having to do anything odd like setting them up in an app or anything.
If your dogs are medium-to-large, I’d probably put a couple of tags on their collars for redundancy, and take care to design your automations on the basis that what you get from Bermuda is an affirmative indication of a maximum distance, not an affirmation of their absense.
That is, if bermuda says it saw the tag at a certain distance, you can be confident that it really did see it, and it really was that far or closer, more or less.
But if you don’t get a reading from Bermuda, it might be for any number of reasons - tag is too far away, tag is covered by sleeping dog mass, proxies failed to be listening at the exact moment the tag broadcast, etc.
Let me know if that doesn’t make any sense or if you’ve got questions about how to set things up differently for say, knowing where the dogs are versus getting an alert when they are (or are not) in an area. Basically though, the more proxies you have the more reliable your results should be.
1 Like
Thanks! I picked up some BlueCharm beacons and they seem to be working well. I’m playing with signal strength on the beacons and placement of my proxies. I’m going to need more proxies for sure. The most important goal for me… I have a large dog that goes in and out at will during the day. There are times when I need to make sure she is outside during the day and I’m not at home, for various visitors etc. Once she’s outside, I can keep prevent the dog door from opening from the outside, but if she’s inside when I toggle that, then it’s blind reliance on the hope that she exits the door before someone shows up. So all of that to say, I really just want to know if she’s inside or out, which would mean no proxies are detecting her beacon. Which room she’s in is just kind of a bonus.
I’ve been testing using EMBC22 beacons with less-than-stellar results. The image below shows the status with the beacon stationary ~1ft from the ESP Proxy.
I did dumps to see what the ESP was reporting, and when the device is ‘Home’ I get a Low of -63, a high: -76, and an average of -70 after multiple tests (20 samples). When the device is ‘away’ the ESP dump doesn’t report anything for the device (no RSSI values).
I’m hypothesizing that to save power, the beacon only reports every ~2min. To counter this, I could increase the timeout value higher but two minutes is way too long for my application. I’m attempting to use this integration to open the doggie door for my pups whenever they’re within ~3ft of the door. Right now it’s simply too unreliable for that use.
Are others having much greater success than me? If so, what Bluetooth beacon(s) are you using that work reliably?
Ahh cool. So I’d suggest thinking in terms that you have a very large room, called “outside” (or back yard).
In the vein of “you can’t prove a negative”, what would probably work best for you is if you have at least one proxy that is not inside the house, so that you can have a positive indication that your puppo is outside. Say, if you have a garden shed, or maybe if you’re really keen, build a solar-powered proxy and nail it to a fence.
Any proxy you can position outside of the door will help greatly, even if it’s only a few metres away from it (verandah railing, porch post) - that way you will get a definitive answer that says “doggo is closer to the outside front stairs than she is to the hallway”, which gives you a solid indication of which side of the door she is on. Or put two proxies, one a metre either side of the doggie door (not to the sides, I mean one proxy 1m to the inside, one 1m to the outside). Then you’ll have a much more reliable indicator, plus you don’t need a recent signal either, you can automate on “If the most recent location was outside, consider her as still being outside”.
You could combine that with PIR sensors on each side of the doggie door, the last one that registered movement tells you if they were moving in or out (hehe you could consider it “quadrature encoding”). However that assumes there are no other potential triggers, but it might be helpful as a combined signal with the BLE.
As far as signal strength goes, unless you’re heavily concerned with battery life optimisation, I’d just leave it at max. The biggest issues I’ve seen tend to be around missing advertisements altogether. The idea is to calculate relative signal strengths, so the absolute strength (ie, tx power) isn’t helpful for calculation, and a reduced tx power simply means that it’s less likely that more distant proxies will pick it up at all, which means you lose the ability to make relative determinations.
Yeah, the integration will only report “away” if there are no received adverts, since anything it can hear is definitively “at home”.
It looks like your proxy is probably just missing a lot of the adverts, this is the most common situation. I’ve written a bit above about why that is (essentially, the esphome has to switch between doing wifi and doing BLE, and we are left making trade-offs around how well that can work).
My first suggestion is to install nRFConnect on your phone, and use that as your first tool for diagnostics. It can listen continuously while in scan mode, so you can tell what your beacon is really doing re advertisements and intervals. My guess is that you’ll find your beacon is fine, and is advertising at regular intervals - whether those intervals are close enough will depend on your situation, but I’d expect you should be able to configure it to advertise every second, depending on what firmware’s running on it.
The next thing is to look at your proxy’s configuration. The defaults are, in my opinion, not well-suited to this. The interval and window paramaters of scan_parameters is what you want to look at. The defaults (320 and 30) mean that the device is only listening 9% of the time! So if you’re sending adverts at 2sec intervals, and only getting 10% of them, expect 20 second gaps between receptions! Except it won’t be an even spread, so sometimes you’ll catch more, sometimes fewer.[1]
Solutions to this include:
- tweaking the
windowandintervalparams. I’m using 900 and 1000 in some cases, and I get 4 to 6 second intervals typically for beacons sending 1s adverts. If you make the ratios too close the device will lock up or fall off the network. - use multiple proxies, even in the same Area. This increases the odds of at least one of them receiving any given advert, since their timings will be different (I assume. Hopefully they’re not locking to a central timestamp!)
- use proxies that have ethernet, so you can run window/interval ratios much closer to 1:1 since they don’t need to use the radio for wifi.
Re your specific application (unlocking door when puppos within 3ft) I’d say yes, Bermuda might not be the best solution, certainly without adding another proxy or something else to avoid missing adverts. I’d consider the above points (especially window/interval tuning, since it’s free), but also perhaps:
- PIR might be a better choice, or combining PIR with Bermuda. Eg, “If puppo’s within several metres, and pir triggers, unlock door”.
- Set up an esp32 at the door, dedicated to listening for adverts. Use the
on_ble_advertiseevent in esphome to process the advertisements locally, and disable its wifi so you can use a 1:1 window/interval. You could connect another esp (even an 8266) to it via gpio to send info back to HA about whether the door’s unlocked etc.
Maybe with some timing experimentation or adding a proxy you’ll find it does what you need, otherwise the above might offer some ideas. I’d be interested to know what you find!
It’s more complicated than this because each advert is sent separately on three different channels, with a slightly randomised delay between, the proxy is only listening on one channel at a time, etc etc. But that’s the gist of it. ↩︎
That’s an interesting approach! I was going down the path of reducing signal strength, so as not to pickup the signal on inside receivers when the beacon was outside. Basically taking an “if no contact then must be outside” approach. I realized pretty quickly that “no contact” could sometimes happen even when inside near receivers, so your approach seems like a better path for sure. I’m currently using the device_tracker logic of the beacons, home = inside, away = outside. If I add an outside beacon or two, then the device_tracker would likely always show as “home”, so I would need to create a couple of new template sensors to show inside/outside based on last receiver contacted. That is trivial, just thinking out loud.
The design intent of Bermuda is that using multiple proxies, we can make estimations on location based on relative received signal strengths, so having as many receivers “hearing” the signal as possible will lead to better results, even (or actually, especially) if they’re in different Areas.
Note the use of the term Area refers to the per-sensor property, Area which you can think of as being a “Room”. Compare to device_tracker’s “Location” which is a binary of either Home or Not Home, which might then extend to HA’s Zone which is a coarse geolocation for broad areas, so Home, Work, Ex Partner's House etc.
The design intent of device_tracker services is a binary “within 200 metres of home” sort of thing.
I added device_tracker support because the existing iBeacon implementation was… weird? Maybe it just doesn’t have enough levers, and (at the time) the way it worked gave me the shits. It’s better now than it was though, I think.
Most importantly though, device_tracker is really about answering “is this thing close to my property, or nowhere near my property?”. That’s it. Down the street is equivalent to Myanmar, as far as device_tracker is concerned (Myanmar residents excluded). Not ideal for keeping tabs on pets.
I suggest you use the Area and/or Distance sensors that the Bermuda integration provides (if you haven’t seen them, go into settings, devices & services, Bermuda, devices and click on any of the devices you’ve included). These should give you much better sensors to build from. Here’s what Bermuda shows for my watch just now:
Note that you set the “Area” of each proxy, and Bermuda will use that to set the “Area” sensor of each device to be the same as its nearest-detected proxy.
Note however how noisy the signal is for my watch, above: I have two proxies in my studio, so it’s pretty solid when I am in the office (and I use it to maintain occupancy for my lighting and fan automation in that room), but the messy purple area is where it flips between bedroom / lounge, despite the watch being on my bedside table some two feet from the proxy! A lot of that is due to missed advertisements, and something I hope to clean up a bit with some filtering/logic fixes soon.
Keep in mind if your dog is sitting in the yard just outside your office window, say, the beacon will (rightly) be reported as being closer to that than the back door - something to keep in mind depending on your property’s layout and your dog’s habits / favourite lounging spots.
Hi Ashley,
Cool project!
A comment and question…
It’s weird how many advertisements you are missing. You mentioned having one successful “read” every 4-6 seconds from a beacon advertising at 1000ms…That sounds weird. Your advice to change window/interval settings on the scanner to 900/1000 is definitely correct. So I am wondering why the long gaps between reads? Maybe the bluetooth scanner is wonky? I would expect to capture one beacon broadcast per second with an only occasional delay of one second when the broadcast lands in the 100ms not-active time zone of the scanner.
Would your trilateration system work with my BCG04 BLE gateway? My gateway scans for beacons then sends this info via MQTT to the MQTT integration running on HA. Can your integration capture that data (MACs, UUIDs, rssi’s) from the MQTT integration? If so, I’d love to play around with it.
Oops, one more comment: I have found much greater success when I smooth the raw rssi values from the beacon. This gets rid of the wacky oddball readings that happen from time to time with anything BLE. Smoothing slows reaction time of the automation a little bit, but definitely within an acceptable range for most automations.
Here’s a sample project that explains how it works:
Hi Thomas!
Thanks, right back at you for your project, nicely done!
Yeah, the missing packets is frustrating. I haven’t dug more deeply into it, but I wonder if the esp is a bit overwhelmed with the number of packets they’re processing. Also I am not 100% certain about how the overall timing works - whether at 1000/900 the thing really is listening 90% of the time, or if there are other tasks that get done between windows, I’m not sure. Given one gets instability if you get closer to 1:1 I suspect the windows are probably consecutive but it’s only an assumption on my part. I’ll also re-examine how I’m doing my checks, since my integration only runs every 5 seconds, but I am using the timestamps that come from the bluetooth backend so that shouldn’t be an issue. An area of work is to have my processing happen on packet arrival, but not quite there yet.
Re smoothing, that’s what I’m in the middle of currently. I just committed to main a version that presents a sensor for the distance to every proxy, so that gives me much better / easier visibility into how the data is on the way in and how well the algo is making choices between locations, so that I can have some clear data on how well different filtering algos work. The data is already super helpful at visualising ways that bad decisions get made, and Jaymunro has already shown some promising results with some of the built-in filters in HA.
I’m thinking of implementing a kalman filter on each distance measurement, but if anyone knows of a better-suited algo please let me know! The key points are that the measurements arrive at unpredictable intervals, and the noise is totally asymmetrical (noise makes distances longer, almost never shorter), so I’d imagine there should be filters tailored to that environment.
Re ingesting data from MQTT, it does not. I’m processing the cached data that the HA bluetooth integration stores. This includes all data received from ble-proxies. I would guess one could either write an integration that fed the mqtt data into the bluetooth stack in HA, or perhaps your device could connect directly to HA as an esphome client does? I don’t plan on supporting alternative sources for ble adverts in Bermuda specifically, but it could certainly be done - but I’d suggest it would be better to inject into the bluetooth integration instead, that way any other integration could benefit from it.
Gah! It’s a bug! Thank you so much for prodding me to take a deeper look at this! Somehow way back when, I was just logging the interval based on when I processed the advertisement. I always intended to be running processing at each packet arrival, but so far it’s been a polled interval scan.
You got me thinking about my assumptions, and it didn’t make sense that the intervals were so unreliable given how responsive the esphome devices are. Sure enough, I’m daftly considering only the interval between my polls, not the actual interval between packets. It seems that the async HA polling is random enough that it looked like random intervals but much of that was just HA’s varying loop time plus the polling interval.
It will take me a bit to shake this one out but wanted to give you a nod right now - fixing this should improve things a lot!
Thanks 
3 Likes
Where should I send my invoice? 
One thing I was also going to mention: When an ESP is scanning for BLE, the scan interval and scan window are key inputs. Scan interval tells it how often to restart the scan. Scan window determines how long the scan will last. Many sources suggest setting these two numbers equal in order to “scan continuously”. But as you and I both discovered, this tends to overwhelm the ESP. Some people smarter than me suggested that this is because the BLE scanner and Wifi both are using the same “radio” on the ESP, so if you scan continuously, it blocks wifi (or at least disturbs it until the ESP goes bonkers). Hence my preference for scan interval 1000 and scan window 900, leaving 100ms of “free time” for the wifi and also for the ESP to do other housekeeping stuff without being bothered by scanning.
Another issue that is vaguely in my brain is the scan period for the scanner. I think this determines how long a “single scan” lasts, so if you set this for 5 seconds, you would have 5 rounds of interval/window during that 5 second period. BUT I think that some scanners will drop any duplicate scans that they receive during this period. So if your scanner “saw” beacon DD:11:22:33:44:55 5 times during this 5 second period, it might only capture/report the first result, and ignore the next 4 results. As I remember, you can fix this by “allowing duplicates” and/or setting the scan period to 1 second (to match the scan interval).
Caveat: I have forgotten most of what I previously only vaguely knew about ESP scanning. Plus I am not sure if the bluetooth dongles that people use with HA follow the same scanning rules as an ESP.
Yes, I use the same window/interval settings of 900/1000 also.
Re not sending dupes, I don’t think the scan period does that (at least not in the current esphome, it may have before some of bdraco’s changes). I get multiple updates from a given beacon during the one scan period. I have a vague recollection of duplicate suppression being removed ages ago, but not sure.
Heh… the usb bluetooth dongles I don’t even use for distance ranging (I do use them for device_tracker) because the stack doesn’t timestamp the advertisements. Currently if I see one from the local usb dongle I apply a stale timestamp to it and log it, but the algo will almost always ignore it due to being too old (on purpose).
So the issue with my update interval and advert interval has mostly turned out to be me badly naming the hist_interval variable. It really reflects the age of the advert at the time we polled the cache, plus a multiple of our polling interval. Which was 5 seconds, but I’m dropping it now to… 0.9seconds. The polling loop only takes 0.004 seconds most of the time (on my dev box anyway) so hopefully won’t affect global temps too much.
Anyway, key is that I must remember when looking at what I call “interval” it already has the polling interval baked into it. Not sure how to do this better, I think it would have to be up to the bluetooth integration to store interval data, which it doesn’t currently. I had thoughts of using a listener to process adverts as they arrive, but my initial efforts didn’t work (due to me not knowing what I’m doing) so I think I’ll see how a short polling interval goes. Hopefully it’s not a problem for little raspi’s.
Are these the values you recommend for my use case?
I’ve been looking for something like this so I could fully integrate my Shelly devices into Home Assistant using the Bluetooth as well so good work.
1 Like
Hi! Thanks for this, after being dissapointed with other solutions, this really looks like a promising project.
I was able to install it, and after setting up three of my shellies with Bluetooth to “active” mode via UI (and not enabling any other HA Bluetooth integrations etc.), I see it’s picking up some devices such as my refrigerator or NUKI locks (they got even their names resolved). I also see some other devices in the dump, having no idea what they are. However, even after setting up my Android Xioami Note 12 to send Beacon via the companion app, it doesn’t seem to pick it up - I can’t find the UUID-or-something-simialr anywhere in the bermuda.dump_devices results nor in the dropdown (I guess it’s the same), nor can I find my phone’s MAC address in the settings. I also can’t find my Mi Band 7 MAC address, but that might be because it’s connected to the phone as others hinted? Is there any way I can debug this to help out with this project? I am more after just tracking “is home” than needing specific area.
My iBeacon setup:
