Bermuda - Bluetooth/BLE Room Presence and tracking [custom integration]

Been really enjoying this integration, thank you for your work. Your recent change to reduce polling interval to 0.9s has been negatively affecting my setup, mainly that my proxies are doing double duty controlling about 25 lights via ble as well. I would recommend a user-configurable value in the option flow for the polling interval.

Also, I noticed the integration is CPU heavy and my HA instance slows to a crawl especially when I reload it - couldn’t figure out why.

1 Like

Howdy, yeah I want sure how well that would go, which is why i haven’t released that change yet, it’s only in the “main” branch in git.

Most likely the issue is the number of sensors. Having a sensor for every device+proxy combination results in a lot of data, and i think it takes it’s toll on the state machine, the data recorder and on memory usage for each browser (but not the proxies, it shouldn’t affect them at all).

If you reduce the number of devices you track that should help, but i plan to add options so that not as many sensors are created, and also to allow controlling (effectively) the update rate. That said, update cycles on my machine only take 0.004 seconds, it’s the number of sensors etc that seems to be the issue. My system went from about 20% to 50%+ cpu.

Resolving UUIDs for beacons isn’t in any of the released versions yet. It is in the “main” branch of the repo which you can choose in hacs, but it’s also where I’m testing some features that might cause issues.

I should be getting a new release out “soon” that will include resolving uuids from beacons.

It’s odd that you can’t find your phone on the list, though. Your phone’s “actual” mac won’t show, but the mac it’s using for beacons should. However that Mac is temporary and will change pretty often.

My production instance is having some issues with the 2024.3 update, most of which don’t seem to be from Bermuda but I wouldn’t recommend combining “main” with 2024.3 just yet unless you’re comfortable with troubleshooting :sweat_smile:

1 Like

Yep, your esphome params look good.

The other thing that i missed before is that you’ve got MAX_RADIUS in Bermuda set to 1m. Certainly while testing I’d crank that right up, maybe around 30m or so. Rssi is pretty approximate, and I’d expect a lot of the time it might read a few metres away even at 1m from the proxy, and if the beacon’s on a collar it might easily have half a dog between it and the proxy :blush:

Thanks! OK, nevermind then, I will prooobably wait then :grin:

So I have this loaded and a bunch of mac’s are showing up most without names some with names that aren’t helpful (TY is a tuya device).
I have a esp32 device set up as a ble Ibeacon using the dump devices service I was able to find it, I think.

08:d1:f9:d2:00:66:
  name: null
  local_name: null
  prefname: 021577d3388a8c4d105598024942ee3e138627b7f206c5
  address: 08:d1:f9:d2:00:66

The Pref name contains the uuid of the ibeacon

esp32_ble_beacon:
  type: iBeacon
  uuid: 77d3388a-8c4d-1055-9802-4942ee3e1386
  tx_power: 9

It does have a bunch of other numbers added at the begining and end but it seemed more than coincidental.

When I try to add it that mac does not come up on the list.

Is this because we ignore ibeacons?

Thus far I have not been able to identify either phone or smartwatch.

I will keep playing but if I am missing something obvious let me know.

Craig

Would you feel comfortable sending me the dump_devices output via pm? The MAC addresses and device names are about the only personal info that it would contain, I think.

If you use the “Fill Example Data” link on the service page and then add your device’s MAC to the addresses paramater it will just dump the info for that device (and the proxies underneath it).

Well done spotting the uuid in the middle of that prefname! I think Bermuda takes the full advertisement payload and dumps it in there if it doesn’t have something nicer to use :slight_smile:

It seems odd to me that it wouldn’t let you select the device to track, it shouldn’t be ignoring ibeacons (nor your watch for that matter!). Does your watch use a randomised MAC?

Perhaps your dump_devices output will help me work out what’s going on. Are you using esphome proxies or shelley’s?

BTW, if you turn on “Active: True” for esphome’s esp32_ble_tracker section (not the bluetooth_proxy section) then it might gather (more?) device names for you by requesting “scan response” packets from devices it notices. It’s not entirely reliable in my experience (or at least, the behaviour is complex enough that my brain has thrown up its hands and decided “I don’t care! I’m just going to consider it random”). Likewise, if you have a usb bluetooth dongle attached to HA you can enable its active scanning as well, which might give you names on the BLE devices.

Maybe you could add a user configurable data entry box for “number of dogs blocking the beacon signal (0.5 - 10 allowable range; units “dogs”). :wink:

1 Like

As Ashley said, the long one with stuff added at the beginning and end is the raw data broadcast.

The 02 means “iBeacon” and the 15 is number of data bytes remaining in the packet (15 is hex for 21). The stuff at the end is the major and minor and calibrated (fixed) rssi, all in hex.

Hi,
im trying to track my Phone (Google Pixel) with it but not able to find it and dont really know why. Im using Shelly as bluetooth proxys but today i also got an ESP Bluetooth Proxy setup but i cant get it to work. Anyone can help me please :slight_smile:

Android phones will be trackable in the next Bermuda release (coming in the next few days). You’ll need the phone to be broadcasting as an iBeacon - the HA companion app can do that, or there are probably other apps around that can also do it.

1 Like

Version 0.5.0 is out! iBeacon/Android! Faster Updates! Smoothing! Oh My!

This is a pretty major release with some nice, juicy features and fixes :slight_smile:
I recommend updating via the HACS UI but you do you!

  • :iphone: iBeacon support means that Android phones using the HA Companion app can now be tracked! Of course, any other iBeacon can also be tracked. Just look for the UUID in the configured devices drop-down (in the CONFIGURE dialog). Don’t worry, iPhone users - IRK support is in the works.
  • :stopwatch: Faster sensor updates - Bermuda now checks for new advertisements every second, meaning that responsiveness to entering an area etc should be much faster now. In order to do this without impacting system performance, I’ve implemented a sort of caching on sensor updates, which you can configure in the the options with the update_interval setting. This sets a maximum interval for sensor updates. Bermuda still checks every second, but if the update is boring (or unlikely to be accurate) the sensor won’t update. This saves database space, browser memory and cpu cycles. However, if Bermuda notices that a device has come closer to an Area, it publishes the update immediately. You can change the update_interval without having to reload HA so it’s easy to ramp it down to 1 second for some troubleshooting or wind it back up to reduce database impact.
  • :chart_with_upwards_trend: Sensor smoothing/filtering. Bermuda now uses a custom filtering chain to try and clean up the very noisy data that is inherent to RSSI measurements. The main thrust of the work is:
    • MAX_VELOCITY setting allows us to ignore any measurement that implies a device has moved away from an area at an improbable speed. These readings are almost always noise-induced.
    • Moving-window average smooths out the noisy data, but always gives preference to “nearer” distance measurements, as they are less susceptible to noise (or more concisely, nearer measurements have less noise). You can configure how many samples this filter uses to have a rough control over the maximum slope angle of distance increases.

Thanks to @jaymunro for his help and ideas while testing these filtering options!

  • :straight_ruler: Unit conversion should now work - nobody filed a bug for it but it’s fixed :tada:

  • :bar_chart: More fun sensors! Bermuda now provides sensors for every combination of device and proxy - but they are disabled by default. This means you can enable any that are of interest, particuarly useful for troubleshooting filters or learning more about how it is behaving. Adding more sensors can have an impact on your database size and system load, so bear that in mind before you turn them all on! Usually enabling a few key sensors on one testing device is enough to get the insight you need.

Your filtering dollars at work:

  • :neutral_face: The history card that pops up when you click a distance sensor now uses the default HA “5 minute averaged min/max/mean” graph instead of individual measurements mode. This results in no data showing until its been running for 5 minutes, but on the up-side it reduces the amount of data that needs to stay loaded in browsers so seems a reasonable compromise. The dedicated “History” tool is so much better these days, so I’d recommend using that any time you want to analyse the data.

Breaking Changes

  • :bomb: For performance improvements, many (most?) of the sensor extra attributes have been moved out to their own sensors. So if you were running automations that relied on attributes they will likely need updating. For example, if you triggered an automation on an Area sensor, and checked the distance attribute of that sensor, you’ll now have to trigger on the Area sensor, and add a separate step to check the separate Distance sensor.

Changes

  • wip: Get tests running again @agittins (#87)
  • chore: Update issue template for config screenshot @agittins (#88)
  • perf: Downgrade TX Power change log to debug @agittins (#89)
  • (feat) Per-proxy distance sensors @agittins (#107)
  • (chore) calmed some warnings down to debug. @agittins (#108)
  • fix: prevent non-scanner devices being saved to config @agittins (#109)
  • Chore: Replace reorder-imports with isort @agittins (#115)
  • feat: iBeacon Support! @agittins (#116)
  • Faster Polling (0.9seconds instead of 5), cleanups @agittins (#117)
  • Add mac_is_random, shorten update_interval, cleanups @agittins (#120)
  • Filtering of Distance sensors, startup and reload fixes, performance and db improvements @agittins (#127)
  • fix: Add timeout logic for distance measurements @agittins (#128)
  • feat: Ignore long distance readings that would require high velocity @agittins (#129)
  • Docs, Sensor Churn fixes, Unit conversions, Max_Velocity via ConfigFlow @agittins (#133)
1 Like

Great update! The faster polling is very noticeable, but the filtering and timeout logic make for a much more stable and accurate “location”. Kudos!

1 Like

v0.5.1 is out, just a small bugfix that stops out-of-range devices from getting “stuck” on the local usb bluetooth dongle (if present). Easily skipped if you don’t have a local bluetooth adaptor on your server and just use proxies.

Awesome, glad it’s working well for you!

Hello some initial feedback for v0.5.1

the area assignment is really much more stable, reactive and accurate:

this is before the new release:

image

this is after the new release (with default configuration):

image

with some changes also due to moving around in the house.

Of course you cannot do miracles, the indicators are really erratic and the positioning of the scanners can also lead to legitimate ambiguous readings.

I have not yet activated the new area distance entities, but I will take more time to test across the week end and try to tune the parameters.

1 Like

I have also noticed a very slight increase in the HA CPU usage:

image

Upgrade done at aroung 8:20am

Another question, how to refresh the name and or area assignement when a scanner is moved to another area ?

Thanks for the feedback, that’s super helpful.

What cpu is your HA machine using? A bit of cpu usage increase is expected, because we’re recalculating everything every second - it’s not really a “lot” of work, but it does mean that HA has to orchestrate it etc. I’m glad to see it’s not a big impact but it’s certainly measurable. My home system has so much going on in it that it’s hard for me to evaluate the load - it’s idling at about 24% and often bursts up to 60% depending on which dodgy integration is doing what! :smiley:

Given your setup seems really consistent it will be interesting to see what happens when you enable other sensors.

I think you’ve found a bug! At worst it should apply on a “reload” of the integration, but it looks like the way I am persisting some data is causing the old area to stick, probably across restarts, too. I’ve raised a ticket for it and will release a fix as soon as I can. Changing Area of a scanner does not get reflected in Bermuda · Issue #137 · agittins/bermuda · GitHub

1 Like

My production HA is based on a repurposed Laptop (Intel i5 5th generation), it is quite powerful (and energy greedy) but with more than 200 devices I never go over 4-5% CPU

1 Like