Bermuda - Bluetooth/BLE Room Presence and tracking [custom integration]

I’m using the current version, 0.5.1, and I’m seeing a constant stream of this in my log file (e.g. more than one a second):

2024-03-20 20:22:56.040 WARNING (MainThread) [custom_components.bermuda] Using freshest advert but it's still too old!

Any hints on debugging this further?

Thanks Scott, I think it might be a logic error on my part - this will happen for iBeacons where Bermuda has chosen the “wrong” source address to copy some metadata from, or if the iBeacon’s timestamp is somehow updated from some other source. All of this is internal to Bermuda though, so not a configuration issue on your end.

Any given iBeacon might have multiple MAC addresses advertising it, typically because devices randomise their MACs for privacy, so there should be a single “current” source mac. It looks like Bermuda is choosing an “older” source mac, so the timestamps on that device’s advertisements are older than the “last-seen” stamp on the beacon.

The other possibility is that I’m updating the timestamp on the beacon’s record from somewhere other than the most recent source device.

Do you have a local bluetooth device attached to your HA machine? If you have bluetooth proxy devices, are they esphome or Shelly Plus units?

I should have a chance in the next 24 hours or so to take a proper look at this, and will hopefully have a release to fix it soon.

I’ve raised an issue at Log spamming "Using freshest advert but it's still too old!" · Issue #138 · agittins/bermuda · GitHub to track the bug.

I don’t have any Shelly devices, they are all esphome based, but I was using the local bluetooth device - I disabled it and restarted, and the log spam stopped.. Nope, it started up again…

1 Like

I see these same messages in the log, and I also have a USB Bluetooth adapter connected directly to my HA machine. Hasn’t caused any trouble that I can see, but it is there.

1 Like

New release: v0.6.1 - IRK / iPhone / iOS / iWatch support and bugfixes!

(these notes are actually for v0.6.0 but I had to push a bugfix, so I’m announcing it with v0.6.1)

:rocket: This release brings the long-awaited support for iPhones and other iOS devices (and any other devices that use private MAC addresses with IRK). Huge thanks to @Jc2k for their guidance with tying in with their Private BLE Device integration!

Adding any device to the Private BLE Device integration will automatically create Bermuda tracking entities for area-based location.

:dart: Also new is that if you change the “area” assigned to a scanner, not only does it now work(!) but it also applies immediately without having to do a restart or reload.

This release includes fixes for:

  • #135 Integrate with Private BLE Device to support iOS / iPhone tracking with IRK
  • #137 Changing Area of a scanner does not get reflected in Bermuda (@adizanni)
  • #138 Log spamming “Using freshest advert but it’s still too old!” (@Scott8586 @sdholden28 - I’ve essentially just turned down the logging output).

What’s Changed

Full Changelog: Comparing v0.5.1...v0.6.0 · agittins/bermuda · GitHub

5 Likes

Heho
Do you have some sponsoring platform or something like this?
I can’t really help with programming, but I think this project has great potential and needs supporting.

1 Like

v0.6.2 is out - ready for 2024.4.0

A few small changes were needed for the upcoming HA 2024.4.0 release, this version includes those fixes.

3 Likes

Hey @RogueThorn - I only had a buymeacoffee set up, but you gave me the prod I needed to add some options… :smiley:

You can support the work:

Any support is extremely welcome - and completely optional!

1 Like

Not here to promote my work, but Bermude integration is quite a good fit to display the presence of the people in the area using my floor3d-card and objects representing people or pets. When I will be more advanced in the creation of my model I will show some picture!!!

Hi @adizanni that looks cool - if you ever create a guide for people on how to do that please link to it from here as well - I think lots of folk would be interested in it!

1 Like

Hi all…
I have a problem, after installing this integration, the BLE Bermuda Tracker is successfully added linked to my iPhone (via BLE IRK) and it works perfectly, but I find the log full of these errors “Could not discern area from scanner gl-s10-bt-proxy(iPhone Macaddress): None.Please assign an area then
reload this integration.”

Did I miss a configuration step?

Thank you

You need to assign your ble proxy to an area. That’s how Bermuda tells you what room your tracked device is in. If your ble proxy doesn’t have an area, then Bermuda has no way to know where the ble proxy is, and thus no way to know where you’re tracked device is.

2 Likes

Thank you so much @sdholden28 :pray:

This is one of the most useful HA projects; just installed a bunch of proxies and now I have room contextual dashboards for each device, by user! It makes managing the 200+ entities in my home so much easier for everyone in the family.

4 Likes

My magsafe mounted HD10+'es suddenly come to life; take one off the wall, walk around and it automatically navigates to the corresponding area view :ok_hand::ok_hand:

2 Likes

@agittins amazing project, kudos for your efforts!

I’m trying to setup Shelly Gen2 4PM as a proxy, but for reason no devices are correctly tracked?

I added all devices discovered, but they all fail to update area automatically (it showed 4 devices with just MAC addresses, no name, no ID).

What I am confused is the Shelly as proxies part, this should work out of the box right? As long as Shelly is installed with native HA integration and Shelly BT Gateway is enabled? I also have aioshelly BLE script 2.0 script installed on the Shelly.

Any help and guidance would be much appreciated :sweat_smile:

EDIT: It started working just like that after a while. While it’s unclear which BLE proxy is sniffing the devices, it seems that it is the Shelly. As a tags, I am tracking RuuviTags that were already configured on HA through it’s own integration. Some issues identified:

  1. Android phone is not recognised even if BT is turned on and discoverable?

  2. RuuviTag area is constantly being changed to “Unknown” and then back to area of the Shelly. What’s up with this? Is this due to only 1 proxy being configured or something Shelly related?

EDIT 2:

  1. Android phone issue was resolved by enabling BLE Transmitter on HA Companion app. I am still wondering why not all devices are picked up automatically? I am sure Android transmits some sort of messages all the time?

  2. I added a second proxy (another Shelly), but the tags just kept jumping randomly between different areas and sometimes showing “unknown” area. Distance between proxies was 10 meters and thick stone/metal walls between. Some how it almost felt like which ever proxy reported RSSI got the “ownership” of the tag and assigned area as per proxy settings. Is this expected/normal or have I misconfigured something? I even looked into lowering TX power for RuuviTags, but that requires building a custom FW which I did not do, as I plan to buy other tags to track if all works out.

1 Like

Awesome, glad the basics fired up for you. Names not showing up at first might be due to the tags not sending their name in the broadcast packet, so until/unless one of the proxies sends a specific request for the name it won’t show up, just the address.

Android phone is not recognised even if BT is turned on and discoverable?

By default Androids keep their mouths shut :slight_smile: If you install the HA companion app you can turn on the “BLE Transmitter” on (under “Sensors”). I use Balanced mode (3Hz) and medium transmitting power.

I’m surprised that it didn’t show up while discoverable, but maybe that’s a difference between BLE and classic bluetooth, I’m not sure.

RuuviTag area is constantly being changed to “Unknown” and then back to area of the Shelly.

There are a few reasons this will happen:

  • In Bermuda’s config, the max_radius defines how close a beacon needs to be to be considered “inside” an area. Note that distances are relative, so if your settings mean the device measures 20m away even though it’s right next to it, it will still treat it as 20m away. You can calibrate the ref power and attenuation to adjust the distance calculation, but at any rate I usually find it preferable to set a very high max_radius like 70m or something, so that as long as a beacon can be heard at all, it will be marked as being in the closest area.
  • devtracker_timeout in Bermuda’s config might also be relevant - even though it’s mainly for the device_tracker sensor, it might also affect area sensors (to be honest I can’t recall right now).
  • If you are not receiving packets very often (due to proxies missing them, or problems with signal strength) this will also cause more "unknown"s. The bermuda.dump_devices service (Developer Tools, Services) will dump the internal state of Bermuda and in that you can see the hist_interval value for a given device/scanner combo, which will give you an idea of how many packets you are receiving. Note that the current release has some problems with the dump_devices service so you might need to update via HACs to the main version if the service doesn’t work - at least until v0.6.4 is out.

I’m happy to take a look at the output of your dump_devices to see if anything looks amiss, just bear in mind it will contain your MAC addresses and area names.

Android phone issue was resolved by enabling BLE Transmitter on HA Companion app. I am still wondering why not all devices are picked up automatically? I am sure Android transmits some sort of messages all the time?

They often don’t, but also they’ll transmit using a random MAC address, so you need to set up the Private BLE Device component in HA, which involves finding the private key for the phone.

Sorry just saw your later edit :slight_smile:

Yes, that’s pretty much how it works, but both shelly’s should report receiving the advert, and Bermuda will assign the area based on which shelly got the stronger signal (more or less).

In the dump_devices service, you can click the Fill Example Data button, and replace the addresses with your tag’s MAC, that will give a much shorter dump showing the tag and the data from each scanner that Bermuda has gathered for it - that will probably help a fair bit with working things out.

Thanks for the quick response @agittins :slight_smile:

I think I need to try the calibration first and see, but I am wondering if this calibration is tag/beacon specfic in the end? Ie. Does mixing and matching different tags cause some issues?

Re:dump, I am running 0.6.3 via HACS (not sure how to update to MAIN branch), it gives an error if you try to get full dump out, but works fine for single MAC. Underneath you can see the report.

e7:34:40:49:d8:4e:
  name: Ruuvi D84E
  local_name: Ruuvi D84E
  prefname: Ruuvi D84E
  address: e7:34:40:49:d8:4e
  options:
    attenuation: 3
    devtracker_nothome_timeout: 30
    max_area_radius: 70
    max_velocity: 3
    ref_power: -55
    smoothing_samples: 10
    update_interval: 10
    configured_devices:
      - CB:DC:EB:AE:63:F3
      - 7F:4F:D5:CA:8A:C5
      - 67:9C:BC:B6:88:52
      - 7A:4A:B6:61:44:1F
      - 40:16:3A:06:A1:6F
      - 4A:35:26:55:B9:37
      - 6D:DD:5A:D8:AF:B6
      - E7:34:40:49:D8:4E
      - C2:32:F3:93:3D:82
      - F9:D1:99:55:05:EC
      - DD:02:93:81:5A:46
      - E6:42:0E:C6:F3:5B
      - E1:B4:7A:3D:A4:B7
      - 5A:B9:80:BA:19:53
      - 4B5C200D0F5D49C98EC6B1ECE752C3FC_100_1
      - 5D:36:B1:9E:91:47
      - 69:5D:D4:2E:54:65
      - 30:C6:F7:83:3D:1E
      - 30:C6:F7:85:13:22
  unique_id: e7:34:40:49:d8:4e
  mac_is_random: false
  area_id: main_house
  area_name: Main House
  area_distance: 3.4145488738336014
  area_rssi: -71
  area_scanner: shellypro4pm-30c6f7851320 (30:C6:F7:85:13:20)
  zone: home
  manufacturer: Ruuvi Innovations Ltd.
  connectable: false
  is_scanner: false
  beacon_type: not a beacon
  beacon_sources: []
  beacon_unique_id: null
  beacon_uuid: null
  beacon_major: null
  beacon_minor: null
  beacon_power: null
  entry_id: null
  create_sensor: true
  create_sensor_done: true
  create_tracker_done: true
  last_seen: 2908.229322223
  scanners:
    30:c6:f7:85:13:20:
      name: shellypro4pm-30c6f7851320 (30:C6:F7:85:13:20)
      area_id: main_house
      parent_device: e7:34:40:49:d8:4e
      stamp: 2908.229322223
      new_stamp: null
      hist_stamp:
        - 2908.229322223
        - 2905.661280528
        - 2903.410244061
        - 2829.811093864
        - 2825.994036562
        - 2824.733017685
        - 2743.745864367
        - 2741.17582978
        - 2738.600795253
        - 2660.22881081
      rssi: -71
      hist_rssi:
        - -71
        - -71
        - -70
        - -70
        - -71
        - -70
        - -71
        - -73
        - -70
        - -72
      hist_distance:
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.1622776601683795
        - 3.1622776601683795
        - 3.4145488738336014
        - 3.1622776601683795
        - 3.4145488738336014
        - 3.9810717055349722
        - 3.1622776601683795
        - 3.686945064519575
      hist_distance_by_interval:
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
      hist_interval:
        - 2.56804169499992
        - 2.2510364670001763
        - 73.59915019699974
        - 3.817057302000194
        - 1.2610188769999695
        - 80.98715331799986
        - 2.570034587000009
        - 2.575034527000298
        - 78.37198444299975
        - 2.5480297330000212
      hist_velocity:
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.1120689146370911
      stale_update_count: 255
      tx_power: -127
      rssi_distance: 3.4145488738336014
      rssi_distance_raw: 3.4145488738336014
      adverts: {}
      scanner_sends_stamps: true
      adapter: shellypro4pm-30c6f7851320
      source: 30:C6:F7:85:13:20
      options:
        attenuation: 3
        devtracker_nothome_timeout: 30
        max_area_radius: 70
        max_velocity: 3
        ref_power: -55
        smoothing_samples: 10
        update_interval: 10
        configured_devices:
          - CB:DC:EB:AE:63:F3
          - 7F:4F:D5:CA:8A:C5
          - 67:9C:BC:B6:88:52
          - 7A:4A:B6:61:44:1F
          - 40:16:3A:06:A1:6F
          - 4A:35:26:55:B9:37
          - 6D:DD:5A:D8:AF:B6
          - E7:34:40:49:D8:4E
          - C2:32:F3:93:3D:82
          - F9:D1:99:55:05:EC
          - DD:02:93:81:5A:46
          - E6:42:0E:C6:F3:5B
          - E1:B4:7A:3D:A4:B7
          - 5A:B9:80:BA:19:53
          - 4B5C200D0F5D49C98EC6B1ECE752C3FC_100_1
          - 5D:36:B1:9E:91:47
          - 69:5D:D4:2E:54:65
          - 30:C6:F7:83:3D:1E
          - 30:C6:F7:85:13:22
    30:c6:f7:83:3d:1c:
      name: shellypro4pm-30c6f7833d1c (30:C6:F7:83:3D:1C)
      area_id: living_room
      parent_device: e7:34:40:49:d8:4e
      stamp: 2887.773992775
      new_stamp: null
      hist_stamp:
        - 2887.773992775
        - 2886.029964972
        - 2882.544909545
        - 2880.577878343
        - 2807.564763375
        - 2804.143713308
        - 2728.692663644
        - 2718.670532547
        - 2717.157512938
        - 2638.768565192
      rssi: -76
      hist_rssi:
        - -76
        - -75
        - -76
        - -63
        - -76
        - -63
        - -63
        - -63
        - -75
        - -63
      hist_distance:
        - 5.011872336272722
        - 4.641588833612778
        - 5.011872336272722
        - 1.847849797422291
        - 5.011872336272722
        - 1.847849797422291
        - 1.847849797422291
        - 1.847849797422291
        - 4.641588833612778
        - 1.847849797422291
      hist_distance_by_interval:
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
      hist_interval:
        - 1.7440278029998808
        - 3.485055427000134
        - 1.9670312019998164
        - 73.01311496800008
        - 3.421050067000124
        - 75.45104966400004
        - 10.022131096999601
        - 1.5130196090003665
        - 78.38894774599976
        - 2638.768565192
      hist_velocity:
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
      stale_update_count: 258
      tx_power: -127
      rssi_distance: 5.011872336272722
      rssi_distance_raw: 5.011872336272722
      adverts: {}
      scanner_sends_stamps: true
      adapter: shellypro4pm-30c6f7833d1c
      source: 30:C6:F7:83:3D:1C
      options:
        attenuation: 3
        devtracker_nothome_timeout: 30
        max_area_radius: 70
        max_velocity: 3
        ref_power: -55
        smoothing_samples: 10
        update_interval: 10
        configured_devices:
          - CB:DC:EB:AE:63:F3
          - 7F:4F:D5:CA:8A:C5
          - 67:9C:BC:B6:88:52
          - 7A:4A:B6:61:44:1F
          - 40:16:3A:06:A1:6F
          - 4A:35:26:55:B9:37
          - 6D:DD:5A:D8:AF:B6
          - E7:34:40:49:D8:4E
          - C2:32:F3:93:3D:82
          - F9:D1:99:55:05:EC
          - DD:02:93:81:5A:46
          - E6:42:0E:C6:F3:5B
          - E1:B4:7A:3D:A4:B7
          - 5A:B9:80:BA:19:53
          - 4B5C200D0F5D49C98EC6B1ECE752C3FC_100_1
          - 5D:36:B1:9E:91:47
          - 69:5D:D4:2E:54:65
          - 30:C6:F7:83:3D:1E
          - 30:C6:F7:85:13:22

Yes, calibration is receiver and transmitter specific in practice, and that’s something Bermuda doesn’t yet take into account (there’s a ticket for it, and it happens to be pretty high on the priority list right now) but that said, with all things being “relative” and for the simple area-based detection, calibration in general doesn’t matter a great deal, except where you might have receivers with widely differing sensitivities (eg a bluetooth dongle that gets a much stronger signal than an esphome proxy can cause the dongle’s “area” to “win” proximity a lot more than it should).

Once we have per-receiver offsets for sensitivity this will improve, but I don’t think that’s the cause of your issues particularly.

Varying tag transmit strengths is less of an issue for the given system, since we are comparing the relative strength of reception across all the receivers. Once we have full trilateration it will be more important, but I suspect it still won’t be as critical as one would intuitively expect.

FWIW, in my setup I did once do a little bit of calibration testing with… some device (I can’t remember which) and I used that to get the default figures I ship Bermuda with, but haven’t done further calibration since - it just doesn’t matter a lot for the current algorithms.

In HACS, Integrations, Bermuda, then top-right “meatballs” menu, choose “Re-download”. Choose main in the list of versions, confirm then restart HA. But since you’re able to get the dump output I wouldn’t worry about it for now.

The output looks pretty good. I can see you have two proxies reporting data for that tag, both are Shelly Pro 4PM’s.

The main issue appears to be that the Shellys will often go a long period without reporting a new BLE packet. In hist_interval on both scanners the interval between packets is typically around 2.5 seconds (hist_interval should always be a multiple of the actual broadcast interval, and hopefully under 4 seconds most of the time). However we see a few instances on both shellys where it’s over a minute between reports. At the time you ran this dump, the living room shelly hasn’t been heard from for over 20 seconds (stamp).

The history indicates that the main house shelly gets a pretty stable rssi between -70 and -71, while the living room seems a bit more volatile, jumping between -63 and -76. The fact that they are seeing similar strengths, combined with the long gaps in reporting packets would explain why you’re seeing a lot of “bouncing” between areas.

I think the core problem is the big gaps in reports from the Shellys. I don’t know for sure, but my guess is that perhaps the Shellys are only reporting packets when they see something “new” to report. Eg, if the rssi doesn’t change, it doesn’t send another update. This makes sense in a typical bluetooth use-case, but in our case we really need to know if a beacon is still actively in a place. I think esphome for a while was also (effectively) caching results and not sending updates if nothing changed, but that was later removed.

I’ll see what I can find out about how the Shellys do their reporting, as hopefully it’s something you can change at your end.