Bermuda - Bluetooth/BLE Room Presence and tracking [custom integration]

Share your Projects! Custom Integrations
77

@agittins amazing project, kudos for your efforts!

I’m trying to setup Shelly Gen2 4PM as a proxy, but for reason no devices are correctly tracked?

I added all devices discovered, but they all fail to update area automatically (it showed 4 devices with just MAC addresses, no name, no ID).

What I am confused is the Shelly as proxies part, this should work out of the box right? As long as Shelly is installed with native HA integration and Shelly BT Gateway is enabled? I also have aioshelly BLE script 2.0 script installed on the Shelly.

Any help and guidance would be much appreciated :sweat_smile:

EDIT: It started working just like that after a while. While it’s unclear which BLE proxy is sniffing the devices, it seems that it is the Shelly. As a tags, I am tracking RuuviTags that were already configured on HA through it’s own integration. Some issues identified:

  1. Android phone is not recognised even if BT is turned on and discoverable?

  2. RuuviTag area is constantly being changed to “Unknown” and then back to area of the Shelly. What’s up with this? Is this due to only 1 proxy being configured or something Shelly related?

EDIT 2:

  1. Android phone issue was resolved by enabling BLE Transmitter on HA Companion app. I am still wondering why not all devices are picked up automatically? I am sure Android transmits some sort of messages all the time?

  2. I added a second proxy (another Shelly), but the tags just kept jumping randomly between different areas and sometimes showing “unknown” area. Distance between proxies was 10 meters and thick stone/metal walls between. Some how it almost felt like which ever proxy reported RSSI got the “ownership” of the tag and assigned area as per proxy settings. Is this expected/normal or have I misconfigured something? I even looked into lowering TX power for RuuviTags, but that requires building a custom FW which I did not do, as I plan to buy other tags to track if all works out.

1 Like
78

Awesome, glad the basics fired up for you. Names not showing up at first might be due to the tags not sending their name in the broadcast packet, so until/unless one of the proxies sends a specific request for the name it won’t show up, just the address.

Android phone is not recognised even if BT is turned on and discoverable?

By default Androids keep their mouths shut :slight_smile: If you install the HA companion app you can turn on the “BLE Transmitter” on (under “Sensors”). I use Balanced mode (3Hz) and medium transmitting power.

I’m surprised that it didn’t show up while discoverable, but maybe that’s a difference between BLE and classic bluetooth, I’m not sure.

RuuviTag area is constantly being changed to “Unknown” and then back to area of the Shelly.

There are a few reasons this will happen:

  • In Bermuda’s config, the max_radius defines how close a beacon needs to be to be considered “inside” an area. Note that distances are relative, so if your settings mean the device measures 20m away even though it’s right next to it, it will still treat it as 20m away. You can calibrate the ref power and attenuation to adjust the distance calculation, but at any rate I usually find it preferable to set a very high max_radius like 70m or something, so that as long as a beacon can be heard at all, it will be marked as being in the closest area.
  • devtracker_timeout in Bermuda’s config might also be relevant - even though it’s mainly for the device_tracker sensor, it might also affect area sensors (to be honest I can’t recall right now).
  • If you are not receiving packets very often (due to proxies missing them, or problems with signal strength) this will also cause more "unknown"s. The bermuda.dump_devices service (Developer Tools, Services) will dump the internal state of Bermuda and in that you can see the hist_interval value for a given device/scanner combo, which will give you an idea of how many packets you are receiving. Note that the current release has some problems with the dump_devices service so you might need to update via HACs to the main version if the service doesn’t work - at least until v0.6.4 is out.

I’m happy to take a look at the output of your dump_devices to see if anything looks amiss, just bear in mind it will contain your MAC addresses and area names.

79

Android phone issue was resolved by enabling BLE Transmitter on HA Companion app. I am still wondering why not all devices are picked up automatically? I am sure Android transmits some sort of messages all the time?

They often don’t, but also they’ll transmit using a random MAC address, so you need to set up the Private BLE Device component in HA, which involves finding the private key for the phone.

Sorry just saw your later edit :slight_smile:

Yes, that’s pretty much how it works, but both shelly’s should report receiving the advert, and Bermuda will assign the area based on which shelly got the stronger signal (more or less).

In the dump_devices service, you can click the Fill Example Data button, and replace the addresses with your tag’s MAC, that will give a much shorter dump showing the tag and the data from each scanner that Bermuda has gathered for it - that will probably help a fair bit with working things out.

80

Thanks for the quick response @agittins :slight_smile:

I think I need to try the calibration first and see, but I am wondering if this calibration is tag/beacon specfic in the end? Ie. Does mixing and matching different tags cause some issues?

Re:dump, I am running 0.6.3 via HACS (not sure how to update to MAIN branch), it gives an error if you try to get full dump out, but works fine for single MAC. Underneath you can see the report.

e7:34:40:49:d8:4e:
  name: Ruuvi D84E
  local_name: Ruuvi D84E
  prefname: Ruuvi D84E
  address: e7:34:40:49:d8:4e
  options:
    attenuation: 3
    devtracker_nothome_timeout: 30
    max_area_radius: 70
    max_velocity: 3
    ref_power: -55
    smoothing_samples: 10
    update_interval: 10
    configured_devices:
      - CB:DC:EB:AE:63:F3
      - 7F:4F:D5:CA:8A:C5
      - 67:9C:BC:B6:88:52
      - 7A:4A:B6:61:44:1F
      - 40:16:3A:06:A1:6F
      - 4A:35:26:55:B9:37
      - 6D:DD:5A:D8:AF:B6
      - E7:34:40:49:D8:4E
      - C2:32:F3:93:3D:82
      - F9:D1:99:55:05:EC
      - DD:02:93:81:5A:46
      - E6:42:0E:C6:F3:5B
      - E1:B4:7A:3D:A4:B7
      - 5A:B9:80:BA:19:53
      - 4B5C200D0F5D49C98EC6B1ECE752C3FC_100_1
      - 5D:36:B1:9E:91:47
      - 69:5D:D4:2E:54:65
      - 30:C6:F7:83:3D:1E
      - 30:C6:F7:85:13:22
  unique_id: e7:34:40:49:d8:4e
  mac_is_random: false
  area_id: main_house
  area_name: Main House
  area_distance: 3.4145488738336014
  area_rssi: -71
  area_scanner: shellypro4pm-30c6f7851320 (30:C6:F7:85:13:20)
  zone: home
  manufacturer: Ruuvi Innovations Ltd.
  connectable: false
  is_scanner: false
  beacon_type: not a beacon
  beacon_sources: []
  beacon_unique_id: null
  beacon_uuid: null
  beacon_major: null
  beacon_minor: null
  beacon_power: null
  entry_id: null
  create_sensor: true
  create_sensor_done: true
  create_tracker_done: true
  last_seen: 2908.229322223
  scanners:
    30:c6:f7:85:13:20:
      name: shellypro4pm-30c6f7851320 (30:C6:F7:85:13:20)
      area_id: main_house
      parent_device: e7:34:40:49:d8:4e
      stamp: 2908.229322223
      new_stamp: null
      hist_stamp:
        - 2908.229322223
        - 2905.661280528
        - 2903.410244061
        - 2829.811093864
        - 2825.994036562
        - 2824.733017685
        - 2743.745864367
        - 2741.17582978
        - 2738.600795253
        - 2660.22881081
      rssi: -71
      hist_rssi:
        - -71
        - -71
        - -70
        - -70
        - -71
        - -70
        - -71
        - -73
        - -70
        - -72
      hist_distance:
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.1622776601683795
        - 3.1622776601683795
        - 3.4145488738336014
        - 3.1622776601683795
        - 3.4145488738336014
        - 3.9810717055349722
        - 3.1622776601683795
        - 3.686945064519575
      hist_distance_by_interval:
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
        - 3.4145488738336014
      hist_interval:
        - 2.56804169499992
        - 2.2510364670001763
        - 73.59915019699974
        - 3.817057302000194
        - 1.2610188769999695
        - 80.98715331799986
        - 2.570034587000009
        - 2.575034527000298
        - 78.37198444299975
        - 2.5480297330000212
      hist_velocity:
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.05234843785154959
        - 0.1120689146370911
      stale_update_count: 255
      tx_power: -127
      rssi_distance: 3.4145488738336014
      rssi_distance_raw: 3.4145488738336014
      adverts: {}
      scanner_sends_stamps: true
      adapter: shellypro4pm-30c6f7851320
      source: 30:C6:F7:85:13:20
      options:
        attenuation: 3
        devtracker_nothome_timeout: 30
        max_area_radius: 70
        max_velocity: 3
        ref_power: -55
        smoothing_samples: 10
        update_interval: 10
        configured_devices:
          - CB:DC:EB:AE:63:F3
          - 7F:4F:D5:CA:8A:C5
          - 67:9C:BC:B6:88:52
          - 7A:4A:B6:61:44:1F
          - 40:16:3A:06:A1:6F
          - 4A:35:26:55:B9:37
          - 6D:DD:5A:D8:AF:B6
          - E7:34:40:49:D8:4E
          - C2:32:F3:93:3D:82
          - F9:D1:99:55:05:EC
          - DD:02:93:81:5A:46
          - E6:42:0E:C6:F3:5B
          - E1:B4:7A:3D:A4:B7
          - 5A:B9:80:BA:19:53
          - 4B5C200D0F5D49C98EC6B1ECE752C3FC_100_1
          - 5D:36:B1:9E:91:47
          - 69:5D:D4:2E:54:65
          - 30:C6:F7:83:3D:1E
          - 30:C6:F7:85:13:22
    30:c6:f7:83:3d:1c:
      name: shellypro4pm-30c6f7833d1c (30:C6:F7:83:3D:1C)
      area_id: living_room
      parent_device: e7:34:40:49:d8:4e
      stamp: 2887.773992775
      new_stamp: null
      hist_stamp:
        - 2887.773992775
        - 2886.029964972
        - 2882.544909545
        - 2880.577878343
        - 2807.564763375
        - 2804.143713308
        - 2728.692663644
        - 2718.670532547
        - 2717.157512938
        - 2638.768565192
      rssi: -76
      hist_rssi:
        - -76
        - -75
        - -76
        - -63
        - -76
        - -63
        - -63
        - -63
        - -75
        - -63
      hist_distance:
        - 5.011872336272722
        - 4.641588833612778
        - 5.011872336272722
        - 1.847849797422291
        - 5.011872336272722
        - 1.847849797422291
        - 1.847849797422291
        - 1.847849797422291
        - 4.641588833612778
        - 1.847849797422291
      hist_distance_by_interval:
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
        - 5.011872336272722
      hist_interval:
        - 1.7440278029998808
        - 3.485055427000134
        - 1.9670312019998164
        - 73.01311496800008
        - 3.421050067000124
        - 75.45104966400004
        - 10.022131096999601
        - 1.5130196090003665
        - 78.38894774599976
        - 2638.768565192
      hist_velocity:
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
        - 0.4396848561468937
      stale_update_count: 258
      tx_power: -127
      rssi_distance: 5.011872336272722
      rssi_distance_raw: 5.011872336272722
      adverts: {}
      scanner_sends_stamps: true
      adapter: shellypro4pm-30c6f7833d1c
      source: 30:C6:F7:83:3D:1C
      options:
        attenuation: 3
        devtracker_nothome_timeout: 30
        max_area_radius: 70
        max_velocity: 3
        ref_power: -55
        smoothing_samples: 10
        update_interval: 10
        configured_devices:
          - CB:DC:EB:AE:63:F3
          - 7F:4F:D5:CA:8A:C5
          - 67:9C:BC:B6:88:52
          - 7A:4A:B6:61:44:1F
          - 40:16:3A:06:A1:6F
          - 4A:35:26:55:B9:37
          - 6D:DD:5A:D8:AF:B6
          - E7:34:40:49:D8:4E
          - C2:32:F3:93:3D:82
          - F9:D1:99:55:05:EC
          - DD:02:93:81:5A:46
          - E6:42:0E:C6:F3:5B
          - E1:B4:7A:3D:A4:B7
          - 5A:B9:80:BA:19:53
          - 4B5C200D0F5D49C98EC6B1ECE752C3FC_100_1
          - 5D:36:B1:9E:91:47
          - 69:5D:D4:2E:54:65
          - 30:C6:F7:83:3D:1E
          - 30:C6:F7:85:13:22
81

Yes, calibration is receiver and transmitter specific in practice, and that’s something Bermuda doesn’t yet take into account (there’s a ticket for it, and it happens to be pretty high on the priority list right now) but that said, with all things being “relative” and for the simple area-based detection, calibration in general doesn’t matter a great deal, except where you might have receivers with widely differing sensitivities (eg a bluetooth dongle that gets a much stronger signal than an esphome proxy can cause the dongle’s “area” to “win” proximity a lot more than it should).

Once we have per-receiver offsets for sensitivity this will improve, but I don’t think that’s the cause of your issues particularly.

Varying tag transmit strengths is less of an issue for the given system, since we are comparing the relative strength of reception across all the receivers. Once we have full trilateration it will be more important, but I suspect it still won’t be as critical as one would intuitively expect.

FWIW, in my setup I did once do a little bit of calibration testing with… some device (I can’t remember which) and I used that to get the default figures I ship Bermuda with, but haven’t done further calibration since - it just doesn’t matter a lot for the current algorithms.

In HACS, Integrations, Bermuda, then top-right “meatballs” menu, choose “Re-download”. Choose main in the list of versions, confirm then restart HA. But since you’re able to get the dump output I wouldn’t worry about it for now.

The output looks pretty good. I can see you have two proxies reporting data for that tag, both are Shelly Pro 4PM’s.

The main issue appears to be that the Shellys will often go a long period without reporting a new BLE packet. In hist_interval on both scanners the interval between packets is typically around 2.5 seconds (hist_interval should always be a multiple of the actual broadcast interval, and hopefully under 4 seconds most of the time). However we see a few instances on both shellys where it’s over a minute between reports. At the time you ran this dump, the living room shelly hasn’t been heard from for over 20 seconds (stamp).

The history indicates that the main house shelly gets a pretty stable rssi between -70 and -71, while the living room seems a bit more volatile, jumping between -63 and -76. The fact that they are seeing similar strengths, combined with the long gaps in reporting packets would explain why you’re seeing a lot of “bouncing” between areas.

I think the core problem is the big gaps in reports from the Shellys. I don’t know for sure, but my guess is that perhaps the Shellys are only reporting packets when they see something “new” to report. Eg, if the rssi doesn’t change, it doesn’t send another update. This makes sense in a typical bluetooth use-case, but in our case we really need to know if a beacon is still actively in a place. I think esphome for a while was also (effectively) caching results and not sending updates if nothing changed, but that was later removed.

I’ll see what I can find out about how the Shellys do their reporting, as hopefully it’s something you can change at your end.

82

Ah, your explanation does make sense! Shelly does have scripting interface that enables the BLE Observer, maybe some adjustments to the script could force them to update regularly?

// aioshelly BLE script 2.0
const queueServeTimer = 100; // in ms, timer for events emitting
const burstSendCount =  5; // number if events, emitted on timer event
const maxQueue =  32; // if the queue exceeds the limit, all new events are ignored until it empties
const packetsInSingleEvent = 16; // max number of packets in single event

let queue = [];
let timerHandler = null;

function timerCallback() {
  for(let i = 0; i < burstSendCount; i++) {
    if (queue.length <= 0) {
      break;
    }

    Shelly.emitEvent(
      "ble.scan_result", [
        2,
        queue.slice(0, packetsInSingleEvent),
      ]
    );
    queue = queue.slice(packetsInSingleEvent);
  }

  timerHandler = null;
  if (queue.length > 0) {
    timerHandler = Timer.set(queueServeTimer, false, timerCallback);
  }
}

function bleCallback(event, res) {
  if (event !== BLE.Scanner.SCAN_RESULT) {
    return
  }

  if (queue.length > maxQueue) {
    return;
  }

  queue.push([
    res.addr,
    res.rssi,
    btoa(res.advData),
    btoa(res.scanRsp)
  ]);

  if(!timerHandler) {
    timerHandler = Timer.set(queueServeTimer, false, timerCallback);
  }
}

// Skip starting if scanner is active
if (!BLE.Scanner.isRunning()) {
  BLE.Scanner.Start({
    duration_ms: -1,
    active: true,
    interval_ms: 320,
    window_ms: 30,
  });
}

BLE.Scanner.Subscribe(bleCallback);

Above is the aioshelly BLE script v2 that gets automatically installed when BLE Active listening is enabled from native HA integration settings.

I was considering getting some M5Stack Lite ESP32 proxies, but I read various issues about the battery issues. Also running BLE proxy with battery seems not recommended. Considering cabling etc, for me Shelly Plus 1PM Mini Gen3 makes the most sense (price, versatility and installation).

Did I read correctly, you mentioned multiple proxies in same room/area could improve reliability? If so, do you have any recommendations for minimum/maximum distances between proxies?

EDIT: Found some discussion about Shelly BLE, there’s also passive script to run on the Shelly:

// aioshelly BLE script 1.0
BLE.Scanner.Subscribe(function (ev, res) {
    if (ev === BLE.Scanner.SCAN_RESULT) {
        Shelly.emitEvent("ble.scan_result", [
            1,
            res.addr,
            res.rssi,
            btoa(res.advData),
            btoa(res.scanRsp)
        ]);
    }
});
BLE.Scanner.Start({
    duration_ms: -1,
    active: false,
    interval_ms: 320,
    window_ms: 30,
});

As far as I understood, active listening sends requests to surrounding devices and asks for updates, while passive just receives broadcasted messages. Not sure if this actually matters in this use case?

83

Yes, I suspect the key will be in fine-tuning the script to get the best responsiveness. Hopefully some peeps in here using Shelly’s can offer some tips.

I don’t have any Shelly’s yet, but taking a look at the docs it seems that

  • advertisements are ignored if they match one seen in the last 3 seconds (this is not great for our use-case, it will cause latency in detection (but not initial detection, so probably no biggie there) but also will cause bouncing between receivers if their distances are similar, since Bermuda might decide the fresher receipt is preferred due to freshness. This might also affect how eventual trilateration behaves, if it decides to discard “older” receipts - but I think I’ll probably allow older receipts for trliateration for this reason.
  • The window timings on those scripts are fairly short - it’s listening for 30ms our of 320ms (although possibly 3x30ms, not sure how they’re slicing it up between the 3 default advert channels). Their docs are pretty clear about the risks of tightening up those timings too much though, as it will depend on what else the mcu has to get done as to what will cause stability - they also mention the api possibly applying hard limits on what you configure anyway.
  • The api docs mention that if scripts take too long it will disable them, I wonder if the longer gaps in your hist_interval are due to the device rebooting or perhaps stopping/disabling/restarting the script?

Also possible is that due to the queue size and burst sizes, maybe the queue of adverts never gets fully cleared, which might result in long periods of a given beacon being at the back of a queue for a while (and never fowarded) and later at the front of the queue, etc. If the intervals of transmission and the timings of queues/windows etc line up a bit it’s easy to imagine getting a slow “wave” of adverts being caught regularly for a while then being missed for a while.

The active param of BLE.Scanner.Start seems to indicate “active scanning” (which is a term used confusingly in different ways) - in this case I think meaning that when the shelly gets a BLE advert, it then sends a scan-request packet to the device, and the device is then expected to reply with a scan-response packet, which often includes extra info like the device’s name, battery etc. This can be nice to get more names listed in the devices, but the downside is that doing this takes extra time for the shelly when it can’t be listening for new adverts or fowarding existing queued ones. I’d experiment with turning this off and seeing if it improves the hist_interval log. Bermuda uses just the passive advertising results (but an active scan will often result in it getting nicer device names on top of that).

That thread you linked has some excellent info on the Shellys, but because we can’t see the source code it’s always a bit hard to know exactly how to proceed, other than direct experimentation or asking Shelly directly. This might not be a use-case they have directly considered, though - so it might be worth asking. If I were submitting a feature request, one idea I’d put forward is that in the 3 second “caching”, if the new rssi is stronger than the previous rssi, it should override and send the new advert, since the stronger signal makes that advert “of interest” to us. If the signal is weaker or the same, that’s “less interesting”, for Bermuda’s purposes.

Bear in mind that as the number of bluetooth devices around you increases, the proxies have to spend more of their time processing and forwarding adverts. This makes fine-tuning inherently a site-specific thing, and probably means having to find a lot of your own trade-offs for getting good performance and stability from your specific environment.

On the github, @jaymunro seems to get good results with their Shelly’s, so I’ve asked them if they had to do anything in particular to get reliable adverts, fingers crossed :slight_smile:

84

@lazmo88 did you turn on active scanning inside HA Shelly integration?

This automatically installs a script on the Shelly to repeat Bluetooth advertisements. Without this the Shelly’s are only going to proxy their own BLU devices (and occasionally other 3rd party ones in my experience).

IMG_0520
IMG_05202239×1316 197 KB

Passive is less noisey but will not interrogate the transmitter for further details (such as name) unless Active is selected.

1 Like
85

FYI, this is the script that gets automatically installed by Shelly’s HA integration. I’ve never had to touch it and have had completely reliable Bluetooth proxying.

IMG_0521
IMG_05211881×1196 173 KB

Full script is:

// aioshelly BLE script 2.0
const queueServeTimer = 100; // in ms, timer for events emitting
const burstSendCount =  5; // number if events, emitted on timer event
const maxQueue =  32; // if the queue exceeds the limit, all new events are ignored until it empties
const packetsInSingleEvent = 16; // max number of packets in single event

let queue = [];
let timerHandler = null;

function timerCallback() {
  for(let i = 0; i < burstSendCount; i++) {
    if (queue.length <= 0) {
      break;
    }

    Shelly.emitEvent(
      "ble.scan_result", [
        2,
        queue.slice(0, packetsInSingleEvent),
      ]
    );
    queue = queue.slice(packetsInSingleEvent);
  }

  timerHandler = null;
  if (queue.length > 0) {
    timerHandler = Timer.set(queueServeTimer, false, timerCallback);
  }
}

function bleCallback(event, res) {
  if (event !== BLE.Scanner.SCAN_RESULT) {
    return
  }

  if (queue.length > maxQueue) {
    return;
  }

  queue.push([
    res.addr,
    res.rssi,
    btoa(res.advData),
    btoa(res.scanRsp)
  ]);

  if(!timerHandler) {
    timerHandler = Timer.set(queueServeTimer, false, timerCallback);
  }
}

// Skip starting if scanner is active
if (!BLE.Scanner.isRunning()) {
  BLE.Scanner.Start({
    duration_ms: -1,
    active: true,
    interval_ms: 320,
    window_ms: 30,
  });
}

BLE.Scanner.Subscribe(bleCallback);

But my suggestion is do not copy the script above, just erase the one you added and then use the HA integration to add the above. This way there’s a better chance it will be supported with future updates.

1 Like
86

Thanks @agittins and @HJM! I have installed the default script from HA by selecting active listening. The aioshelly BLE script 2.0 was installed, but maybe I tried fine tuning it without any clue what does what :joy:

I guess I’ll just need to take leap of faith and order more Shelly devices to see if accuracy improves with more proxies/beacons in the area/rooms. I’m anticipating outdoor/terrace/glass walls to be extremely tricky. Then again, I’m circling back to idea of embedding proxies to everyday items in the room. Perhaps using the M5Stack boards with batteries/small solar panels. Stronger signal is alway stronger signal which should help Bermuda to pick the right beacon location.

About Bermuda, can we adjust the update period somehow? Let’s say Bermuda would process updates only every 15 or 30 seconds and consider all broadcasted packages during this time frame? This should remove the “unknown” “unavailable” flickering I have seen :thinking:

This would allow Shelly/any proxy device to send multiple patches of data and then Bermuda would select the strongest signals for mapping tags to beacons.

1 Like
87

Great, let’s see how your detection goes with the vanilla setup (and thanks @HJM for chipping in!)

It should only go to Unknown after about 30 seconds, I think. By that point, it’s safe to say something is broken. BLE devices are deemed “unconnectable” by linux if they don’t broadcast at least every… 2.8 seconds, I think?

If it’s still flapping, take a look at the History page, and add the entities you are tracking, then narrow down the time range displayed so you can see what’s going on.

Can you also check the logbook on your shelly devices and see if perhaps their network connection might be dropping out? Do any of their sensors go “unavailable” periodically? You can add their wifi signal or other sensors to the history page so you can get a fuller picture of what’s going on, perhaps.

The problem with considering all the packets received over a 30 second block is that the data is pretty useless by then. If you walk down a hallway you could have passed through five areas in that time at a very leisurely pace - Bermuda then has to work out where you are now so recent data is critical. That said, you can always add a time element to any automations so they only trigger after a state has remained for a certain time, etc - or you could create template sensors that average out the values further. But so far your proxies have been not sending any data for over a minute at a time - so I am not sure a 30 second filter would entirely help :confused:

You might want to take a look at some of the discussion (and pretty graphs) in Discussion for further sensor filtering and algorithmic improvements · Issue #134 · agittins/bermuda · GitHub and the previous discussion at Add per-proxy distance sensors for configured beacons · Issue #99 · agittins/bermuda · GitHub for some ideas on how the work has gone in getting the area and distance detection into it’s current state.

You might notice there that there are a lot more sensors you can enable if you want to do some deeper testing - you can enable sensors for raw and filtered distances between every combination of device and proxy - bear in mind doing so will cause your database to grow more quickly, and your UI might get a little more sluggish on phones/tablets, but a handful of sensors on a few chosen devices should be no problem, and it makes the graphs a lot more exciting :smile:

If you have just the two proxies so far, then yes I’d definitely suggest that adding a few more should help. Also, right now you seem to have the two proxies fairly equidistant from where you were last trying to get a fix (when you did the dump_devices export). Bermuda does a fair bit of smoothing and averaging to try and reduce the flapping, but it’s just an unfortunate reality that if you are equally close to both proxies, it’s going to either oscillate between them, or take too long to change when you move toward one. The rssi signal is noisy, and there’s a real tradeoff between latency and volatility. I think the units you have are DIN-rail mounted, so you are probably pretty limited in where you can put them for now!

I use clones of the Wemos D1mini32 - they cost me about AU$6ea and are easy to pop into anything with a spare USB port around the house (although I also use a number of wall-warts I’ve scavenged over the years). I wouldn’t expect battery-power to be much fun, since the wifi part will use a lot of juice. But any way you can add more devices (or have them more centrally located in an “area” will help.

Once we have full trilateration I expect that having proxies actually in an area might not be necessary at all, but having a generous amount of them spaced around the house will always be key to getting good results - but 4 should be fairly solid for starters, providing they are reliably forwarding packets - which is the core of the issue you’ve been facing so far.

Once you’ve had a chance to see how the last changes go, could you send through some fresh data from the dump_devices service call? In particular I’m interested in the hist_interval stats. Fingers crossed! :slight_smile:

88

Oh, something else just popped into my head… You could enable the “distance to …” sensors for a device, and with your automaton you could check the distances - you could treat it as “being within 8 metres of living room is living room, regardless if that means 5m away from kitchen”. It won’t solve the dropout issue you were having but it might solve the flipping between areas problem, at a pinch.

89

I’ll give it a shot! Does Bermuda also consider distance value in the calculations? Another thought popped in my mind, would it be possible to keep area unchanged until there’s enough confirmations for the area determination? Ie. Wait for 5 confirmations/data points before changing device area? Or would it be possible to publish new locations to MQTT directly from Bermuda?

Second thought I had was adjusting tx power of the tag itself. Unfortunately RuuviTags I have requires manually compiled firmware, as there’s no software adjustments as there is on BlueUp tags. I would assume lowering the TX power and adding more proxies/anchors should make tracking much more accurate :thinking:

I’ll call BlueUp tomorrow and see if their tags are easily available for testing :sweat_smile: For proxies I’ll stick with the Shelly plan, but may play around with battery powered proxies as well.

There’s also mesh protocol called Wirepas, which works on bluetooth, also the proxies/anchors.

EDIT: BlueUp not easily available, but the tags seems very versatile and easy to config. Wirepas is commercial bluetooth mesh, which operates purely on battery devices (pretty flexible), but isn’t very fast with updates. Gone through a rabbit hole of different locating technologies and all of them have their pros/cons and in the end it boils down to the planned use case. There’s not a lot of open source or community development around this topic, most of indoor location systems are commercial services aimed for niche market, none of them is consumer friendly.

@agittins hats off for embarking on this path of a pioneer! It is truly remarkable work :ok_hand: It seems like AoA/AoD, Mesh tech stack is been kept as hostage by few software development companies, even if the hardware is quite freely available for consumers. I know we are not going there (yet), but I’m sure this project will inspire others to push the boundaries :wink:

1 Like
90

I have been watching the RuuviTags now, and maybe it is RuuviTag issue… They seem to send updates only when one of the sensors change (temperature, humidity, pressure, battery voltage, movement detected). Maybe I have been looking this issue wrong all the time and root cause is the tags. If so, it does make sense from battery management perspective. Which actually made me thing about the MQTT idea, solution for this could be automation that triggers on Bermuda area change, ignores unavailable status and posts all other updates to MQTT topic, which I could turn into more stable sensors for HA dashboard? :thinking:

EDIT: Contacted RuuviTag team, above assumption was not completely correct. RuuviTags doed broadcast MAC with 1285 ms interval, which is much higher than iBeacon normally do, but still less than 10000 ms that is said to be the upperbound for iBeacons. It is still lower than 2.8s threshold.

Shelly logs didn’t show continuous or repeating connection issues (3 disconnect events in past 24h which IMHO is fine for WiFi). Neither did Shellys on device logs show anything weird.

I looked into Bermuda addon debug logs and used the service dump, but I can’t see any error errors, but I see some duplicate creation messages and beacon type “not a beacon” parameters in the dump.

Strange this is, this exactly same behaviour is also happening with HA Android Companion App tracker (my mobile). That would point again that RuuviTag may not be the culprit after all, then again Shelly logs (device or integration) doesn’t show any errors. Native Shelly integration works through RPC as it seems, not sure if other protocol could be more stable.

TLDR; Something seems to be off with the Shelly as a proxy or bluetooth_le_tracker config that causes devices to go “Unavailable” all the time.

Logs: Bermuda Add-On Debug Log2024-04-23 14:28:49.045 WARNING (SyncWorker_0) [home - Pastebin.com

Bluetooth from configuration.yaml:

bluetooth:
device_tracker:
platform: bluetooth_le_tracker
track_new_devices:true
track_battery: true
track_battery_interval: 3600

1000055848
10000558481440×3216 152 KB

1000055850
10000558501438×2803 151 KB

1000055847
10000558471440×2791 255 KB

1 Like
91

I use the addon to track who is at home with my G Tag key fob. This works very well with Shelly and bluetooth proxy. When I restart Home assistant and I am not at home, the message not aviable always appears. Can’t I save the Mac address somewhere so that this doesn’t happen when I restart?

92

Hmm. Good question. HA’s thinking on this generally is that if we don’t “know” something, we set it to unknown or unavailable. But in our case, our only way of knowing something is “away” is simply not hearing from it, which logically is no different to “not heard from yet”.

I could change the logic so that “configured” devices default to being “away” at startup.

The downside to this is that automations based on away status would trigger at startup, as would automations set to trigger on transition from away to home, once a packet is detected.

Do devices stay stuck at “unknown” until you return home, or do they change to “away” after the configured away timeout? The latter is what it probably should do.

93

That makes sense. It would be good if he could remember the last status? The status remains unknown until I return home.

It would be best if I could enter the Mac address somewhere in the config so that the system would still recognize it when I restart and then set it to present if it is reachable or to absent if it is not reachable.

The problem now is that it no longer recognizes the Mac address after a restart. Or can I save it somewhere now? I may not have seen that.

94

Hmmm… that doesn’t sound right. Can you post some screenshots of what you are seeing and how it’s configured?

If you are getting actual Bermuda sensors for something, then it implies that it must either be set up in the configuration under Settings, Devices&Services, Bermuda, Configure, “Configured Devices” - or it must be a Private BLE Device in which case it will be saved in the config of that integration.

If it’s a random MAC address then it’s either sending iBeacon adverts (in which case it will be added via “Configured Devices” using the iBeacon UUID/Minor/Major, and/or via the Private BLE integration. Neither should require you to keep track of the MAC address, because it should just update the configured sensor whenever it knows something.

Unless of course you’re actually looking at a sensor provided by the iBeacon integration, in which case all of the above three posts can be ignored, and you need to add your thing via Bermuda, Configure, Configured Devices.

Some screenshots of what you’re seeing, and of your Bermuda configuration might be helpful if there’s any confusion

95

So bluetooth_le_tracker is completely separate from Bermuda, and you don’t need any of those entries in configuration.yaml for Bermuda to work (in fact, the poor state of bluetooth_le_tracker is one of the reasons Bermuda exists, that plus I knew it wouldn’t be a good fit for core for at least a while).

I can see you have the Devtracker Timeout in seconds to consider a device as "Not Home" set to (the default of?) 30 seconds. This is fairly short, and while I find that 180 seconds is pretty good, you could bump yours up as much as you need to (perhaps 4 minutes (240s) or more) to stop Bermuda from setting them as away as eagerly. Arrivals should still be instant, it’s only the “departure” that gets delayed.

In your dump_devices service output, you can see this block that shows the number of seconds between each received packet for that device (updated only every update_interval, so usually won’t show less than 1s):

     hist_interval:
        - 2.5520358320209198
        - 77.1560656990041
        - 2.591035165998619
        - 2.5540346179623157
        - 77.11702469200827
        - 2.5480331380385906
        - 2.5510331269470043
        - 3.8650500990333967
        - 73.26307638001163
        - 3.7490643459605053

The newest entry is at the top. So our last packet arrived 2.5 seconds after the previous one we saw, but before that it was over a minute (77 seconds) since we heard from that device. Then 2.5 seconds twice more, and 77 seconds again, and so on.

Bermuda gets this info by looking in Home Assistant’s internal cache of bluetooth packets, which (except for ones from usb bluetooth adaptors) will have timestamps on them supplied by either esphome or HA (not sure which part applies the stamps).

Given you are regularly getting 77 second gaps between packets, and you have the devtracker timeout in Bermuda set to the default 30 seconds, I think you’d probably find a big improvement by setting the timeout to at least 180, since “away” tends to not need to be as time-sensitive a status as “home”.

Is that likely to improve the situation for you?

96

Sorry for the delay in my response, combinations of real life and adhd mean I sometimes think I’ve already replied, or it slips my mind etc!

I plan for Bermuda to be an HA-native integration, so I don’t plan on doing any mqtt-specific things, but since they are just sensors on HA I expect they should all be available via the mqtt integration, if I understand correctly.

Re ideas for algorithmic improvements, there’s a few threads that are pretty much mandatory reading at this stage :slight_smile: I’ve tried to spell out the reasoning behind how I’ve set it up, so that people can evaluate that and provide ideas with the background knowledge on why it is currently how it is. I think most of the points you raise are well-covered in those, but let me know what you think after taking a read through:

Generally speaking, for Bermuda’s purposes, the maximum TX power is best, as it allows Bermuda to “hear” your devices from further away, maximising the opportunity to use multiple receivers (proxies) to make decisions about your device’s location.

If you have a fairly simple setup and just want to reduce the radius in which bermuda will detect the device, you can try lowering the power, but this is only a reasonable strategy when you have very few proxies. If you have more proxies the device should be “moved” to a closer area instead - but given the gaps in reception you are seeing it might be a reasonable option for you, at least until we have trilateration implemented.