iBeacon Integration keeps adding Devices even if it is configured not to add new devices

Configuration
1

I have been using iBeacons (a few BC011) for “Home or Away” detection, the beacons once tuned work very well, for about 3-4 months now.

I have two ESPhome-based BLE Proxies and 3 beacons.

Last month I discovered the hard way that the integration has an “Enable newly added entities” switch that can be disabled, when I found about 150 unknown ibeacon devices that got silently added to my instance, I guess from people walking by (one of my BLE proxies is in my garage).

I disabled “adding new entities” in the ibeacon integration settings and I manually cleaned the registry (I had to manually delete entries from the entity registry, while HA was shut down, because manually deleting them from the GUI didn’t work (the delete operation errored out). After restarting HA, the devices where all gone however, I notice new ibeacon devices getting added daily (a few per day) together with their 5 entities, and these cannot be deleted from the GUI.

Also “discovery:” is not included in my configuration.yaml (so no autodiscovery)

One thing I noticed is that the manual removal I’ve been doing has been in the entity registry, I did not manually edit the device registry. However, just by deleting the entities, in the GUI the ibeacon device count went from 150 to the 3 I really have. I wonder if I also need to manually clean the device registry

Has anyone noticed this issue working with ibeacons, any hint on how to truly make it stop adding devices & entities?

2

This was infuriating to me as well, I live in a relatively dense urban environment so was getting tons of orphaned iBeacon devices. So I eventually removed the iBeacon Integration after following these steps.

On each of my ESPHome BLE Proxies, I added the following config:

binary_sensor:
  - platform: ble_presence
    mac_address: 0A:1B:2C:3D:4E:5F
    name: "Bike 1 on ${friendly_name} Beacon"    
  - platform: ble_presence
    mac_address: 1B:2C:3D:4E:5F:0A
    name: "CAR 1 on ${friendly_name} Beacon"    
  - platform: ble_presence
    mac_address: 2C:3D:4E:5F:0A:1B
    name: "CAR 2 on ${friendly_name} Beacon"
  - platform: ble_presence
    mac_address: 3D:4E:5F:0A:1B:2C
    name: "Bike 2 on ${friendly_name} Beacon"

with the MAC address of each iBeacon
(Also on each BLE proxy, I have to give me the substitutions and unique names)

substitutions:
  friendly_name: 'Bluetooth Proxy 1'

After installing each, now each device has 4 binary sensors, one for each of my iBeacons.

Next I created a template

      - name: "Bike 1 BLE is home"
        state: >
          {% if is_state('binary_sensor.bike_1_on_bluetooth_proxy_1_beacon','off') and is_state('binary_sensor.bike_1_on_bluetooth_proxy_2_beacon','off'') and is_state('binary_sensor.bike_1_on_bluetooth_proxy_3_beacon','off') and is_state('binary_sensor.bike_1_on_bluetooth_proxy_4_beacon','off') %}
            off
          {% else %}
            on
          {% endif %}
        icon: >
          {% if this.state == "on" %}
            mdi:bluetooth-connect
          {% else %}
            mdi:bluetooth-off
          {% endif %}

What this template does is only if all BLE Proxies detect that the sensor is off/away, does it mark the device as off/away

The delay is still about 5 minutes from when the device actually leaves the BLE Proxies and when it is marked as away, so no real loss in going from home to away. From away to home is pretty close to immediate.

Another helpful tool is using HACS, you can create a device tracker for the finaly binary sensor. GitHub - pnbruckner/ha-composite-tracker: Home Assistant Composite Device Tracker

3

Thanks Alex! this is a great alternative. I just finished cleaning up 150+ entries in the Device registry. If I notice that ibeacons keep getting added, I’ll shift gears to your approach.
I have the theory now that the devices getting added where the ones that were already in the Devices database that I hadn’t manually deleted before (I just manually deleted the entries). This is all indeed very annoying

4

Yup! I have had these Blue Charm Beacon devices installed via the iBeacon integration since 2022.11 and found that with the number of devices being added (while still having do not add new entities/devices set) that I was spending more time cleaning up iBeacon devices than was worth it. So I did a final purge, switched to this ESPhome method and haven’t had to clean a single one up since then. Much happier.

5

~24h check after the full entity & device cleanup, still no ghost beacon additions showing up. This is with HA 2023.2.4. I think last time it took a few days for the ghost beacons to start showing up, so I will keep checking every day or two.

6

Well the iBeacon integration continues to be badly broken in 2023.2.4, it’s confirmed. Half a dozen ghost devices and 40 entities appeared today. There is a github bug opened but I doubt it will get any traction, it seems this integrations
has a fundamental design flaw, assuming unknown beacons don’t pop up, which seems unrealistic in any urban setting.

@alex3m3ti8 I was wondering if it’s possible to get the the distance estimate with your approach? I have a distance-based automation for one of my beacons that works reasonably well

7

I saw the two github bugs and yeah, not sure they will get that much attention.

ESPHome bluetooth proxy has no way to disable discovery · Issue #88129 · home-assistant/core (github.com)

ibeacon keeps adding New Devices even if it’s configured not to do so · Issue #88111 · home-assistant/core (github.com)

@ariel, when you are asking for the distance estimate, can you expand on what you are looking for so I answer you correctly? Let me try and answer and see. I will say that when I had the iBeacon Tracker integration loaded, I didn’t pay much mind to the estimated distance. Primarily because I didn’t find the distance to be very accurate or reliable.

My car or bike (as well as my ESPHome devices) could be stationary for a whole day, and I would get 200 different distances with a high amount of variability. In my experience, the present or not present/home or away was much more reliable, so the template that I am using above relies solely upon that. With a decent spread of ESPHome Devices, (I have five strategically laid out around the house and my lot) I am able to very reliably determine when one of the beacons leaves or returns.

So I am not really using distance as a metric in any way. I also looked at trying to use Source Attribute (which will say which device it is connected to), but this switched recently from the friendly name of the ESPHome device to the MAC address, which turns out to not be as useful for me.

Hope that answers your inquiry, if not, let me know where I can expand.

8

Thanks Alex! yes I was referring to the “distance estimate”. It is true that it has a lot of random variability, but I found that if I use it ‘filtered’ with a moving average sensor, it works reliably enough for me to tell whether the garbage bin is outside in the curb or not (a difference of some 12-13 meters between one state and the other), so on garbage day I can get an alert if we forgot to get the bin out.

Do you know if ESPhome gets the beacon distance as an attribute that can be exposed to HA? I use bluecharm beacons and I noticed they have this parameter “RSSI @1m” which I’m guessing could be used to derive a distance estimate if ESPHome could expose measured RSSI. I found this formula that uses RSSI@1m and measured RSSI and gives a distance estimation:

That I guess I could implement that in HA with a template sensor, if HA knew the instant RSSI

9

Yes, Ariel, that’s exactly correct. That’s what the fixed “rssi @1 meter” is supposed to be used for. I would guess that ESPHome is using that in their calculation.

Also your approach to use a moving average of rssi live figures is great also.

One other additional thing that I use when coding ESPs is a “remove whacky numbers” prefilter. Basically, the code looks at the live rssi and if the latest reading is more than 10 above or below the previous reading, the code will then adjust the current reading to only 5 more of less than the previous reading. This prevents occasional whacky readings from dragging your moving average too far away from reality. I’m not sure though how to do this in HA.

10

Yup, all good suggestions to use the Filter Sensor, which I considered, but ultimately that level of precision didn’t really provide a use case for me (though for you, garbage bins and proximity is a cool use of them).

What others said about RSSI seems right based on my review of the stat, but honestly, I just couldn’t come up with good use cases in my scenario.

11

True. It’s definitely overkill for Ariel to get too technical with fine degrees of RSSI. Trash cans out or in…is pretty broad. :joy:

My suggestion about a pre-filter (if that is even possible with HA) is more appropriate for people who want things like automatically turning on/off lights or opening/closing a garage door. You really want those actions to be accurate and fast, especially on the ON/Open side of the automation. Standing in a dark room for 2 seconds feels like hours!

I’ve played around with the iBeacon Tracker implementation and gotten good results for “arriving” (within 1-2 seconds; plus you can tune it to trigger before you enter the room) but “leaving” automations take up to 2 mins. So that probably needs some work.

I’ve heard good things about ESPresence and ESPHome though; lots of our beacon users seem to be happy with their results. I haven’t had time to play around with them yet.

12

Hi Thomas, thanks for chiming in. Yes, I’d agree with the setup that I’ve done, specific binary sensors in ESPHome YAML, and Templates, your Blue Charms Beacons are doing exactly what I want them to do and with the removal of the native iBeacon Integration I am very happy with the results. And my OCD has chilled its pants too now that I’m not angrily deleting 50 dead iBeacon devices a week.

13

Thanks folks for all the pointers.

Researching and experimenting a bit with ESPhome, I could get the RSSI of each beacon exposed to home assistant, extending Alex idea like so:

# BLE beacon detection to avoid having to use the ibeacon integration
binary_sensor:
  - platform: ble_presence
    mac_address: 0A:1B:2C:3D:4E:5F
    name: "Beacon Car presence on ${friendly_name}"
  - platform: ble_presence
    mac_address: 0B:1B:2C:3D:4E:5F
    name: "Beacon Garbage Bin presence on ${friendly_name}"
sensor:
  - platform: ble_rssi
    mac_address: 0A:1B:2C:3D:4E:5F
    name: "Beacon Car RSSI on ${friendly_name}"
  - platform: ble_rssi
    mac_address: 0B:1B:2C:3D:4E:5F
    name: "Beacon Garbage Bin RSSI on ${friendly_name}"

image

Later this week I’ll add the filters to the rssi for the bin sensor, transform rssi to distance and remove the ibeacon integration. Which in the last 48h added 5 ghost beacons, so annoying.

@BlueCharmBeacons, in the formula: Distance = 10^((Measured Power - Instant RSSI)/10*N), I can get Measured Power from each beacon (RSSI@1m), can you share what would be the N factor that you recommend using? In the example they use 2.4 but they don’t explain why.

1 Like
14

The N factor is just a handy “adjustable knob” for tuning the system for different locations. In some locations, you might set it at 2.0 and get accurate distances. In a different location with the exact same ESP and beacon, you might set it at 2.4 and get accurate distances. So probably start at their suggested 2.4.

On the other hand, all this is doing is converting RSSI numbers to meters. So usually I don’t bother with a formula but instead just use the RSSI numbers directly. For turning on the lights in my office, I might set the target RSSI for ON to >-56. Turning OFF set to <-65 (using the smoothed rssi values mentioned earlier). I would arrive at these figures by testing it out and seeing how early I want to trigger the lights on; in this case, I want them to turn on when I am about 2 meters away from the office door walking at a normal speed. For my office, this looks like >-56.

For my garage door, it makes more sense to open the door at >-79 (I want it to be “easy” to open because I am using a usb-powered beacon plugged into the car’s usb power outlet; as soon as I turn the ignition key, the beacon starts broadcasting, and the door immediately opens) and close at <-85.

1 Like
15

One more thing I will say regarding the template that I created. Note that each template is built on if the state of the binary_sensor is off. This was a deliberate choice on my part that ONLY if all states are actually OFF does the cumulative template binary sensor go off.

This means that if one of your bluetooth proxies is offline (for whatever reason), the state of all iBeacons on that particular Bluetooth proxy are Unknown/Unavailable not Off. This means that you could be in a situation where if you offlined a bluetooth proxy for a valid reason, your cumulative template will never report as away until that bluetooth proxy comes back online and is reporting state of the iBeacon trackers again.

Another way you can do this is to slightly reverse the logic and do if the state is not ‘on’. This would include state off OR unknown/unavailable. Very subtle difference, but might be more appropriate for you.

16

Here’s a better version (still under test but looking good) of the relevant ESPhome configuration.

I had to add a “delayed_off” filter to the beacon presence sensor, one of my 3 beacons had a weird behavior where it would go ‘off’ suddenly and then back ‘on’ a second later, the delay fixed that.

I also added the moving average directly to the rssi sensor (esphome is quite awesome), and lastly I also added the distance estimator using a lambda/C++ function (esphome is amazing). I got the “measured_power” constant from the beacon app, and I totally made up the n_factor so that the distance measured under normal conditions looks OK.

In the meantime the ibeacon integration kept discovering passers-by, 18 in 3-4 days - until I removed

# BLE beacon detection to avoid having to use the ibeacon integration
binary_sensor:
  - platform: ble_presence
    mac_address: ${mac_car}
    name: "Beacon Car presence on ${node_name}"
    device_class: presence
    filters:
      delayed_off: 2 min
  - platform: ble_presence
    mac_address: ${mac_bin}
    name: "Beacon Garbage Bin presence on ${node_name}"
    device_class: presence
    filters:
      delayed_off: 2 min
sensor:
  - platform: ble_rssi
    mac_address: ${mac_car}
    name: "Beacon Car RSSI on ${node_name}"
    id: rssi_car
    filters:
      - sliding_window_moving_average:
          window_size: 10
          send_every: 10
  - platform: ble_rssi
    mac_address: ${mac_bin}
    name: "Beacon Garbage Bin RSSI on ${node_name}"
    id: rssi_bin
    filters:
      - sliding_window_moving_average:
          window_size: 10
          send_every: 10
  - platform: template
    name: "Beacon Car estimated distance on ${node_name}"
    unit_of_measurement: "m"
    lambda: |-
      const float measured_power = -67.0;  /* rssi@1m of beacon */
      const float n_factor = 0.3;
      return ( powf(10 , ( (measured_power - id(rssi_car).state) / 10.0) * n_factor) );
    accuracy_decimals: 1
    update_interval: 30s
  - platform: template
    name: "Beacon Garbage Bin estimated distance on ${node_name}"
    unit_of_measurement: "m"
    lambda: |-
      const float measured_power = -67.0;  /* rssi@1m of beacon */
      const float n_factor = 0.5;
      return ( powf(10 , ( (measured_power - id(rssi_bin).state) / 10.0) * n_factor) );
    accuracy_decimals: 1
    update_interval: 30s
1 Like
17

Hi, not sure if you all who have this configuration have seen this problem since 2023.7.0 and ESPHome 2023.6.3, but looks like if you have a

binary_sensor:
  - platform: ble_presence

AND

bluetooth_proxy:
  active: true

They conflict and the ble_presence sensor will go off in 5 to 10 minutes. It will stay off even if it is nearby and won’t turn back on until the ESPHome Device is rebooted.

Current resolution (apart from reverting back) is to add to your ESPHome yamls.

external_components:
  - source: github://pr#5073
    components: [ esp32_ble_tracker ]

if you can to resolve this issue. This puts the code which is currently in development into your device.

See Xiaomi LYWSDCGQ and CGDK2 no longer report Temerature and Humidty. · Issue #4641 · esphome/issues (github.com)

18

Same questions on those foreign/ghost devices here:

I vote for a “do not automatically add devices to the integration” system option. Not having entities enabled is one (already good) thing, but the list of devices still explodes. Other platforms like OpenMQTTGateway (OMG) can even disable auto discovery for the ESP itself, right from HA. I miss that for my ESPHome based Bluetooth Proxy a lot - which would also block iBeacon Tracker collecting devices like crazy.

grafik

1 Like