ESPHome data gaps when using esp32_ble_tracker for BLE beacon device/sensor

meierjo · June 16, 2023, 6:17pm

Hi all -

I have a BLE Beacon device/sensor that throws out 3 sensor values every 0.5 seconds. I am getting the sensor data pretty reliably, but sometimes (at random) there’s a 6 - 10 second drop in data - no data is seen in the ESPhome logs. In the following log snippet, the sensor device is sitting within 6 inches (15.25 cm) of the ESP32 device, so signal attenuation shouldn’t be a problem.

Here’s the log snippet:

[22:08:42][V][sensor:059]: 'OnSemi BLE Temperature': Received new state 24.760000
[22:08:42][D][sensor:110]: 'OnSemi BLE Temperature': Sending state 76.56800 °F with 2 decimals of accuracy
[22:08:42][V][sensor:059]: 'OnSemi BLE Humidity': Received new state 41.490002
[22:08:42][D][sensor:110]: 'OnSemi BLE Humidity': Sending state 41.49000 % with 2 decimals of accuracy
[22:08:42][V][sensor:059]: 'OnSemi BLE Battery': Received new state 2881.000000
[22:08:42][D][sensor:110]: 'OnSemi BLE Battery': Sending state 2881.00000 mV with 1 decimals of accuracy
[22:08:42][V][component:204]: Component esp32_ble_tracker took a long time for an operation (0.06 s).
[22:08:42][V][component:205]: Components should block for at most 20-30ms.
[22:08:43][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:43][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:43][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:43][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:43][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:43][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:43][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:43][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:44][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:44][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:44][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:44][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:44][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:44][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:45][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:45][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:45][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:45][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:45][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:45][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:46][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:46][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:46][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:46][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:46][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:47][V][esp32_ble:179]: (BLE) gap_event_handler - 3
[22:08:47][V][sensor:059]: 'OnSemi BLE Temperature': Received new state 24.760000
[22:08:47][D][sensor:110]: 'OnSemi BLE Temperature': Sending state 76.56800 °F with 2 decimals of accuracy
[22:08:47][V][sensor:059]: 'OnSemi BLE Humidity': Received new state 41.500000
[22:08:47][D][sensor:110]: 'OnSemi BLE Humidity': Sending state 41.50000 % with 2 decimals of accuracy
[22:08:47][V][sensor:059]: 'OnSemi BLE Battery': Received new state 2870.000000
[22:08:47][D][sensor:110]: 'OnSemi BLE Battery': Sending state 2870.00000 mV with 1 decimals of accuracy

I usually see a lot of the “gap_event_handler - 3” log messages preceeding the data gaps. Wondering if it’s related, but I’m not finding much info on the net about the gap_event_handler message. Any ideas of where to look or investigte the data drop outs?

thanks

BlueCharmBeacons · June 16, 2023, 9:36pm

Maybe it is due to the interval and window settings you have configured in the ESP Home? The defaults leave lots of time when the ESP in not scanning.

BlueCharmBeacons · June 16, 2023, 9:57pm

Another idea just came to mind: What channel is the beacon/sensor advertising on during each broadcast? Some beacons rotate channels, so first broadcast is on channel 1, second broadcast on channel 2, third broadcast on channel 3, then back to 1…

The ESP is also rotating which channel it is scanning on, so during the first window, it scans channel 1, during second window it scans channel 2, third window it scans channel 3, then back to 1…

So with both the beacon and the scanner rotating channels, it might set up a sort of “rhythm” where every x period, the beacon will broadcasting on one channel but the ESP is listening on another channel. This might last some certain number of seconds until they get back in step.

Just speculation as I don’t know anything about your beacon/sensor.

In case you are curious, Blue Charm Beacons broadcast on all three channels during each broadcast. The broadcasts are squeezed up against each other and fire consecutively (not simultaneously, which would be impossible with this hardware). It might look like one single broadcast, but actually it is the same broadcast three times, one on each channel.

Here’s a cool chart showing how our beacon broadcasts one all three channels each time. “S” is the scanner, and “A” is the beacon in this example:

BlueCharmBeacons · June 16, 2023, 10:05pm

OK, last idea, I promise!

Maybe try adjusting the broadcast interval of the beacon/sensor to one of the oddball numbers that Apple recommends. You mentioned the current value is 0.5s, i.e. 500ms. You might get better results using 546.25ms or 417.5ms.

There’s a nice explanation about this stuff here: rf - BLE Scan interval and window - Electrical Engineering Stack Exchange

meierjo · June 16, 2023, 11:06pm

I have the window and the interval both set at 1000ms . I’d have to do more testing to confirm, but in the past, playing around with the window & interval didn’t help, so I set them the same for continuous scanning.

BlueCharmBeacons · June 16, 2023, 11:37pm

How is the data getting from the ESP to the HA? I assume either MQTT or something that goes via Wifi.

Why do I ask?

If you set window and interval to the same number, yes, you will have continuous scanning BUT you will also have a very busy ESP. Sometimes this overwhelms the little brain of the ESP and causes wifi reports to be unreliable (since the wifi and BLE share the same “radio”, and the BLE continuous scanning is dominating that radio).

Maybe try giving the ESP a little open time for wifi usage. So, for example, interval at 1000 and window at 800ms. Something like that. You need to play around with those numbers to find the best combination, and there doesn’t seem to be one “correct” answer. It depends on your usage cause and the sensitivities of the user.

Edit: It looks like the gap_event is generated by the ble.h file, so it seems as if the BLE is having trouble, not the wifi. I can’t find any info to define the “3” in the gap_event. Still, it could be the wifi and BLE conflict that is causing this.

meierjo · June 17, 2023, 12:34am

That’s a good idea - I have access to the sensor device firmware and can change the broadcast interval. I can also double check it’s broadcasting on all three channels (I believe it is, just need to comb the code!)

Thanks so much for the ideas!

Fallingaway24 · September 14, 2023, 3:07pm

hey, long time no speak. Just thought i’d share this with you. I was able to get battery data into HA from your beacons. I havn’t gotten any other data yet because the only service uuid being broadcast is for the battery or atleast thats my impression. I’m really just feeling my way around in the dark here so, i could be wrong. The issue I was having is in the esphome Docs it sais it needs a 16,32, or 128 bit uuid but in the logs the service uuid is none of those unless you strip off the 0x prefix of the service uuid. Doing that will give you a floating point number for a sensor is the same value that kbeacon is showing for my batteries so, i assume its good. I’m still not able to really use my beacons for anything that requires quick updates or high accuracy or the buttons on them lol.

This is from my esp log, the service uuid shows as 0x2080 which esphome throws an error about unless you use ‘2080’ which then returns my battery percent.
Parse Result:
[08:58:15][VV][esp32_ble_tracker:412]: Address: DD:34:02:07:E3:D8 (RANDOM)
[08:58:15][VV][esp32_ble_tracker:414]: RSSI: -84
[08:58:15][VV][esp32_ble_tracker:415]: Name: ‘BlueCharm_extra’
[08:58:15][VV][esp32_ble_tracker:423]: Ad Flag: 6
[08:58:15][VV][esp32_ble_tracker:429]: Manufacturer data: 02.15.77.77.77.2E.6B.6B.6D.63.6E.2E.63.6F.6D.23.23.23.0F.63.12.F0.C5 (23)
[08:58:15][VV][esp32_ble_tracker:432]: iBeacon data:
[08:58:15][VV][esp32_ble_tracker:433]: UUID: 2323236D-6F63-2E6E-636D-6B6B2E777777
[08:58:15][VV][esp32_ble_tracker:434]: Major: 3939
[08:58:15][VV][esp32_ble_tracker:435]: Minor: 4848
[08:58:15][VV][esp32_ble_tracker:436]: TXPower: -59
[08:58:15][VV][esp32_ble_tracker:440]: Service data:
[08:58:15][VV][esp32_ble_tracker:441]: UUID: 0x2080
[08:58:15][VV][esp32_ble_tracker:442]: Data: 64.04.0F.63.12.F0 (6)
[08:58:15][VV][esp32_ble_tracker:445]: Adv data: 02.01.06.1A.FF.4C.00.02.15.77.77.77.2E.6B.6B.6D.63.6E.2E.63.6F.6D.23.23.23.0F.63.12.F0.C5.09.16.80.20.64.04.0F.63.12.F0.10.09.42.6C.75.65.43.68.61.72.6D.5F.65.78.74.72.61 (57)
[08:58:15][V][sensor:043]: ‘BLE Sensor’: Received new state 100.000000
[08:58:16][D][sensor:094]: ‘BLE Sensor’: Sending state 100.00000 with 1 decimals of accuracy
[08:58:16][VV][api.service:140]: send_sensor_state_response: SensorStateResponse {
[08:58:16] key: 4034686637
[08:58:16] state: 100
[08:58:16] missing_state: NO

Fallingaway24 · September 14, 2023, 3:11pm

any idea why the uuid’s appear reversed(backwards)?
[08:58:15][VV][esp32_ble_tracker:433]: UUID: 2323236D-6F63-2E6E-636D-6B6B2E777777

My UUID is 777777E2B6B6-D636…
I assume this is an esphome/arduino thing but just curious.

BlueCharmBeacons · September 14, 2023, 10:43pm

Hi Justin,

I am not sure how you are doing that, but when I need to get battery power of our beacons using an ESP (using Arduino IDE), I use this code:

if (advertisedDevice->getServiceData(NimBLEUUID("2080")) != "") 
                        {
                          batteryPower = (advertisedDevice->getServiceData<uint8_t>(NimBLEUUID("2080")));
                          Serial.print(". Battery power is: ");
                          Serial.print(batteryPower);
                          Serial.print("%");
                        }

NimBLE is a free Arduino/ESP32 library for scanning beacons.

This code is doing a “scan request” and receiving a “scan response” back from the beacon with the battery data. This uses some of the beacon’s battery power, so you don’t want to do this too often. Once a day or week should be plenty.

One tricky thing to note, the data response from the beacon is in uint8_t format.

BlueCharmBeacons · September 14, 2023, 10:45pm

Yeah, this is an Arduino thing as far as I know. This previously caused me troubles when I was using Arduinos to do a raw BLE scan. Now I always use BLE scanning libraries (like NimBLE), and they handle this issue behind the scenes.

ESPHome is probably using a popular beacon scanning library from Neil Kolban. From the link below, this issue seems to have been fixed recently in the library, but I am not sure how long it will take for that change to work its way into ESPHome:

github.com/espressif/arduino-esp32

BLE on ESP32: BLEBeacon::getProximityUUID() returns reversed UUID string compared to expected value

opened 07:01PM - 16 Feb 23 UTC

closed 11:30AM - 20 Feb 23 UTC

johny-tee

Status: Solved Area: BLE

### Board ESP32 Dev Module ### Device Description ESP-WROOM-32 board. ### Ha…rdware Configuration No external connections except USB connection to the Host PC. ### Version v2.0.6 ### IDE Name Arduino IDE 2.0 ### Operating System Windows 10 ### Flash frequency 40 MHz ### PSRAM enabled yes ### Upload speed 921600 ### Description Calling **BLEBeacon::getProximityUUID()** after constructing BLEBeacon object **returns reversed UUID string** compared to what is expected. **The code:** ``` BLEBeacon oBeacon = BLEBeacon(); oBeacon.setData(strManufacturerData); // strManufacturerData is obtained via calling advertisedDevice.getManufacturerData() during BLE Scan System.println(oBeacon.getProximityUUID().toString().c_str()); ``` outputs: c1655d97-b7ac-d1b2-e940-23f200eb28d4 expected value is: d428eb00-f223-40e9-b2d1-acb7975d65c1 After analyzing the related code, the following was found in BLEBeacon.cpp: ``` BLEUUID BLEBeacon::getProximityUUID() { return BLEUUID(m_beaconData.proximityUUID, 16, **false**); } ``` The last parameter decides whether bytes should be reversed in memory in the implementation of BLEUUID. After modyfying getProximityUUID method to: ``` BLEUUID BLEBeacon::getProximityUUID() { return BLEUUID(m_beaconData.proximityUUID, 16, **true**); } ``` **The above code returns UUID value as expected:** outputs: d428eb00-f223-40e9-b2d1-acb7975d65c1 expected value is: d428eb00-f223-40e9-b2d1-acb7975d65c1 **Conclusion:** the last parameter in BLEUUID constructor call in BLEBeacon::getProximityUUID should be changed to true. ### Sketch ```cpp /* Based on Neil Kolban example for IDF: https://github.com/nkolban/esp32-snippets/blob/master/cpp_utils/tests/BLE%20Tests/SampleScan.cpp Ported to Arduino ESP32 by Evandro Copercini Changed to a beacon scanner to report iBeacon, EddystoneURL and EddystoneTLM beacons by beegee-tokyo */ #include <Arduino.h> #include <BLEDevice.h> #include <BLEUtils.h> #include <BLEScan.h> #include <BLEAdvertisedDevice.h> #include <BLEBeacon.h> #define ENDIAN_CHANGE_U16(x) ((((x)&0xFF00) >> 8) + (((x)&0xFF) << 8)) int scanTime = 5; //In seconds BLEScan *pBLEScan; class MyAdvertisedDeviceCallbacks : public BLEAdvertisedDeviceCallbacks { void onResult(BLEAdvertisedDevice advertisedDevice) { if (true == advertisedDevice.haveManufacturerData()) { std::string strManufacturerData = advertisedDevice.getManufacturerData(); uint8_t cManufacturerData[100]; strManufacturerData.copy((char *)cManufacturerData, strManufacturerData.length(), 0); int iRSSI = -9999; if (25 == strManufacturerData.length() && 0x4c == cManufacturerData[0] && 0x00 == cManufacturerData[1]) { if (advertisedDevice.haveRSSI()) { iRSSI = advertisedDevice.getRSSI(); } Serial.println("Found an iBeacon!"); Serial.println("Raw data:"); Serial.print(" "); for (int iCnt = 0; 25 > iCnt; iCnt++) { Serial.printf("[%02X]", cManufacturerData[iCnt]); } Serial.println(); BLEBeacon oBeacon = BLEBeacon(); oBeacon.setData(strManufacturerData); Serial.println("\nDecoded data:"); Serial.printf(" ID: %04X Major: %d Minor: %d UUID: %s Power: %d\n", oBeacon.getManufacturerId(), ENDIAN_CHANGE_U16(oBeacon.getMajor()), ENDIAN_CHANGE_U16(oBeacon.getMinor()), oBeacon.getProximityUUID().toString().c_str(), oBeacon.getSignalPower()); Serial.printf(" RSSI: %04d\n", iRSSI); } } return; } }; void setup() { Serial.begin(115200); BLEDevice::init(""); pBLEScan = BLEDevice::getScan(); //create new scan pBLEScan->setAdvertisedDeviceCallbacks(new MyAdvertisedDeviceCallbacks()); pBLEScan->setActiveScan(true); //active scan uses more power, but get results faster pBLEScan->setInterval(100); pBLEScan->setWindow(99); // less or equal setInterval value Serial.println("Configuration finished..."); } void loop() { Serial.println("******************************************************************************"); // Start scanning BLEScanResults foundDevices = pBLEScan->start(scanTime, false); // Print scan results summary Serial.printf("Devices found: %d total\n", foundDevices.getCount()); // Clear scanning results to release memory pBLEScan->clearResults(); Serial.println("Scan done!"); Serial.println("=============================================================================="); delay(10000); } ``` ### Debug Message ```plain No error messages. This is output of the snip. You can clearly see how UUID is reversed compared to raw data (UUID start on index 4 and ends on index 19 of Manufacturer data for iBeacon). ID, Major, Minor and Power values are as expected: Found an iBeacon! Raw data: [4C][00][02][15][D4][28][EB][00][F2][23][40][E9][B2][D1][AC][B7][97][5D][65][C1][00][01][00][10][BF] Decoded data: ID: 004C Major: 1 Minor: 16 UUID: c1655d97-b7ac-d1b2-e940-23f200eb28d4 Power: -65 ``` ### Other Steps to Reproduce Reproduced also on Linux. ### I have checked existing issues, online documentation and the Troubleshooting Guide - [X] I confirm I have checked existing issues, online documentation and Troubleshooting guide.

BlueCharmBeacons · September 14, 2023, 11:11pm

Re the button: As you have probably read, some of our beacons (like the BC021) have this nifty feature that allows you to use the beacon as normal, but also allow you to configure a certain button press (single, double, triple, or long press) to temporarily modify the broadcast uuid for a configurable amount of time. Sort of like having a second beacon.

So the beacon broadcasts its normal uuid, but then when you do the proper click on the button, the uuid will be slightly modified (the last few digits of the uuid are changed by a certain hex math amount; see the Quick Start Guide button trigger section on our website for more details).

Your scanner will see the regular uuid, and maybe do something or not according to your code, then it will suddenly see the changed uuid. It can then do something else according to your code setup. Let’s call this a “interchangeable uuid scenario”.

Another way to use this is to have the beacon broadcast nothing, but then broadcast the modified uuid when the button is clicked properly. Sort of like a beacon “remote control”. Let’s call this a “click generated single modified uuid scenario”. (catchy name!)

But here is the question: Can the ESPHome or ESPresence handle this??? (I don’t know the answer for sure)

-If the ESP is using only the MAC of the beacon as the unique identifier, then no, the button feature won’t work, since the MAC of the beacon never changes. The ESP is not even doing anything with the uuid. Your scanner can react to the beacon’s MAC only.

-If the ESP is using the MAC and the UUID of the beacon as a matched pair of identifiers, for example, it is looking for DD:34:02:07:E3:D8 and 2323236D-6F63-2E6E-636D-6B6B2E777777 as a pair of identifiers, then it might work.

“interchangeable uuid scenario”
You could set up two different automations, one to trigger on:
DD:34:02:07:E3:D8 and 2323236D-6F63-2E6E-636D-6B6B2E777777
and another to trigger on:
DD:34:02:07:E3:D8 and 2323236D-6F63-2E6E-636D-6B6B2E777788 (or whatever the actual changed uuid is)

Or

“click generated single modified uuid scenario”
You could set the beacon to do nothing until after the button is clicked properly, so the ESP automation would trigger on:
DD:34:02:07:E3:D8 and 2323236D-6F63-2E6E-636D-6B6B2E777788 (or whatever the actual changed uuid is)

You will need to test your specific ESP platform (Home or Presence) to see how it is handling the scanning of beacons and how you can trigger automations based on this.

Hope that makes sense. I have a phone call in a few minutes so I am rushing this. Ask questions as needed.

Fallingaway24 · September 15, 2023, 5:05pm

I’m doing it through esphome, that’s what i was trying to explain. I was only mentioning it because I’ve seen it asked and since you are the “spokesperson” and people come to you with questions, I was letting you know how i did it. I understand there are other ways like using Arduino and C++ but this isn’t the Arduino forum so it’s not very useful here. Likewise I could just look at the Kbeac![Screenshot from 2023-09-15on App to see the distance of a beacon and it’s battery state but that doesn’t help me incorporate that data into my home automation system and have all my relevant data in one place. HA uses Esphome to integrate esp boards and you need to use esphome/C++ syntax. This is how you do it with esphome.

esp32_ble_tracker:
  on_ble_service_data_advertise:
    - mac_address: DD:34:02:07:E3:D8
      service_uuid: '2080'
      then:
        - lambda: 'id(beacon1battery).publish_state(x[0]);'    
 

    - mac_address: DD:34:02:07:E1:37
      service_uuid: '2080'
      then:
        - lambda: 'id(beacon2battery).publish_state(x[0]);'

BlueCharmBeacons · September 15, 2023, 7:02pm

Oops, my bad! I didn’t read clearly enough. I thought you were still trying to figure it out.

Nice work!

Fallingaway24 · September 15, 2023, 7:34pm

sorry if that came off rude. My external HD that HA is on is making a grinding noise and i’ve been trying to restore a backup since yesterday and it’s driving me nuts. It was both really. I’m still trying to get some things working but also sharing what I figured out. The beacons are great though, I don’t have an issue with them. It’s how the other platforms handle the beacons. I’m wondering if it just wouldn’t be easier to have a second esp32 with my own Arduino program that handles things like button presses and polling the battery. Unfortunately, the person who maintains the bluetooth software for esphome isn’t very active and IMO seems to have lost interest in maintaining and updating BLE.

BlueCharmBeacons · September 15, 2023, 8:56pm

Not rude at all. No worries!

How is ESPHome in terms of triggering automations based on a beacon coming into and out of range?

Can you adjust the range accurately?

Is the reaction time fast, i.e. when the beacon achieves the defined range goal, does the automation trigger in 1-2 seconds or is it slower?

Does it trigger any false positive/negative automations?

Just curious since I have never worked with ESPHome before.