Home Assistant Companion App BLE Presence Detection + RSSI

ESPHome
1

Hi,

A simple question. Maybe.

I’m running ESP Home on an ESP32 Dev Kit C, and the home assistant companion app on a Samsung S20. I’ve enabled the BLE beacon function and I’ve configured the esp32 to look for it as a binary sensor (ibeacon_uuid: ‘blah blah blah’). This is working and the sensor is on or off depending on the phone proximity…however…eventhough I move quite a long way away (I go upstairs) it can still be detected in the room downstairs.

My initial thought was to use the RSSI values I see in the log trace to filter readings (somehow, maybe set a minimum) but I can’t see how to get the RSSI. I can add a sensor for it using the ble_rssi platform…but unlike the ble_presence platform I can’t specify an ibeacon_uuid…only a service_uuid.

The companion app ble function only seems to work as an ibeacon_uuid, and listing it as a service_uuid for ble_rssi only results in a NaN return…so no dice.

What am I missing? I’ve googled for quite some time and I’ve come up empty handed except for suggestions to use another approach like room-assistant.

Sigh.

Thanks peeps!

CP.

2

I’m not sure but it may have to do with the last few lines of this where they mention rssi.

“After the ground work here is done, it should be relatively simple to mirror the changes to ble_rssi as well.”

You could try asking the devs on the ESPHome Discord server if you don’t get a response here.

3

For whatever reason, the ble_rssi component does not yet support iBeacons. I wrote a little workaround, that also adds filtering. Details here: ESPHome component for ESPresense-like room tracking

1 Like
4

So the workaround approach appeals…though to be fair I feels like I have no idea what I’m doing…so it’s probably true.

With my Studio Code addon I created custom_components folder under esphome, and a lib folder under that. I placed your ble_dist.h in custom_components and OneEuro under the lib sub folder.

I then tooled up a yaml that I thought might work:

esphome:
  name: example
  includes:
    - custom_components/lib/OneEuro.h
    - custom_components/ble_dist.h
  on_boot:
    then:
      lambda: |-
        addTracker("bt_s20_ble", "0d591b07-5d2c-4551-a8cc-39a38968e462");

esp32:
  board: esp32dev
  framework:
    type: arduino

api:
  
ota:
 
#Set WiFi
wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

logger:
  level: DEBUG
  logs:
    esp32_ble_tracker: INFO

esp32_ble_tracker:
  scan_parameters:
    duration: 60s
    active: false
  on_ble_advertise:
    then:
      - lambda: |-
          parseAdvertisement(x);

sensor:
  - platform: template
    name: "BT S20 BLE (Lounge)"
    id: lounge_bt_s20_ble_dist
    update_interval: 30s
    unit_of_measurement: "ft"
    device_class: ""
    state_class: "measurement"
    accuracy_decimals: 1
    entity_category: "diagnostic"
    lambda: |-
      return getTracker("bt_s20_ble").get_dist();

It is slightly different to your reference example…it just complained with that one that:

‘board’ is a required option for [ESP32]

And when I added that detail is wasn’t happy with the Platform: ESP32 bit…so…snipping done, edit window happy, lets install!

Then I get this error:

Compiling /data/example/.pioenvs/example/FrameworkArduino/esp32-hal-rmt.c.o
Compiling /data/example/.pioenvs/example/FrameworkArduino/esp32-hal-sigmadelta.c.o
Compiling /data/example/.pioenvs/example/FrameworkArduino/esp32-hal-spi.c.o
Compiling /data/example/.pioenvs/example/FrameworkArduino/esp32-hal-time.c.o
/data/cache/platformio/packages/framework-arduinoespressif32/cores/esp32/esp32-hal-spi.c: In function 'spiTransferBytesNL':
/data/cache/platformio/packages/framework-arduinoespressif32/cores/esp32/esp32-hal-spi.c:922:39: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
                 uint8_t * last_out8 = &result[c_longs-1];
                                       ^
/data/cache/platformio/packages/framework-arduinoespressif32/cores/esp32/esp32-hal-spi.c:923:40: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
                 uint8_t * last_data8 = &last_data;
                                        ^
Compiling /data/example/.pioenvs/example/FrameworkArduino/esp32-hal-timer.c.o
Compiling /data/example/.pioenvs/example/FrameworkArduino/esp32-hal-touch.c.o
Compiling /data/example/.pioenvs/example/FrameworkArduino/esp32-hal-uart.c.o
Compiling /data/example/.pioenvs/example/FrameworkArduino/libb64/cdecode.c.o
Compiling /data/example/.pioenvs/example/FrameworkArduino/libb64/cencode.c.o
Compiling /data/example/.pioenvs/example/FrameworkArduino/main.cpp.o
In file included from src/main.cpp:30:0:
src/ble_dist.h: In member function 'void BeaconTracker::update(int, int)':
src/ble_dist.h:61:14: error: 'to_string' is not a member of 'std'
         s += std::to_string(b) + ", ";
              ^
src/ble_dist.h:61:14: note: suggested alternatives:
In file included from src/esphome/core/preferences.h:6:0,
                 from src/esphome/core/application.h:6,
                 from src/esphome/components/api/api_connection.h:4,
                 from src/esphome.h:2,
                 from src/main.cpp:3:
src/esphome/core/helpers.h:398:20: note:   'esphome::to_string'
 inline std::string to_string(const std::string &val) { return val; }
                    ^
src/esphome/core/helpers.h:398:20: note:   'esphome::to_string'
In file included from src/esphome/components/sensor/sensor.h:3:0,
                 from src/esphome/core/application.h:16,
                 from src/esphome/components/api/api_connection.h:4,
                 from src/esphome.h:2,
                 from src/main.cpp:3:
src/ble_dist.h: In function 'BeaconTracker& getTracker(std::__cxx11::string)':
src/esphome/core/log.h:117:99: warning: format '%s' expects a matching 'char*' argument [-Wformat=]
   esp_log_printf_(ESPHOME_LOG_LEVEL_WARN, tag, __LINE__, ESPHOME_LOG_FORMAT(format), ##__VA_ARGS__)
                                                                                                   ^
src/esphome/core/log.h:151:28: note: in expansion of macro 'esph_log_w'
 #define ESP_LOGW(tag, ...) esph_log_w(tag, __VA_ARGS__)
                            ^
src/ble_dist.h:105:3: note: in expansion of macro 'ESP_LOGW'
   ESP_LOGW(TAG, "BeaconTracker %s not recognized, returning the first tracker (hopefully one exists)");
   ^
Compiling /data/example/.pioenvs/example/FrameworkArduino/stdlib_noniso.c.o
Compiling /data/example/.pioenvs/example/FrameworkArduino/wiring_pulse.c.o
Compiling /data/example/.pioenvs/example/FrameworkArduino/wiring_shift.c.o
*** [/data/example/.pioenvs/example/src/main.cpp.o] Error 1
========================== [FAILED] Took 4.33 seconds ==========================

I’m sure it is something simple…but I don’t know much about this code…

Help?

Thanks,
CP.

5

Ah, I guess it’s not compatible with the arduino framework as written. I actually have a ESP32C3 board which is much better supported on the esp-idf platform, so I haven’t tested on arduino.

Looks like there’s 3 issues.

  1. Some kind of SPI thing. It’s a warning and unrelated so ignore it I guess?
  2. no std::to_string. Try replacing all instances of std::to_string with esphome::to_string and if that doesn’t work try just to_string without a namespace.
  3. replace line 105 with ESP_LOGW(TAG, "BeaconTracker %s not recognized, returning the first tracker (hopefully one exists)", n.c_str());. The difference is the extra argument at the end. This is a silly mistake on my part, not sure how I didn’t catch it.
6

Oh I just noticed you have a C3 part as well. My esp32 and esphome sections read as:

esphome:
  name: env-2
  includes:
    - custom_components/lib/OneEuro.h
    - custom_components/ble_dist.h
  on_boot:
    then:
      lambda: |-
        addTracker("phone", "uuid-xxx-xxx");

esp32:
  board: esp32-c3-devkitm-1
  variant: esp32c3
  framework:
    type: esp-idf
    sdkconfig_options:
      CONFIG_BT_BLE_50_FEATURES_SUPPORTED: y
      CONFIG_BT_BLE_42_FEATURES_SUPPORTED: y
      CONFIG_BOOTLOADER_WDT_ENABLE: n
      CONFIG_BOOTLOADER_WDT_DISABLE_IN_USER_CODE: y
      CONFIG_ESP_INT_WDT: n
      CONFIG_ESP_TASK_WDT: n
      CONFIG_INT_WDT: n
      CONFIG_TASK_WDT: n

The various sdkconfig_options are workarounds for my setup, you may or may not need them too. If you’ve successfully uploaded a build with bluetooth without them, you can ignore them. More details here: Cannot use esp32_ble_tracker on ESP32C3 · Issue #2941 · esphome/issues · GitHub

7

OK…progress! But not totally…of course…

I did these things:

Replaced one instances of std::to_string with esphome::to_string
Replaced line 105 with `ESP_LOGW(TAG, “BeaconTracker %s not recognized, returning the first tracker (hopefully one exists)”, n.c_str());
Changed my framework to esp-idf
Changed line 53 to “dist_buf.insert(dist_buf.begin(), pow(10.0, (p-r)/(10.0 * PATH_LOSS)));” removing the conversion to ft (sorry, but SI units rule) ;-p

I have changed my yaml to:

substitutions:
  devicename: esp32study

esphome:
  name: $devicename
  includes:
    - custom_components/lib/OneEuro.h
    - custom_components/ble_dist.h
  on_boot:
    then:
      lambda: |-
        addTracker("study_bt_s20_ble", "0d591b07-5d2c-4551-a8cc-39a38968e462");

esp32:
  board: esp32dev
  framework:
    type: esp-idf
   
api:
  
ota:
 
#Set WiFi
wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password
  
  manual_ip:
    static_ip: 192.168.20.150
    gateway: 192.168.20.1
    subnet: 255.255.255.0
    dns1: 192.168.1.6

logger:
  level: DEBUG
  logs:
    esp32_ble_tracker: DEBUG


esp32_ble_tracker:
  scan_parameters:
    duration: 60s
    active: false
  on_ble_advertise:
    then:
      - lambda: |-
          parseAdvertisement(x);

sensor:
  - platform: template
    name: "BT S20 BLE (Study)"
    id: study_bt_s20_ble_dist
    update_interval: 30s
    unit_of_measurement: "ft"
    device_class: ""
    state_class: "measurement"
    accuracy_decimals: 1
    entity_category: "diagnostic"
    lambda: |-
      return getTracker("bt_s20_ble").get_dist();

it now compiles, uploads and boots. I can’t however detect my beacon…I’ve tried it as:

addTracker("study_bt_s20_ble", "0d591b07-5d2c-4551-a8cc-39a38968e462");
addTracker("study_bt_s20_ble", "0d591b07-5d2c-4551-a8cc-39a38968e462-100-1");
addTracker("study_bt_s20_ble", "iBeacon:0d591b07-5d2c-4551-a8cc-39a38968e462-100-1");

None allow it to be detected…yet it does work in that iBeacon format when I set up a sensor in HA for ESPresence (the other one I’m playing with). All I get back is:

[14:23:23][D][esp32_ble_tracker:214]: Starting scan...
[14:23:51][W][ble_dist:105]: BeaconTracker bt_s20_ble not recognized, returning the first tracker (hopefully one exists)
[14:23:51][D][sensor:125]: 'BT S20 BLE (Study)': Sending state nan ft with 1 decimals of accuracy

Is there a way to get it to show you what it does find? I suspect it just doesn’t like the formatting of my beacon id…

CP.

8

Ah damn, I totally forgot to document this. The iBeacon UUID decoding is byte reversed. Details here: https://github.com/esphome/issues/issues/2445

You can either follow my instructions there to modify the source in your esphome library (this will be overwritten when you upgrade esphome) or set the tracker UUID as the byte reversed version of the real UUID. for example A1B2-C3 becomes C3B2-A1.

9

Wow…that is a pretty major bug - “Oops - we accidentally turned it inside out!”

Still no joy on my side…reversed or not reversed I can never get past the whole “not recognised” issue for the UUID. I just cannot land the correct way to write in the string so it gets detected…

Is there a way to get it to return a lit of detected beacons?

10

Yeah, set the logging for the tracker component to very verbose, it’ll spit out details for every BLE advertisement it sees. Not every advertisement will be sent during a scan period, so it might take some time to catch the right one. According to the issue I linked, the UUID in the tracker logs will be reversed from what you should use.

As a note, the format should match the first example you gave, no major/minor numbers and no iBeacon: prefix.

Also, it looks like your get_tracker() call is using a different name than your add_tracker() call. It returns the first tracker by default so it shouldn’t make a difference with only one tracker,

11

Yup…just noticed I have a name mismatch…

[D][sensor:125]: 'BT S20 BLE (Study)': Sending state nan m with 1 decimals of accuracye

…that’s fixed now but still getting NaN…so still not pinning the beacon.

I’ll switch logging level and see what I find…back soon.

12

Now I’m seeing the UUID getting pciked up:

[13:12:03][VV][esp32_ble_tracker:544]: Adv data: 03.03.9F.FE.17.16.9F.FE.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00 (28)
[13:12:04][VV][esp32_ble_tracker:494]: Parse Result:
[13:12:04][VV][esp32_ble_tracker:511]:   Address: 6E:B2:09:94:71:33 (RANDOM)
[13:12:04][VV][esp32_ble_tracker:513]:   RSSI: -68
[13:12:04][VV][esp32_ble_tracker:514]:   Name: ''
[13:12:04][VV][esp32_ble_tracker:528]:   Manufacturer data: 02.15.0D.59.1B.07.5D.2C.45.51.A8.CC.39.A3.89.68.E4.62.00.64.00.01.C5 (23)
[13:12:04][VV][esp32_ble_tracker:531]:     iBeacon data:
[13:12:04][VV][esp32_ble_tracker:532]:       UUID: 62E46889-A339-CCA8-5145-2C5D071B590D
[13:12:04][VV][esp32_ble_tracker:533]:       Major: 100
[13:12:04][VV][esp32_ble_tracker:534]:       Minor: 1
[13:12:04][VV][esp32_ble_tracker:511]:   Address: 46:E2:37:12:13:42 (RANDOM)
[13:12:04][VV][esp32_ble_tracker:513]:   RSSI: -53
[13:12:04][VV][esp32_ble_tracker:514]:   Name: ''
[13:12:04][VV][esp32_ble_tracker:525]:   Service UUID: 0xFE9F
[13:12:04][VV][esp32_ble_tracker:539]:   Service data:

And with this in my yaml:

addTracker("study_bt_s20_ble", "62E46889-A339-CCA8-5145-2C5D071B590D");

I now get something:

[13:26:27][D][sensor:125]: 'BT S20 BLE (Study)': Sending state 3.94919 m with 1 decimals of accuracy
[13:26:27][VV][api.service:140]: send_sensor_state_response: SensorStateResponse {
  key: 3329046011
  state: 3.94919
  missing_state: NO
}

It says 3.9m…but it is more like 3.9cm as the phone is right next to the ESP32…but at least it is progress… Changing debug level now as it keeps losing connection/crashing…will play for a bit more…

13

Happy it works, thanks for working through debugging as my first beta tester lol.

You should put the phone 1m from the ESP32 and monitor the RSSI values printed to the log. Set the lowest repeated value you see as the TX power in the HA companion app, the default is probably quite a bit off depending on what power setting you’re using. Once that’s set, the distances should be more accurate, but the path loss constant is another knob to tweak if the reported distance doesn’t match actual distance once you move past 1m. The wikipedia page has some example values.

Once the TX power is calibrated, you can tweak the filter constants, number of retained samples, and timeout to get the behavior you want.

14

Nice! So I have done a bit of tinkering just with the RSSI readings…I put it exactly 1m away from the ESP32 and then I let it run for a while so I could get an average reading of the RSSI…which turned out to be about -78. I’ve updated the sensor config in the app and now I’m getting a reasonable value of 0.8m at 1m…which I think is going to be good enough given all the variables such as interference, positioning, and whether or not I’ve got the phone in my pocket or hands.

Ultimately I just want to set a distance range at which I would consider the beacon to be ‘in the room’ or ‘out of the room’ so it only need to be good enough to get a <5m type reading…plus once I have more of them I’ll take the ‘closest’ reading to zero in on the most likely location. So this is great now!

What is the syntax for adding additional trackers? Obviously I can simply duplicate the sensor block, but I’m not sure how to set the on_boot section…for example:

esphome:
  name: $devicename
  includes:
    - custom_components/lib/OneEuro.h
    - custom_components/ble_dist.h
  on_boot:
    then:
      lambda: |-
        addTracker("study_bt_s20_ble", "62E46889-A339-CCA8-5145-2C5D071B590D");
        addTracker("study_mp_s20_ble", "9584DD79-FFBE-589A-8B14-AAE3E3BE33E3");

Thanks!

15

Figured it out…use a comma, semicolon at the end. So it should be:

on_boot:
    then:
      lambda: |-
        addTracker("study_bt_s20_ble", "62E46889-A339-CCA8-5145-2C5D071B590D"),
        addTracker("study_mp_s20_ble", "9584DD79-FFBE-589A-8B14-AAE3E3BE33E3");

And that just works!

Yay. Time to soak test and she if it is stable.

16

That should work fine for multiple trackers, just make sure you pass the right name when calling get_tracker().

edit: huh that’s weird, lambdas should be straight C++ so a semicolon was correct (and what I have in my config) and a comma should error out.

17

I’ve been trialing Espresense lately and for some reason I’m getting very inaccurate distance readings.
I’ve tried different values in “Factor used to account for absorption, reflection, or diffraction” but still all over the place.

I’ve heard that esphome has problems with ble tracking when it comes to Android devices so I haven’t tried it.

Curious to see if anyone here can compare the 2?

18

As it happens I run both ESPresence and the custom ESPHome component from Raj (rpatel300 → ESPHome component for ESPresense-like room tracking). This is because I’m not ready to implement BLE room sensing yet (too busy) but I’ve got enough time to run a couple of naked ESP32s in two different rooms to see how they go…

My observations as follows:

  1. BLE sensing works pretty reliably, but distance measurements are a complete crap shoot. I think they’d work well under ideal conditions (an empty warehouse with the BLE beacon and the BLE sensor suspended from the ceilings using nylon fishing line), but in the real world there is too much scattering, interference, and absorption. The strength of the signal depends entirely on the orientation of the phone relative to the ESP32 (north, south, east, west), the material the phone is placed on (hard surface, soft mat), how many objects are around it or in the line-of-sight path to the ESP32 (e.g. books, tools, laptops, cup of coffee, etc), whether or not the phone cover is open of closed (mine is in a wallet case with cards in it), where I am in relation to phone (as a big hunk meat I will absorb signal). Even if I get it ‘perfect’ as soon as I move things around or conditions change it is all over the place again. The way I intend to use it in the long run is simply to find the ESP32 with the strongest signal and use that to indicate likely presence.
  2. The ESPHome solution works well and has been rock solid and stable since I set it up. I’m intending to clear a ‘room’ sensor for each room that will measure temperature, lux, humidity, CO2/VOx, and BLE so I will need to add a whole pile of extra modules to it when I come to actually use it in earnest. This may or may not affect stability…
  3. ESPresence is an awesome project that rolls together the BLE part with the option of additional sensors, all out of the box. There’s even some server-code that can use you BLE devices and do some triangulation, and it has automatic OTA…which is cool. But, this comes at the cost of not having much ability to control the configuration or behaviour, and recently there have been issues with boot loops, crashing and other spurious events which made the solution unworkable. These are resolved (for now) and my solution seems to be pretty rock solid again - the solution is under active development (it’s really active!) so it is likely future bugs may strike again. And, as per my ESP Home solution, the actual addition of sensors to the mix may impact stability again. So we will have to see…

At the moment I’m still very much on the fence. It is likely that I will continue to run two platforms all the way through the sensor build and then see if one stands out…I have NodeRed, so the wrangling of BLE data from multiple sources can be done on the back end…so the server-side code for ESPPresence it not a deal maker for me (in fact I’d have preferred a Node Red package).

A bit of a ramble…but hopefully you’ll find some value in it.

CP.

19

As CP said, distance is gonna be pretty rough because of environmental effects, not much you can do besides filtering the shit out of the raw signal and dealing with the resulting lag. The way the filtering works, it should pick up new presence fairly quickly and it’ll take a bit to drop off once you leave.

The default path loss value should be good enough, what you should tune is the 1 meter reference distance transmitted with the advertisements (this is described above). Both ESPresence and ESPHome are identical in this, they use the same BLE hardware and software stack to receive advertisements.

Personally, I want to use motion detectors and other instant read sensors to detect entering a space and then BLE to determine whether I’m still there even if I’m sitting still so motion won’t capture anything. I’m a little busy to set up anything more complicated than just BLE_presence = BLE_distance < threshold but eventually I’ll add a template sensor or Bayesian sensor or automation to do it this way.

The idea of triangulation with multiple nodes is cool, but setting up the coordinates of each node and room and tuning everything with noisy distance values feels like way too much work for so little gain.

20

I know this is slightly old but I’m trying to ge this working, I have everything set up right I think, I’m getting nan on the sensor so I guessing it’s just the device it’s looking for is wrong, but everything else is set up right?
I’ve tried my uuid forwards and backwards but neither seem to give me a reading, any additional help would be great

Edit: sorted it out, I had an error in my uuid. Seems to be working great now. Thanks

1 Like