ESPHome BLE Gateway and other BLE components

Share your Projects!
102

This project has been through a few iterations and congrats on getting a big part of it included in the home assistant code.

Having said that, I get confused about what has changed and I need a quick confirmation of where I am going.

I have multiple Xiaomi sensors which were working fine through the ESPHome setup.
But I use them outside and tire of the occasional replacement and waiting 14 days for delivery from Alibabba.
So looking at Thermal Pro. But I digress.

Anyway, it seems I can use the “Passive BLE Monitor” integration without having to throw a dongle on my FreeBSD-hosted VM and continue to use the ESPHome as the relay.

Where I am confused is if I still need to integrate the BLE Gateway or if that is the component that has been incorporated into the core system.

103

Hello,

You should use this code instead (before you try code for ESPHome 2021.12 version or earlyer and code for local component):

external_components:
  - source: github://myhomeiot/esphome-components
2 Likes
104

Hello,

Only the logic was incorporated into core system, the core system Bluetooth proxy can do much more. If you need to use Passive BLE Monitor than you should use BLE Gateway. Fortunately Bluetooth proxy and BLE Gateway can coexists on same ESPHome device.

105

Ahhhh! Thank you! I’ve tried the first time to leave out:

source: github://pr#2854

but I kept also

components: [esp32_ble_tracker]

so it gave me an error. I didn’t think about leaving out that as well! :man_facepalming:
Now I’m able to build it correctly!

...
Linking /data/atom-bluetooth-proxy-855f10/.pioenvs/atom-bluetooth-proxy-855f10/firmware.elf
RAM:   [==        ]  16.4% (used 53632 bytes from 327680 bytes)
Flash: [=======   ]  68.8% (used 1262669 bytes from 1835008 bytes)
Building /data/atom-bluetooth-proxy-855f10/.pioenvs/atom-bluetooth-proxy-855f10/firmware.bin
Creating esp32 image...
Successfully created esp32 image.
esp32_create_combined_bin(["/data/atom-bluetooth-proxy-855f10/.pioenvs/atom-bluetooth-proxy-855f10/firmware.bin"], ["/data/atom-bluetooth-proxy-855f10/.pioenvs/atom-bluetooth-proxy-855f10/firmware.elf"])
Wrote 0x145ac0 bytes to file /data/atom-bluetooth-proxy-855f10/.pioenvs/atom-bluetooth-proxy-855f10/firmware-factory.bin, ready to flash to offset 0x0
======================== [SUCCESS] Took 168.06 seconds ========================

********************************************************************************
We found 591.19MB of unnecessary PlatformIO system data (temporary files, unnecessary packages, etc.).
Use `pio system prune --dry-run` to list them or `pio system prune` to save disk space.
INFO Successfully compiled program.
INFO Resolving IP address of atom-bluetooth-proxy-855f10.local
INFO  -> 192.168.10.109
INFO Uploading /data/atom-bluetooth-proxy-855f10/.pioenvs/atom-bluetooth-proxy-855f10/firmware.bin (1268416 bytes)
Uploading: [============================================================] 100% Done...


INFO Waiting for result...
INFO OTA successful
INFO Successfully uploaded program.
INFO Starting log output from atom-bluetooth-proxy-855f10.local using esphome API

Now in the log I can see the packets picked up by the Atom! :smiley:

106

Well, do I need to use Passive BLE Monitor?
I do not want to add a BLE dongle to the server. It sounded like I could use BLE Gateway on the ESP device to send the BLE data onto the home assistant server. And maybe my confusion is what component then parses the ble data there?
I thought Passive BLE Monitor could do that, but I get a warning that “There are no BT controllers present in the system!”

107

If all your devices supported by Home Assistant Core Bluetooth integration than you can use it with ESPHome Bluetooth Proxy. If some devices not supported but has support in Passive BLE Monitor than you can use it with BLE Gateway for send BLE advertaisement packet from ESP instead having dongle. In Passive BLE Monitor you can disable looking for BT controllers by changing bt_interface parameter to disable.

2 Likes
108

==============================================

UPDATE: I took out a new ESP32, used the BLU Proxies site ESPHome Bluetooth Proxy installed it, ran it and everything just showed up as device integrations
Thanks for all the work here, I learned quite a bit I think.

============================================================

I have successfully added several sensors and moved my Xiaomi sensor processing up into HA thanks to your work.

I have one waterproof sensor made by SwitchBot that is not supported by Passive BLE, but IS supported by HA BLE and has a SwitchBot integration there.

However, that requires a BLE adaptor on the HA server, which seems very impractical to me and I am trying to use ESPHome as I do with all the other BLE sensors I am using.

Is there some way to install the messaging parsing used by the HA SwitchBot integration?
Or do I need to request the sensor be added to the Passive BLE parsing?

I believe I do have it hooked up correctly as I see the packets being harvested in ESP32:
D2:A3:1C:F0:4C:CC is the MAC of the sensor:

03:35:27 [D] [ble_gateway:063] [D2:A3:1C:F0:4C:CC] Packet 043E2602010001CC4CF01CA3D21A0201060FFF6909D2A31CF04CCC9F0F069A250006163DFD7700E4CA
03:35:27 [D] [ble_gateway:063] [D2:A3:1C:F0:4C:CC] Packet 043E2602010001CC4CF01CA3D21A0201060FFF6909D2A31CF04CCC9F0F069A250006163DFD7700E4CA
03:35:29 [D] [ble_gateway:063] [D2:A3:1C:F0:4C:CC] Packet 043E2602010001CC4CF01CA3D21A0201060FFF6909D2A31CF04CCC9F0F069A250006163DFD7700E4CD

And the HA logs show it being received there:
2023-08-12 02:44:04.130 INFO (MainThread) [custom_components.ble_monitor.ble_parser.switchbot] BLE ADV from UNKNOWN Switchbot DEVICE: RSSI: -50, MAC: D2:A3:1C:F0:4C:CC, ADV: 06163dfd7700e4

2023-08-12 02:44:12.136 INFO (MainThread) [custom_components.ble_monitor.ble_parser.switchbot] BLE ADV from UNKNOWN Switchbot DEVICE: RSSI: -54, MAC: D2:A3:1C:F0:4C:CC, ADV: 06163dfd7700e4

2023-08-12 02:44:42.220 INFO (MainThread) [custom_components.ble_monitor.ble_parser.switchbot] BLE ADV from UNKNOWN Switchbot DEVICE: RSSI: -54, MAC: D2:A3:1C:F0:4C:CC, ADV: 06163dfd7700e4

110

If waterproof sensor made by SwitchBot supported by HA BLE than you can use any/all your ESPHome devices as external BLE adapters, just add flowing lines into configuration:

bluetooth_proxy:

If your BLE device requires send commands to if from HA, you need to enable active connections by:

bluetooth_proxy:
  active: true

bluetooth_proxy it’s buildin component of ESPHome and you don’t need to use ble_gateway. ble_gateway works for Passive BLE Monitor, bluetooth_proxy works for HA BLE.
More info you can find here.

If your BLE device requires active connection it’s can’t be added to Passive BLE Monitor, if it’s doesn’t requires active connections and doesn’t supported yet by Passive BLE Monitor than you can ask for device support in Passive BLE Monitor github.

1 Like
111

Hey, thanks for great component. I just wanted to share my trick about service usage rather than eve t based automation.

Instead of generating an event, i am calling the respective home assistant service from esp directly which had decreased the load on server side a lot.

Contrary to what you suggested, i had setup discovery as true for all my esp devices and home assistant get clogged by the time, especially the frontend which listens to events a lot.

So, either go with manual discovery management or keep it on with relying homeassistant service calls.

112

Nice finding, thanks for information!

From first version of integration I mention this way in the doc, but suggest to use event instead.
Now you confirm that the service call decreases the load and can be used for heavy loaded BLE environment with enabled discovery (BLE messages from all devices goes to Passive BLE Monitor).

1 Like
113

Hello!

I’m ussing Passive BLE integration to retrieve data from 8 Xiaomi LYWSD03MMC temperature sensors (stock firmware) in RPi 4 integrated BLE, working with Hass OS. Few days ago I tried to improve reception of some sensors with a ESPHome BLE Gateway. It’s working well now but I’m confussed about PASSIVE OR ACTIVE SCAN. The ESP used code in first intance was:

esphome:
  name: blegateway
  friendly_name: blegateway

esp32:
  board: esp32dev
  framework:
    type: arduino

# Enable logging

logger:

# Enable Home Assistant API
api:
  encryption:
    key: "xxx"

ota:
  password: "xxx"

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Blegateway Fallback Hotspot"
    password: "xxx"

captive_portal:

esp32_ble_tracker:

external_components:
  - source: github://myhomeiot/esphome-components

ble_gateway:
  id: blegateway

  devices:
    - mac_address: !secret Estudi_mac
    - mac_address: !secret Golfes(Est)_mac
    - mac_address: !secret Golfes(Oest)_mac
    - mac_address: !secret HabGran_mac
    - mac_address: !secret HabNenes_mac
    - mac_address: !secret Menjador_mac
    - mac_address: !secret TerrassaNord_mac
    - mac_address: !secret TerrassaSud_mac

  on_ble_advertise:
    then:
      homeassistant.service:
        service: ble_monitor.parse_data
        data:
          packet: !lambda return packet;
          gateway_id: $gateway_id

After that the log of the ESP BLE Tracker notify that is using a ACTIVE CONNECTION:

12:08:04][C][esp32_ble_tracker:621]:   Scan Type: ACTIVE

Reading the specifications of esp32_ble_tracker shows that the scan type is ACTIVE y default (https://esphome.io/components/esp32_ble_tracker.html). Why the BLE Gateway code example is with this parameter as active? And does it represents that the BLE Gateway is eating more battery of sensors ?

I tried to recompile ESP with the option PASSIVE SCAN:

esp32_ble_tracker:
  scan_parameters:
    active: False

The ESP log:

[12:42:38][C][esp32_ble_tracker:617]: BLE Tracker:
[12:42:38][C][esp32_ble_tracker:618]:   Scan Duration: 300 s
[12:42:38][C][esp32_ble_tracker:619]:   Scan Interval: 320.0 ms
[12:42:38][C][esp32_ble_tracker:620]:   Scan Window: 30.0 ms
[12:42:38][C][esp32_ble_tracker:621]:   Scan Type: PASSIVE
[12:42:38][C][esp32_ble_tracker:622]:   Continuous Scanning: True

I don’t see any change, I continue receiving a lot of data from my sensors. I can see this in ESP log and in history data in HA, that is recording values every 5 minuts now. Before the modifications introduced by BLE Gateway, the history data only record every time that the integrated BLE received a passive transmission. I have Passive option in Passive BLE Component in HA also selected.

How I can corfim that the integrated Rpi4 BLE and the ESP is working in PASSIVE MODE ? And why the history in HA is recording data every 5 minuts now ? The LYWSD03MMC sensors default update interval time is much more than 5 minuts (10 minutes I think)


Thank you!

114

Hello,

BLE Gateway can work with any configuration of esp32_ble_tracker (some users adjust interval, window, and duration parameters). Usually active scan requires more device powers, and can bring more data from device, but LYWSD03MMC doesn’t requires active scan, so to disable it was a good idea. Active scan can’t change how many BLE advertisement send by device it’s can get more data from sent advertisement.

AFAIK LYWSD03MMC with stock firmware send data when it’s changes or every 10 minutes. You can check how many BLE advertisements send from one LYWSD03MMC device by checking the ESPHome log. To make this simpler, leave only one LYWSD03MMC device in BLE gateway configuration.

115

Active scan is asking close by devices to advertise new data rather than waiting then self advertise. So it asks but doesn’t wait for the reply, just processes incoming ble data.

116

Debouncing spammy devices

Thank you for this amazing tool!
In the past I was using it to forward BLE advertisements from my 2 Hörmann garage door openers to Passive BLE monitor which worked like a charm.

Now i wanted to add a LYWSD03MMC and noticed that the bluetooth of my ESP32 can’t keep up with the high frequency of advertisements and dropping some/many of them. This is mostly due to the Hörmann devices being very spammy and basically broadcasting every other ms. In the past this was no issue because there were enough advertisements from the door openers going through for HA to update the sensors. However now I am at a point where I can not reliably include the Xiaomi because the single advertisements from it get dropped so easily due to the spam.

I played around with scan interval and -window however the amount of advertisements coming from the Hörmanns is insane.
Is there any way to debounce those or any idea how I can have both device types on a single ESP32-gateway?

117

I have also Horrman garage door, you are right about spamming but not sure if I am dropping any messages.

Are you using service call or event generation?

118

I am using the service call provided by the Passive BLE Monitor-Integration.

Do you have other BLE devices on the same gateway/ESP32 as the Hörmann?

Of course I could increase the advertising rate of the Xiaomis and hope that more of them make it through consistently and thus living with the decreased battery life. However I was hoping for a more elegant solution.

On the other hand the pvvx repo even says that dropped advertisements are to be expected when using an ESP32 with ESPHome for many (or therefore also few very spammy) devices.

119

Please can you share the exact service name you are calling?

I have couple other sensors nearby garage door but I am not processing them.

120

Yes of course.

Here is the relevant YAML from my ESPHome device (ESP32 D1 Mini):

# Enable ble tracker
esp32_ble_tracker:
  scan_parameters:
    interval: 200ms
    window: 100ms
    active: False

# Config BLE Gateway
ble_gateway:
  id: blegateway
  devices:
    # Hörmann-Antriebe
    - mac_address: !secret garage_links
    - mac_address: !secret garage_rechts
    # Xiaomi Hygrometer
    - mac_address: !secret hygrometer_wohnzimmer_mac
  on_ble_advertise:
    then:
      homeassistant.service:
        service: ble_monitor.parse_data
        data:
          packet: !lambda return packet;
121

I personally followed the advice here and didn’t play with interval and window values.

122

I did not check the BT proxy documentation before. Thank you for pointing that out.

1 Like