ESPHome BLE Gateway and other BLE components

ESPHome BLE Gateway and other BLE components.

Please :star: this repo if you find it useful.

To use this repository you should confugure it inside your yaml-configuration:

external_components:
  - source: github://myhomeiot/esphome-components

or download component into custom_components folder (you can use another name) and add following lines to your yaml-configuration:

external_components:
  - source: custom_components

You can take a look at samples of usage of those components in examples folder.

BLE Client

BLE Client allow to read characteristics from devices.
Difference from build-in ESPHome BLE Client:

  • Always disconnects from device after reading characteristic, this will allow to save device battery. You can specify update_interval, defaults to 60min.
  • Uses lambda for parsing and extracting data into specific sensors make this component very flexible and useful for prototyping.
  • There is no limit to the number of BLE Clients used (build-in BLE Client has limit of 3 instances). This component uses BLE Host component which you should count as one instance of build-in BLE Client. All BLE clients are processed sequentially inside the host component at time when they was detected and update interval reached.

BLE Host

Used by BLE Client component.

BLE Gateway

BLE Gateway component will allow you to forward BLE Advertising data packets for external processing to Home Assistant or other systems.

If the heart of your Home Automation system is Home Assistant or another similar system and you use ESPHome devices to extend BLE coverage and process data from BLE sensors, you can dramatically decrease system complexity by remove all BLE data processing from ESPHome devices and forward raw BLE Advertising data to external components like Passive BLE Monitor.

Requirements:

If you use ESPHome 2021.12 version or earlyer you need to make following changes in ESPHome esp32_ble_tracker component.
In order to apply this PR you can use following ESPHome configuration (requires ESPHome 2021.11 or later):

external_components:
  - source: github://myhomeiot/esphome-components
  - source: github://pr#2854
    components: [esp32_ble_tracker]

ESPHome configuration example

Note: This example use event, you can use direct ble_monitor.parse_data service call

ble_gateway:
  devices:
    - mac_address: 01:23:45:67:89:AB
    - mac_address: !secret lywsd03mmc_mac
  on_ble_advertise:
    then:
      homeassistant.event:
        event: esphome.on_ble_advertise
        data:
          packet: !lambda return packet;

Home Assistant Passive BLE Monitor configuration example

Note: Remove automation if you use direct ble_monitor.parse_data service call

ble_monitor:
  discovery: false
  restore_state: true
  decimals: 1
  period: 300
  devices:
    - name: Living Room Thermo
      mac: 01:23:45:67:89:AB
    - name: Bedroom Thermo
      mac: !secret lywsd03mmc_mac

automation:
  - alias: ESPHome BLE Advertise
    mode: queued
    trigger:
      - platform: event
        event_type: esphome.on_ble_advertise
    action:
      - service: ble_monitor.parse_data
        data:
          packet: "{{ trigger.event.data.packet }}"

Advanced configuration where ESPHome devices gets MAC addresses from Passive BLE Monitor configuration

Note: Be sure that you turn on the input_boolean.settings_ble_gateway if you want to receive BLE packets from BLE Gateway’s.

Important note: New device address will be populated to ESPHome devices only after Passive BLE Monitor receives first BLE packet and creates entities for it. If in your configuration you don’t have BLE stick and you have only ESPHome devices you can use one of the following methods:

  1. Add this device MAC address manually into the input_text.settings_ble_gateway_add_device. After Passive BLE Monitor creates entities for new device, you can remove address.
  2. Enable discovery for Passive BLE Monitor and ESPHome BLE Gateway input_boolean.settings_ble_gateway_discovery. After required devices will be discovered turn off discovery options (for Passive BLE Monitor and ESPHome BLE Gateway) and clean up unneeded devices that got detected if any.
# ESPHome
ble_gateway:
  id: blegateway
  on_ble_advertise:
    then:
      homeassistant.event:
        event: esphome.on_ble_advertise
        data:
          packet: !lambda return packet;

binary_sensor:
  - platform: homeassistant
    id: ble_gateway_discovery
    entity_id: binary_sensor.ble_gateway
    attribute: discovery
    on_state:
      then:
        lambda: id(blegateway).set_discovery(x);

text_sensor:
  - platform: homeassistant
    id: ble_gateway_devices
    entity_id: binary_sensor.ble_gateway
    attribute: devices
    on_value:
      then:
        lambda: id(blegateway).set_devices(x);

switch:
  - platform: template
    id: switch_ble_gateway_discovery
    name: BLE Gateway Discovery
    icon: mdi:bluetooth-connect
    lambda: return id(blegateway).get_discovery();
    turn_on_action: [lambda: id(blegateway).set_discovery(true);]
    turn_off_action: [lambda: id(blegateway).set_discovery(false);]
    disabled_by_default: true
    entity_category: config

# Home Assistant
input_boolean:
  settings_ble_gateway:
    name: BLE Gateway
    icon: mdi:bluetooth
  settings_ble_gateway_discovery:
    name: BLE Gateway Discovery
    icon: mdi:bluetooth-connect

input_text:
  settings_ble_gateway_add_device:
    name: BLE Gateway Add Device
    icon: mdi:bluetooth-connect
    initial: ''

template:
  - binary_sensor:
      - name: BLE Gateway
        icon: mdi:bluetooth
        state: "{{ is_state('input_boolean.settings_ble_gateway', 'on') }}"
        attributes:
          discovery: "{{ is_state('input_boolean.settings_ble_gateway_discovery', 'on') }}"
          # devices: "{{ states | selectattr('entity_id', 'search', '^(device_tracker|sensor).ble_') | selectattr('attributes.mac address', 'defined') | map(attribute='attributes.mac address') | unique | sort | join('') | replace(':', '') ~ (states('input_text.settings_ble_gateway_add_device') | replace(':', '') | trim) if is_state('binary_sensor.ble_gateway', 'on') }}"
          # Important note: In Passive BLE Monitor version 7.8.2 and later 'attributes.mac address' was changed to 'attributes.mac_address', please update your config
          # devices: "{{ states | selectattr('entity_id', 'search', '^(device_tracker|sensor).ble_') | selectattr('attributes.mac_address', 'defined') | map(attribute='attributes.mac_address') | unique | sort | join('') | replace(':', '') ~ (states('input_text.settings_ble_gateway_add_device') | replace(':', '') | trim) if is_state('binary_sensor.ble_gateway', 'on') }}"
          # Note: In Home Assistant 2022.x, Passive BLE Monitor version 8.x and later you can use device attribute identifiers
          devices: >-
            {% set devices = namespace(items = []) %}
            {% for s in states | selectattr('entity_id', 'search', '^(device_tracker|sensor).ble_') | map(attribute='entity_id') %}
              {% set devices.items = devices.items + [device_id(s)] %}
            {% endfor %}
            {% set ns = namespace(items = []) %}
            {% for s in devices.items | unique %}
              {% set ns.items = ns.items + [(device_attr(s, 'identifiers') | first)[1]] %}
            {% endfor %}
            {{ ns.items | unique | sort | join('') | replace(':', '') ~ (states('input_text.settings_ble_gateway_add_device') | replace(':', '') | trim) if is_state('binary_sensor.ble_gateway', 'on') }}

More configuration examples you can find in examples folder.

11 Likes

Hi. First of all i’d like to thank you very much.
This is exactly what i needed for a while and now you made it real.
I’ve completed my wifi and zigbee coverage before and was struggling with raspberry pi 0s and virtualhere for bt coverage (had tons of issues to solve). If someone would have asked me what i wish for home assistant , this was the answer. As you can understand, i can’t thank you enough.

TLDR;
Most probably i was doing it wrong, due to my lack of knowledge, but i tried to change the esp32_ble_tracker.cpp and .h fiiles under esphome > build > esp32name > src > esphome > components > esp32_ble_tracker > , tried to reinstall but got an error. When i checked the updated fies again, my updates were gone. What am i doing wrong, or better, what should i do and how should i do ?

Since it needs some tweaks on esp32_ble_tracker at the moment ( i am aware of your pending request ) , i am having issues to complete that step.

I am very new to esp ecosystem, so i need someone to walk through this process with me. I hope you or someone else can help me to understand what’s really needed.

What i did;
I created a fresh system, only installed esphome, let’s encrypt, file editor add-ons.
I searched for a file named esp32_ble_tracker.cpp or esp32_ble_tracker.h but couldn’t find anything.

Then i pluged in an ESP32 and flashed it over web browser with esphome add-on inside home assistant.
After initial setup i couldn’t find those files again.

Then i added

external_components:
  - source: github://myhomeiot/esphome-components

searched for the files again, couldn’t find anything.

Then i inserted ble_gateway: and it asked for esp32_ble_tracker and i added that too.
After that i was able to find those files under;
esphome > build > esp32name > src > esphome > components > esp32_ble_tracker >
i edited esp32_ble_tracker.cpp and esp32_ble_tracker.h files with the changes i saw on github.

And i tried to install again over web browser with esphome add-on inside home assistant.
got an error like

Compiling /data/esp32-myhomeiot-test/.pioenvs/esp32-myhomeiot-test/src/esphome/components/ble_gateway/ble_gateway.cpp.o
src/esphome/components/ble_gateway/ble_gateway.cpp: In member function 'virtual bool esphome::ble_gateway::BLEGateway::parse_device(const esphome::esp32_ble_tracker::ESPBTDevice&)':
src/esphome/components/ble_gateway/ble_gateway.cpp:60:56: error: 'const class esphome::esp32_ble_tracker::ESPBTDevice' has no member named 'get_scan_result'
     auto packet = scan_result_to_hci_packet_hex(device.get_scan_result());
                                                        ^
*** [/data/esp32-myhomeiot-test/.pioenvs/esp32-myhomeiot-test/src/esphome/components/ble_gateway/ble_gateway.cpp.o] Error 1
========================== [FAILED] Took 9.71 seconds ==========================

I checked those files again, and my changes were gone. I realized that with my lack of knowledge, i was doing it wrong and asked for help at a level for newbies. What should i do and how should i do.

Thanks in advance.

I am very glad to hear that you needed such a component, I wrote it because I have been looking for such one for a long time. :wink:

You can patch esp32_ble_tracker using this configuration:

external_components:
  - source: github://myhomeiot/esphome-components
  - source: github://pr#2854
    components: [esp32_ble_tracker]

Hope this helps.

Update: Added this to README.

Since you made it, yep, that helped :smiley:
Thanks a lot, gateway configuration was flashed to device without error.
I will continue implementing whole process and let you know the results.
You are the one, cheers…
EE

Hello!

Can i use this also as a bluetooth range extender for my Oral-b toothbrush or Switchbot Curtains?

@Cinamon Yes, why not, it’s sends BLE advertisement packets as is just wrap it into HCI header for Passive BLE Monitor. If Passive BLE Monitor support Oral-b toothbrush or Switchbot Curtains devices, you will get it in Home Assistant.

@myhomeiot thx for your reply. Passive BLE monitor doesn’t support (yet) oral-b or Switchbot curtains.
Is my esphome config correct:
Just to be sure: (After that i will try to make an addition at Passive BLE monitor)

esphome:
  name: ble-gateway

external_components:
  - source: github://myhomeiot/esphome-components
  - source: github://pr#2854
    components: [esp32_ble_tracker]

esp32:
  board: esp32doit-devkit-v1

# Enable logging
logger:

# Enable Home Assistant API
api:

ota:
  password: "!secret ota_password "

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_pass

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "!secret wifi_backup_ssid"
    password: "!secret wifi_backup_pass"

# Example configuration entry
esp32_ble_tracker:


captive_portal:

ble_gateway:
  id: blegateway
  devices:
    - mac_address: D0:2E:AB:33:5C:C8   #Toothbrush
    - mac_address: DE:B2:AC:CF:33:67   #Switchbotcurtain
  on_ble_advertise:
    then:
      homeassistant.event:
        event: esphome.on_ble_advertise
        data:
          packet: !lambda return packet;

text_sensor:
  - platform: homeassistant
    id: ble_gateway_devices
    entity_id: binary_sensor.ble_gateway
    attribute: devices
    on_value:
      then:
        lambda: id(blegateway).set_devices(x);

@Cinamon Looks good, after you get it working you can remove devices section with MAC addresses because you configure binary_sensor.ble_gateway and probably use Advanced configuration method where all devices MAC addresses populated to BLE Gateways from Home Assistant.

In the ESP32 log you can see if it’s receives something from your devices and sent it to Home Assistant, as well you can see it if in Home Assistant Development Tools menu you subscribe to esphome.on_ble_advertise event.

Hi. I am trying to understand what exactly ESPHome BLE Gateway does. If I am not mistaken, Home Assistant’s Passive BLE Monitor integration can receive data from BLE devices. What can ESPHome BLE Gateway accomplish that HA’s Passive BLE Monitor integration cannot? Thank you.

Not BLE Gateway but same method can be used in special cases for example you can feed Passive BLE Monitor with MiFlora (HHCCJCY01) battery data which requires active BLE connection.

1 Like

Just another question: If I use a dozen of the MiFlora (HHCCJCY01) and moving them around in the house (3 floors), maybe the garden too.

Can I “register” all sensor to seversl ESPHome devices?

Or do I have to track, my sensor and check which gateway is able to receive certain sensors?

If you mean regilar BLE data from MiFlora (HHCCJCY01) like temperature/humidity, etc than yes, sure you can register all sensors at all esp’s or even better, you can populate all esp with all MAC’s configured in Passive BLE Monitor using advanced configuration. In this case when you add new device you don’t need to recompile and upload new ESP firmware.

If yo mean MiFlora battery data which uses myhomeiot_ble_client and send result to Passive BLE Monitor as BLE packet (like in this example) than, also yes, but you should add this configuration at all esp’s and should expect than MiFlora device will be polled from all ESP for which MiFlora device is “visible”.

The PR was merged in ESPHome 2022.1, so now using BLE Gateway become even easier.

Has something changed with recent updates. I had this setup and working but it no longer pushes the data to home assistant. You can see it grabbing the data in the esphome logs but it doesn’t update Home Assistant anymore.

Nothing changes, except that with ESPHome 2022.1 you don’t need to modify esp32_ble_tracker component or specify PR (source: github://pr#2854).

You can check in Home Assistant Developer Tools that you getting events esphome.on_ble_advertise and that in Home Assistant log you don’t have errors connected to Passive BLE Monitor:

I’m trying to integrate two Xiami devices: i. a Smoke Detector (Model: JTYJGD03MI), and ii. a Door/Window Sensor (Model: JTYJGD03MI).

I used the Xiaomi Cloud Tokens Extractor successfully to extract the BLE KEY for the Door/Window Sensor. Although their packets are seen in Home Assistant by listening to the esphome.on_ble_advertise, their entities are NOT created automatically. I tried the Advanced Configuration with no success.

When I integrate ATC sensors, their entities are created just fine. Any thoughts what has been happening?

As I can see both models are supported by Passive BLE Monitor with encryption. If you see packets on esphome.on_ble_advertise events in Home Assistant everything should be fine. I think the problem can be only with BLE keys which you extract from Xiaomi Cloud.

Please check that sensors works in MiHome and try to extract the keys again, check that the keys properly defined in Passive BLE Monitor configuration.
If it’s doesn’t help I think if you PM the packets and encryption keys to @Ernst he will be able to help you.

Thank you for your guidance, @myhomeiot – re-issuing the BLE KEYS from the Xiaomi Cloud did the trick. Many thanks!

Hi – I noticed that the Xiaomi BLE sensors (temp/humidity), although they broadcast at the same intervals as previously, their values are updated far less frequently in HASS than before the BLE Gateway. Any ideas why that is? Many thanks!

Gateway send data as is and when they received, update interval can be slightly different due the difference between computer BT and ESP BLE implementation but it’s can’t be a minutes. AFAIK Xiaomi BLE sensors (temp/humidity) with stock firmware sends data every 10 minutes so if you see updates every 10 minutes - it’s ok. If not, try to place sensor at the front of ESP and check if you get updates more frequently maybe you have problem with coverage. If it’s doesn’t help, describe your previous setup where you get updates more frequently and put here some temp/humidity charts where we can see update time.