ESPHome BLE Gateway and other BLE components

If you have device already defined in Passive BLE Monitor (with MAC, BLE key and token) all data will come under this device and it’s create all sensors. If you are waiting for other BLE devices to be created automatically you should add these devices MAC addresses at ESPHome BLE Gateway under devices section:

ble_gateway:
  devices:
    - mac_address: 54:EF:44:C1:D5:44
    - mac_address: XX:XX:XX:XX:XX:XX
    - mac_address: XX:XX:XX:XX:XX:XX

EDIT: Don’t forget to set discovery to True at Passive BLE Monitor, more info you can find here
It’s not good idea to left discovery on after you finish setup, because it’s can be a huge number of BLE devices nearby.

@myhomeiot I’m sorry to keep bothering you. Thank you for your patience. I have the ESPHome portion working well I think:

esp32_ble_tracker:

    
ble_gateway:
  devices:
    - mac_address: 54:EF:44:C1:D5:44
    - mac_address: 54:EF:44:C1:D4:64
  on_ble_advertise:
    then:
      homeassistant.event:
        event: esphome.on_ble_advertise
        data:
          packet: !lambda return packet;

As mentioned, these events are being broadcast as I can see them in Node Red via the Home Assistant nodes.

ble_monitor:
  bt_interface: "disable"
  bt_auto_restart: False
  active_scan: False
  discovery: True
  period: 60
  use_median: False
  decimals: 1
  log_spikes: False
  restore_state: False
  report_unknown: False
  devices:
    # sensor
    - mac: '54:EF:44:C1:D5:44'
      name: 'LR Smoke Detector'
      encryption_key: '3ac8778889b571f55e5e4cb72cddfbe4'
    - mac: '54:EF:44:C1:D4:64'
      name: 'Hallway Smoke Detector'
      encryption_key: '11ff9d874569ba9566c5618883a7112f' 

I’ve added this to my configuration.yaml file. I saw in the docs it said you could use one configuration or the other. I’m guessing that the component will use whatever is in configuration.yaml if available and then web ui settings?

Does my config look correct? I’m still not seeing these devices. Would I see errors in a log somewhere?

This is the output from ESPHome:

[21:07:32][V][api.connection:809]: Hello from client: 'Home Assistant 2022.4.7 (::FFFF:C0A8:19)'
[21:07:32][D][api.connection:827]: Home Assistant 2022.4.7 (::FFFF:C0A8:19): Connected successfully
[21:07:34][D][esp32_ble_tracker:726]: Found device 66:86:16:9E:2C:F6 RSSI=-85
[21:07:34][D][esp32_ble_tracker:747]:   Address Type: RANDOM
[21:07:55][D][esp32_ble_tracker:726]: Found device 74:6E:A0:4C:DD:80 RSSI=-87
[21:07:55][D][esp32_ble_tracker:747]:   Address Type: RANDOM
[21:07:57][D][esp32_ble_tracker:726]: Found device 46:F0:82:C7:96:BA RSSI=-85
[21:07:57][D][esp32_ble_tracker:747]:   Address Type: RANDOM
[21:08:32][D][esp32_ble_tracker:726]: Found device 00:9E:C8:AE:AE:90 RSSI=-90
[21:08:32][D][esp32_ble_tracker:747]:   Address Type: PUBLIC
[21:09:17][D][ble_gateway:063]: [54:EF:44:C1:D4:64] Packet 043E320201000064D4C144EF54260201060E1695FE105997098464D4C144EF5413094D6920536D6F6B65204465746563746F7200CB
[21:09:17][D][esp32_ble_tracker:726]: Found device 54:EF:44:C1:D4:64 RSSI=-53
[21:09:17][D][esp32_ble_tracker:747]:   Address Type: PUBLIC
[21:09:17][D][esp32_ble_tracker:749]:   Name: 'Mi Smoke Detector'
[21:09:23][D][ble_gateway:063]: [54:EF:44:C1:D5:44] Packet 043E330201000044D5C144EF54270201060F1695FE305897090544D5C144EF540813094D6920536D6F6B65204465746563746F7200D4
[21:09:23][D][esp32_ble_tracker:726]: Found device 54:EF:44:C1:D5:44 RSSI=-44
[21:09:23][D][esp32_ble_tracker:747]:   Address Type: PUBLIC
[21:09:23][D][esp32_ble_tracker:749]:   Name: 'Mi Smoke Detector'
[21:09:33][D][ble_gateway:063]: [54:EF:44:C1:D4:64] Packet 043E320201000064D4C144EF54260201060E1695FE105997098464D4C144EF5413094D6920536D6F6B65204465746563746F7200D0

@myhomeiot

I installed a BT dongle to my PC and I’m now seeing one of the two smoke detectors:

Is this how they should appear? I more optimistic seeing this but the unknown values are concerning.

If you get one of the smoke detectors in the HA this mean that Passive BLE Monitor get packet and at least parse it and get some data like type of the sensor. Probably you have a problem with encryption key, try to reconnect sensor to mihome, and extract new token like it’s suggested here

When you get it working, you can disconnect BT dongle from PC and see if you get info over BLE Gateways at ESPHome (you don’t need to change anything in configuration).
Your configuration in previous post looks ok.

Released 1.0.8, changes:
BLE Gateway

  • Added Discovery, thanks @ilarrain
  • Updated documentation

First I would like to thank for the work in the project. But I’m afraid that I need some help to put it to work.

First I flashed an ESP32 board with the following ESPHome code:

substitutions:
  devicename: blegateway # (lower-case) hostname, ap
  friendlyname: blegateway  # mqtt, sensors.

packages:
  device: !include packages/devices/esp32-esp-wroom-32.yaml
  base: !include packages/base-esp32.yaml

external_components:
  - source: github://myhomeiot/esphome-components

esp32_ble_tracker:        #  <!---- Is this required?

ble_gateway:
  id: blegateway
  on_ble_advertise:
    then:
      homeassistant.event:
        event: esphome.on_ble_advertise
        data:
          packet: !lambda return packet;

binary_sensor:
  - platform: homeassistant
    id: ble_gateway_discovery
    entity_id: binary_sensor.ble_gateway
    attribute: discovery
    on_state:
      then:
        lambda: id(blegateway).set_discovery(x);

text_sensor:
  - platform: homeassistant
    id: ble_gateway_devices
    entity_id: binary_sensor.ble_gateway
    attribute: devices
    on_value:
      then:
        lambda: id(blegateway).set_devices(x);

switch:
  - platform: template
    id: switch_ble_gateway_discovery
    name: BLE Gateway Discovery
    icon: mdi:bluetooth-connect
    lambda: return id(blegateway).get_discovery();
    turn_on_action: [lambda: id(blegateway).set_discovery(true);]
    turn_off_action: [lambda: id(blegateway).set_discovery(false);]
    disabled_by_default: true
    entity_category: config

Then I added the Bluetooth plugin to HA and left configured like:

Then added a yaml package file with this code:

input_boolean:
  settings_ble_gateway:
    name: BLE Gateway
    icon: mdi:bluetooth
  settings_ble_gateway_discovery:
    name: BLE Gateway Discovery
    icon: mdi:bluetooth-connect

input_text:
  settings_ble_gateway_add_device:
    name: BLE Gateway Add Device
    icon: mdi:bluetooth-connect
    initial: ''

template:
  - binary_sensor:
      - name: BLE Gateway
        icon: mdi:bluetooth
        state: "{{ is_state('input_boolean.settings_ble_gateway', 'on') }}"
        attributes:
          discovery: "{{ is_state('input_boolean.settings_ble_gateway_discovery', 'on') }}"
          devices: >-
            {% set devices = namespace(items = []) %}
            {% for s in states | selectattr('entity_id', 'search', '^(device_tracker|sensor).ble_') | map(attribute='entity_id') %}
              {% set devices.items = devices.items + [device_id(s)] %}
            {% endfor %}
            {% set ns = namespace(items = []) %}
            {% for s in devices.items | unique %}
              {% set ns.items = ns.items + [(device_attr(s, 'identifiers') | first)[1]] %}
            {% endfor %}
            {{ ns.items | unique | sort | join('') | replace(':', '') ~ (states('input_text.settings_ble_gateway_add_device') | replace(':', '') | trim) if is_state('binary_sensor.ble_gateway', 'on') }}

automation:
  - alias: ESPHome BLE Advertise pkg
    mode: queued
    trigger:
      - platform: event
        event_type: esphome.on_ble_advertise
    action:
      - service: ble_monitor.parse_data
        data:
          packet: '{{ trigger.event.data.packet }}'
          #gateway_id: '{{ trigger.event.data.gateway_id }}'

That created three entities:
imagen

I really don’t know what they are for. I switched discovery, but nothing happened in the textbox or HA.
I bought 5 LYWSD03MMC, and powered on the first. Activated discovery, but nothing happened. So I looked at the logs and saw the MAC. I added that TO the textbox, I think and then appeared at the BLE integration. I configured the device but left empty the encryption key.

That created this device, with Temp/Hum deactivated:


I activated them but saw no activity, so used the Telink tool as indicated and activated the device and got the key.

But nothing changed. There’s no activity and even in the Gateway log I can’t see nothing.

INFO Successfully connected to esphome-blegateway.local
[20:55:51][D][sensor:125]: 'blegateway_Uptime': Sending state 3.81222 minutes with 0 decimals of accuracy
[20:56:06][D][sensor:125]: 'blegateway_RSSI': Sending state -56.00000 dBm with 0 decimals of accuracy
[20:56:51][D][sensor:125]: 'blegateway_Uptime': Sending state 4.81222 minutes with 0 decimals of accuracy
[20:57:02][I][ota:105]: Boot seems successful, resetting boot loop counter.
[20:57:02][D][esp32.preferences:114]: Saving preferences to flash...
[20:57:02][D][esp32.preferences:131]: NVS data not changed skipping 233825511  len=4
[20:57:03][D][esp32_ble_tracker:217]: Starting scan...
[20:57:03][D][esp32_ble_tracker:726]: Found device 44:1C:A8:6E:5F:9E RSSI=-83
[20:57:03][D][esp32_ble_tracker:747]:   Address Type: PUBLIC
[20:57:15][D][esp32_ble_tracker:726]: Found device A4:C1:38:AC:4B:E9 RSSI=-54
[20:57:15][D][esp32_ble_tracker:747]:   Address Type: PUBLIC
[20:57:15][D][esp32_ble_tracker:749]:   Name: 'LYWSD03MMC'
[20:57:51][D][sensor:125]: 'blegateway_Uptime': Sending state 5.81218 minutes with 0 decimals of accuracy
[20:58:06][D][sensor:125]: 'blegateway_RSSI': Sending state -63.00000 dBm with 0 decimals of accuracy
[20:58:51][D][sensor:125]: 'blegateway_Uptime': Sending state 6.81218 minutes with 0 decimals of accuracy

Can you help me to make it work, please?
The Telink tool showed Temp and Hum values, along with The token and key, but the id was empty.
Does the token has to be put somewhere?
Does the mac has to be altered to be put somewhere?
Thanks in advance.

Edit: Is still required to hard-code the sensor MACs somewhere? I only added the shown code, but I can see something related to ble_monitorhere that I don’t know if it’s required if using the BLE integration.
The documentation is not clear to me as showing the required or not required steps and their order.

Everything looks good but you should put correct encryption key for this device in Passive BLE Monitor manually.

Yes, I had the encryption key there and it didn’t work. But something weird happened.
When I used the Telink web for the first time, I connected the device, I activated It and stored the token and key safely. I entered the key in the Passive BLE monitor but it never worked.

Yesterday at those “it’s too late hours”, I tried to flash the device with the custom firmware, but it couldn’t be activated a 2nd time. Weird chars where appearing as a response to the activation process.
Then one of those times it went well and thrown another key. So I flashed it.
And then it worked. I could receive sensor data form it. The pairing info I was previously receiving (screen above) was from the gateway itself.

I guess if all of this has been modifications to the firmware in new devices done by the maker that renders them incompatible.

@myhomeiot
Whats the purpose of the discovery switch? I know that it makes the gateway go berserk, but I can’t see nothing new in the logs that wasn’t there in normal mode. Only himselfs entries.

Then, it’s really needed the esp32_ble_tracker include in the gateway device? It seems that it’s always tracking /discovering new devices.

The HA textbox (add device) how it works? Should it have to show a list of registered devices? Or it only works by writing a MAC there and pushing return, and then? Leave it there? Erase?
How you can erase gateway registered devices?
When it starts says that there are four devices registered, but I only have one temp sensor (it’s one of them).

Yes, discovery will disable MAC address filters and allow to send BLE packets from all nearby BLE devices. Don’t forget to enable discovery options at Passive BLE Monitor as well. You can also see BLE devices around your from mobile phone with application like LightBlue.

Yes, you need esp32_ble_tracker, because BLE Gateway based on it and send specially formatted BLE advertising data received by esp32_ble_tracker to Passive BLE Monitor.

Yes, HA textbox (add device) temporary adds device MAC address to gateway, after device will be discovered by Passive BLE Monitor you can erase content of this textbox.

You can erase registered device at Passive BLE Monitor configuration look into docs

BLE Gateway doesn’t parse and process any data from BLE packets, just receive it and send to Passive BLE Monitor, so if you have problem with data from your device you should look into existing’s issues of Passive BLE Monitor and add your if needed.

I hope I understand correctly and answer to all your questions.

Thanks @myhomeiot

So the procedure to add a device is:

Leave the discovery off and put the known mac in the textbox.
OR
Turn on discovery and leave the box blank. All devices will be added.

I’ve already added 5 thermometers, but none of them was automatically added to the integration.
I had to go to the BLE integration, and leaving the auto adding check unchecked, just manually add the device with mac and encryption key, and then wait for some (a good amount) of minutes. Then the device appears (or not. Then delete and recreate, wait… repeat)

If I leave the Integration auto add check on, it does add a varying and unrecognized amounts of devices, but none of them is the desired one.

Also discovery had to be on AND with the MAC in the box.

Yes.

For devices with encryption I think the best way to add it’s:

  • Disable discovery at Passive BLE Monitor
  • Add device MAC and encryption key to Passive BLE Monitor integration manually
  • Enable discovery at BLE Gateway and wait until device data will appear in Passive BLE Monitor integration
  • Disable discovery at BLE Gateway

ESP devices usually has less coverage than USB dongle, for initial setup I suggest to temporary place BLE device near ESP.

1 Like

In fact, after having left it all done yesterday in the evening, today none of the sensors work. It has been +13h since the last received data. The nearest sensor is 3m away with one wall in between and there’s another at eyesight. All of them where working past evening.
I don’t know what could be failing, the ATC custom firmware as I noticed during the setup that some devices where randomly hanging and needed battery replacing.
I have connectivity with the ESP gateway and it seems to be receiving packets:

[10:57:22][D][sensor:125]: 'blegateway_RSSI': Sending state -66.00000 dBm with 0 decimals of accuracy
[10:57:25][D][ble_gateway:063]: [A4:C1:38:2A:2B:EE] Packet 043E2B02010000EE2B2A38C1A41F0201060F1695FE30585B05D1EE2B2A38C1A4080B094C5957534430334D4D43AA
[10:57:25][D][esp32_ble_tracker:726]: Found device A4:C1:38:2A:2B:EE RSSI=-86
[10:57:25][D][esp32_ble_tracker:747]:   Address Type: PUBLIC
[10:57:25][D][esp32_ble_tracker:749]:   Name: 'LYWSD03MMC'
[10:57:27][D][ble_gateway:063]: [A4:C1:38:2A:2B:EE] Packet 043E3602010000EE2B2A38C1A42A0201061A1695FE58585B05D2EE2B2A38C1A4459F1C58B0050000AFF595790B094C5957534430334D4D43AD
[10:57:35][D][sensor:125]: 'blegateway_Uptime': Sending state 676.95490 minutes with 0 decimals of accuracy
[10:57:40][D][esp32_ble_tracker:726]: Found device 55:29:59:4F:8D:29 RSSI=-66
[10:57:40][D][esp32_ble_tracker:747]:   Address Type: RANDOM
[10:57:54][D][ble_gateway:063]: [A4:C1:38:EA:E4:0D] Packet 043E29020100000DE4EA38C1A41D10161A18A4C138EAE40D00CE49640C34D70B094154435F454145343044AF
[10:58:04][D][ble_gateway:063]: [A4:C1:38:AC:4B:E9] Packet 043E2902010000E94BAC38C1A41D10161A18A4C138AC4BE900D044590BC3C10B094154435F414334424539B8
[10:58:07][D][ble_gateway:063]: [A4:C1:38:5F:5F:3A] Packet 043E29020100003A5F5F38C1A41D10161A18A4C1385F5F3A00D744640C3C540B094154435F354635463341AA
[10:58:13][D][ble_gateway:063]: [A4:C1:38:EA:E4:0D] Packet 043E29020100000DE4EA38C1A41D10161A18A4C138EAE40D00CE49640C34D70B094154435F454145343044AF
[10:58:14][D][ble_gateway:063]: [A4:C1:38:83:9E:68] Packet 043E2902010000689E8338C1A41D10161A18A4C138839E6800D546640C64290B094154435F383339453638AF
[10:58:14][D][esp32_ble_tracker:726]: Found device A4:C1:38:83:9E:68 RSSI=-81
[10:58:14][D][esp32_ble_tracker:747]:   Address Type: PUBLIC
[10:58:14][D][esp32_ble_tracker:749]:   Name: 'ATC_839E68'
[10:58:35][D][sensor:125]: 'blegateway_Uptime': Sending state 677.95483 minutes with 0 decimals of accuracy
[10:58:36][D][ble_gateway:063]: [A4:C1:38:83:9E:68] Packet 043E2902010000689E8338C1A41D10161A18A4C138839E6800D546640C64290B094154435F383339453638AF
[10:58:57][D][ble_gateway:063]: [A4:C1:38:83:9E:68] Packet 043E2902010000689E8338C1A41D10161A18A4C138839E6800D546640C64290B094154435F383339453638AE
[10:59:22][D][sensor:125]: 'blegateway_RSSI': Sending state -64.00000 dBm with 0 decimals of accuracy
[10:59:35][D][sensor:125]: 'blegateway_Uptime': Sending state 678.95483 minutes with 0 decimals of accuracy
[10:59:39][D][ble_gateway:063]: [A4:C1:38:83:9E:68] Packet 043E2902010000689E8338C1A41D10161A18A4C138839E6800D546640C642A0B094154435F383339453638AD
This is the furthest away => [11:00:19][D][ble_gateway:063]: [A4:C1:38:2A:2B:EE] Packet 043E2B02010000EE2B2A38C1A41F0201060F1695FE30585B05D5EE2B2A38C1A4080B094C5957534430334D4D43AD
[11:00:20][D][ble_gateway:063]: [A4:C1:38:EA:E4:0D] Packet 043E29020100000DE4EA38C1A41D10161A18A4C138EAE40D00CF48640C32DA0B094154435F454145343044A5
[11:00:22][D][sensor:125]: 'blegateway_RSSI': Sending state -66.00000 dBm with 0 decimals of accuracy

But none of this makes to HA.
This is the nearest:

WTF!??
imagen
The automation was deactivated. I only renamed it yesterday, but it seems it turned itself off. :man_facepalming:

If you see lines like [A4:C1:38:EA:E4:0D] Packet in the log this means that device sends data.
ESPHome BLE Advertise automation get data from ESP and sends it to Passive BLE Monitor, if it’s off sure you will not see anything in the Passive BLE Monitor. You can check in the log when and why it’s was turned off.

If you use custom firmware I suggest you to try pvvx in one of device.

Are there any differences? I didn’t try it as I supposed that it was not possible to customize the devices behavior like in yours.
BTW I don’t use the custom announcements but the Mi-Like one.

pvvx has lot of settings (for example you can connect dry contact to the window and get open/close status) and less power consumption than ATC custom firmware, but if you use regular Mi firmware it’s also good…

1 Like

Hello guys,
Since homeassistant 2022.08 now comes with built-in BLE support, I am wondering whether ble gateway in esphome will work for it without BLE monitor?

2 Likes

I was wondering the same thing. I don’t think there is currently anything like the parse_data service in the new core integrations.

That’s hopefully and definitely the next step: find a way to make esphome ESP32 work as remote adapters, otherwise the new Bluetooth integration may lack interest due to Bluetooth coverage “issues”.

+1 for that idea

4 Likes

well I don’t think it is working 100% for me, in fact my devices are saying they are offline but I open up logs via WiFi and it is still spitting out info so it is on but getting some random messages now when parsing data that I did’t get before