ESPHome BLE Gateway and other BLE components

[ew-ewim (Wim)]

Hey, interesting reading on the components you made. I am however looking for a way to send BLE messages from an esp32 using esphome: not being a programmer I have not found anything yet that is useful. The idea is to use BLE as a way to send info and use as less power as possible in a battery operated co2 sensor. Any suggestions or ideas or even examples ?

[danps]

Hi @myhomeiot
Firstly, thanks so much for this custom component. I was previously doing the same sort of thing with ESPHome and NodeRed, but much much messier, and this approach works so much better!
I’ve done 1 thing a bit different to how you’ve suggested, and was wondering if there’s a good reason that I shouldn’t do it this way, because it seems to work really well?
Instead of on_ble_advertise creating an event, which is then picked up by an automation, which then calls the service, I just call the service directly from ESPHome…

  on_ble_advertise:
    then:
      homeassistant.service:
        service: ble_monitor.parse_data
        data:
          packet: !lambda return packet;
          gateway_id: $gateway_id

My logic being it saves some additional steps / overhead, and it works great, but what do you think?
Thanks again
Dan

[philpo (Phil)]

Dan, I have to agree - using the service has the advantage of using the Home Assistant Passive BLE Monitor integration to do the decryption (you don’t have to put the encryption keys on the espNodes, just the MAC addresses), it supports more parameters (RSSI and battery voltage) and it supports many more device types. The load on the espHome nodes is reduced as well.
Thanks also,
Phil

[janpfo (Jan Petter)]

Is it possible to auto discover devices with this?
I added the Passive BLE monitor integration in HA with auto discovery on.

Esphome config:

ble_gateway:
 on_ble_advertise:
    then:
      homeassistant.service:
        service: ble_monitor.parse_data
        data:
          packet: !lambda return packet;

[mdmc95 (John)]

I am currently using Passive BLE Monitor with a BT dongle in the HAS unit and for remote a M5Stick CP with OpenMQQTGateway. (OMG)

I like OMG and it works well but the range of BLE compatible devices is far smaller than Passive BLE Monitor. (Though it does have IR and RF433 which I will be getting into with something other than the M5Stick at a later date).

I have been looking for a “remote BLE gateway” to replace the M5Stick. That is a device remote from the main HAS unit that can receive BLE signals and report them back to Passive BLE Monitor.

So the question is: what BLE Gateway is best to use?

One answer I have read is to create another HA instance on a Pi Zero and “join” that to the main instance using a custom component. I was really looking for an off the shelf box similar to the M5Stcik that could send the info back to the HAS.

Cheers

[danps]

Hi @janpfo
Yes and No!
You still need to have the Mac address in the list of advertisements that the gateway will forward, but after that the auto-discovery works.
My workflow here is that with a new device, I watch the ESPHome logs to find the Mac address of the device, then add that to the gateway list, and the rest auto-discovers.
Dan

[1technophile]

@mdmc95

like OMG and it works well but the range of BLE compatible devices is far smaller than Passive BLE Monitor.

Hello,

That’s mainly an antenna/board topic. I would advise the use of the Olimex esp32 gateway with an external antenna (EA) and enclosure:

ESP32-GATEWAY - Open Source Hardware Board

[mdmc95 (John)]

I set up an old Pi3b as my remote running HA and linking to my main HA instance. It’s a solution but not an elegant or economical one. I’ll look into the Olimex device. Thanks. Appreciate your time.

[sukzessiv]

Hello,

this custom component seems like my problem solver. But before starting I have two questions:

1. So I could configure every of my 20 ESP32 as BLE Gateway in my house and ~ 8 of them getting the same advertised data from e.g. Mijia temperature humidity sensor. In my understanding HA will get 8x the same data, is there a function implemented to prevent multiple measurement points of the same event?

2. At this moment I have only one ESP32 (in the middle of the house) as designated BLE tracker to collect around 10 Mijia Sensor data. Mostly a lot of the advertised data is not collected by the esp32, gabs between 2-8 minutes at an advertising interval of 2.5 seconds. Can this BLE Gateway improve the coverage or reduce the packet losses?

Thanks

[dinki]

@digiwolf

I’m trying to integrate two Xiami devices: i. a Smoke Detector (Model: JTYJGD03MI), and ii. a Door/Window Sensor (Model: JTYJGD03MI).

I used the Xiaomi Cloud Tokens Extractor successfully to extract the BLE KEY for the Door/Window Sensor. Although their packets are seen in Home Assistant by listening to the esphome.on_ble_advertise, their entities are NOT created automatically. I tried the Advanced Configuration with no success.

When I integrate ATC sensors, their entities are created just fine. Any thoughts what has been happening?

@myhomeiot

As I can see both models are supported by Passive BLE Monitor with encryption. If you see packets on esphome.on_ble_advertise events in Home Assistant everything should be fine. I think the problem can be only with BLE keys which you extract from Xiaomi Cloud.

Please check that sensors works in MiHome and try to extract the keys again, check that the keys properly defined in Passive BLE Monitor configuration.
If it’s doesn’t help I think if you PM the packets and encryption keys to @Ernst he will be able to help you.

@myhomeiot Hoping to get some clarification. I am using the same smoke detector (Model: JTYJGD03MI) mentioned. You seem to be saying that is device is supported by the project. Is this correct?

Further, I’m running HA Core Docker on a PC without built in bluetooth. Can I install a bluetooth dongle and use it with the docker container? I’m not sure if this requires BlueZ or if that is available in the container.

Any advice is most appreciated.

[myhomeiot]

@dinki Hello, BLE Gateway it’s ESPHome component which sends the BLE data as is to Passive BLE Monitor integration for Home Assistant, so you should look for device support at Passive BLE Monitor. As I see the smoke detector (Model: JTYJGD03MI) is supported.

If you have ESPHome compatible device, you can use it with BLE Gateway and Passive BLE Monitor without Bluetooth dongle, if you install Bluetooth dongle into PC (yes, it’s will work inside container, check instruction here) when you don’t need ESPHome compatible device with BLE Gateway, but sure you can use it as well to get better BLE coverage.

[myhomeiot]

@sukzessiv

1. So I could configure every of my 20 ESP32 as BLE Gateway in my house and ~ 8 of them getting the same advertised data from e.g. Mijia temperature humidity sensor. In my understanding HA will get 8x the same data, is there a function implemented to prevent multiple measurement points of the same event?

Yes, this function implemented inside Mijia BLE protocol (it’s has packet id) and Passive BLE Monitor. It’s a nature of BLE that all nearby device receives same data packets.

@sukzessiv

2. At this moment I have only one ESP32 (in the middle of the house) as designated BLE tracker to collect around 10 Mijia Sensor data. Mostly a lot of the advertised data is not collected by the esp32, gabs between 2-8 minutes at an advertising interval of 2.5 seconds. Can this BLE Gateway improve the coverage or reduce the packet losses?

Coverage no, reduce the packet losses - I think yes, because BLE Gateway doesn’t process any data and sends it as is to Passive BLE Monitor. Regular ESP components for Mijia sensors spend time for data processing and can miss the packets, because all BLE data processing in ESPHome is sequential with buffer for 16 packets (as I remember). Moving data processing from ESPHome to Home Assistant will also free up memory and CPU of ESP and as result ESP will works much stable and can process more BLE packets from devices.

[myhomeiot]

@mdmc95

So the question is: what BLE Gateway is best to use?

Best use of BLE Gateway to replace BT dongle or extend BT coverage for Passive BLE Monitor. BT dongles usually has better coverage because of better antenna when many ESP devices in the market has PCB antenna. Here the example how you can replace antenna in M5Stack Atom.

If you change OMG in M5Stick to BLE Gateway and ESPHome you can also use IR and RF433, see update section at the end of the topic.

[myhomeiot]

If you want to send data by BLE you can look into Passive BLE Monitor DIY sensors, but with ESPHome I think better to use WiFi to send data and deep sleep.

[1technophile]

@myhomeiot

change OMG in M5Stick to BLE Gateway and ESPHome you can also use IR and RF433

OMG supports also 433 and IR along with BLE

[dinki]

@myhomeiot

@dinki Hello, BLE Gateway it’s ESPHome component which sends the BLE data as is to Passive BLE Monitor integration for Home Assistant, so you should look for device support at Passive BLE Monitor. As I see the smoke detector (Model: JTYJGD03MI) is supported.

If you have ESPHome compatible device, you can use it with BLE Gateway and Passive BLE Monitor without Bluetooth dongle, if you install Bluetooth dongle into PC (yes, it’s will work inside container, check instruction here) when you don’t need ESPHome compatible device with BLE Gateway, but sure you can use it as well to get better BLE coverage.

Thank you @myhomeiot for the quick and detailed response. I did have an ESP32 laying around and I flashed it with ESPHome BLE Gateway. I can see my device in the verbose logs. I ran token extractor to get the information needed for configuration:

Please forgive me as I have tried looking in the logs, but how do I get the integration to use the values from my ESPHome BLE Gateway instead of trying to find a local BT device?

[dinki]

I found the problem and resolved it. I had not used the external_components to install the esp32_ble_tracker.

I am using this configuration:

esp32_ble_tracker:

    
ble_gateway:
  devices:
    - mac_address: 54:EF:44:C1:D5:44
  on_ble_advertise:
    then:
      homeassistant.event:
        event: esphome.on_ble_advertise
        data:
          packet: !lambda return packet;

I have set up the component via HACS and I’m guessing that this will accept the events from that mac address via esphome, right?

I am seeing things like this in the ESPHome logs:

[18:33:35][D][ble_gateway:063]: [54:EF:44:C1:D5:44] Packet 043E330201000044D5C144EF54270201060F1695FE305897090144D5C144EF540813094D6920536D6F6B65204465746563746F7200D0
[18:33:37][D][ble_gateway:063]: [54:EF:44:C1:D5:44] Packet 043E330201000044D5C144EF54270201060F1695FE305897090144D5C144EF540813094D6920536D6F6B65204465746563746F7200D4
[18:33:39][D][ble_gateway:063]: [54:EF:44:C1:D5:44] Packet 043E330201000044D5C144EF54270201060F1695FE305897090144D5C144EF540813094D6920536D6F6B65204465746563746F7200D1
[18:33:40][D][ble_gateway:063]: [54:EF:44:C1:D5:44] Packet 043E330201000044D5C144EF54270201060F1695FE305897090144D5C144EF540813094D6920536D6F6B65204465746563746F7200D1

I added the device in the component with the MAC and the BLE KEY value for the Encryption Key value. Is that correct? Should I expect to see a device listed in the component if all is well? I am not seeing one.

Thank you for your patience with me. I think I’m pretty close.

EDIT I went into Node Red and put an event listener for esphome.on_ble_advertise event and I am definitely seeing these so I feel like the connection between the smoke detector → ESPHome BLE Gateway → Home Assistant is working

[myhomeiot]

@dinki

Please forgive me as I have tried looking in the logs, but how do I get the integration to use the values from my ESPHome BLE Gateway instead of trying to find a local BT device?

I’m glad that you finally solve all problems and get things working. Just to clarify, it’s a normal that local BT device and ESP receives same packets and send it to Passive BLE Monitor, this is nature of BLE (all nearby devices hear all BLE advertaisement packets), Passive BLE Monitor can handle this.

EDIT At event listener for esphome.on_ble_advertise event you should call Passive BLE Monitor service parse_data and send packet data received by ESPHome BLE Gateway for processing, in HA automation it’s looks like here:

- service: ble_monitor.parse_data
  data:
    packet: "{{ trigger.event.data.packet }}"

[myhomeiot]

@1technophile

OMG supports also 433 and IR along with BLE

BLE Gateway component discuses here is a component for ESPHome

[dinki]

Thank you for your response. To be clear, I do not think I have things working. Should I be seeing devices created for devices that are emitting home assistant events? I am not seeing any new devices created. I am doing this through the web ui. Should I be using YAML due to this not being a temperature sensor?