A new approach for Xiaomi BLE Temperature sensors with ESPHome, MQTT and pyscript

Background

I bought 8 of the newer Xiaomi bluetooth low energy temperature and humidty sensors. These are the versions that have encrypted payloads. These identify themselves as LYWSD03MMC flavor devices.

I did re-flash these with the custom firmware project without too much trouble, and they worked great. Just for clarity, after these devices got flashed with the custom firmware, I used the “Custom” advertising type, rather than the “Mi Like” alternative.

I knew that ESPHome on the ESP32 had specific support for these temperature/humidity devices, but I took a slightly different approach. I wanted to emulate the same arrangement I already use with Sonoff RF bridges, where the messages are they are received are just relayed to MQTT. My goal here was twofold:

  1. As I add new devices, I don’t need to rebuild the ESP32 firmware image as the devices are added or replace.
  2. I can add additional ESP32 devices in different parts of the house to get increased coverage areas. I don’t need to associate any particular sensor to one ESP32; any or all of the ESP32 devices can receive and relay the BLE announcements to MQTT. Code at the far end can drop duplicate messages if they’re present.

This has worked out pretty well! I have only a single ESP32 at the moment, though will be updating another one already in the house with some other sensors on it, to also act as a relay.

How does this work?

I use the esp32_ble_tracker component in ESPHome to listen for the BLE service announcements. Within that component, I use the on_ble_advertise “hook” to capture these service advertisement events and trigger running some code in a lambda.

Here’s an excerpt of the code. I admit this is a little ugly, and if I knew C++ and the standard library better, I could come up with something more elegant. But you’ll get a sense of it…

esp32_ble_tracker:
  on_ble_advertise:
    - then:
        - lambda: |-
           {
              const char topicstr[] = "%s/%s_adv/%s/%s";
              static char jsonstr[1024];
              static char topic[100];
              static char hexdata[150];

              for (auto data : x.get_service_datas()) {
                  int i;
                  for (i = 0; i < 2 * data.data.size(); i+=2) {
                    sprintf(hexdata+i, "%02X", data.data[i/2]);
                  }
                  sprintf(jsonstr,
                    "{ \"mac\": \"%s\", \"service_data\": \"%s\", \"len\": %2d, \"data\": \"%s\" }",
                      x.address_str().c_str(),    /* MAC address */
                      data.uuid.to_string().c_str(), /* UUID */
                      data.data.size(),           /* length */
                      hexdata);                   /* hex representation of data */

                  sprintf(topic, topicstr, 
                          id(mqtt_client).get_topic_prefix().c_str(),
                          "service", 
                          x.address_str().c_str(), data.uuid.to_string().c_str());

                  id(mqtt_client).publish(topic, jsonstr);
              }

              for (auto data : x.get_manufacturer_datas()) {
                  int i;
                  for (i = 0; i < 2 * data.data.size(); i+=2) {
                    sprintf(hexdata+i, "%02X", data.data[i/2]);
                  }
                  sprintf(jsonstr,
                    "{ \"mac\": \"%s\", \"manufacturer_data\": \"%s\", \"len\": %2d, \"data\": \"%s\" }",
                    x.address_str().c_str(),        /* MAC address */
                    data.uuid.to_string().c_str(),  /* UUID */
                    data.data.size(),               /* length */
                    hexdata);                       /* hex representation of data */

                  sprintf(topic, topicstr,
                          id(mqtt_client).get_topic_prefix().c_str(),
                          "manufacturer",
                          x.address_str().c_str(), data.uuid.to_string().c_str());

                  id(mqtt_client).publish(topic, jsonstr);                  
              }
            }

The general idea is that MQTT messages are published to a topic that ends with the MAC address and the service announcement UUID as the last two component of the MQTT topic. The payload is a chunk of JSON which include a hex encoding of the service announcement payload.

Example

Here’s what an MQTT message looks like, both topic and payload:
topic: 19916/ble_relay/service_adv/A4:C1:38:FC:C9:F9/18:1A
payload: { "mac": "A4:C1:38:FC:C9:F9", "service_data": "18:1A", "len": 13, "data": "A4C138FCC9F900AD224E0B5D1F" }

Home Assistant

On the Home Assistant side of things, you can subscribe to an MQTT topic (or topics) to receive these messages. Note that all of the BLE service announcements are relayed to MQTT; there’s quite a collection of these floating around from various device!

I found it useful to subscribe to a topic like this: 19916/ble_relay/service_adv/+/18:1A with a single MQTT path component wildcard for the MAC address, and looking for the 18:1A service type (which is what this custom firmware generates).

How

There’s a variety of ways these MQTT messages can be received and processed. You could imagine building a Node RED flow to receive these messages, parse the payloads and create entities within Home Assistant. Likewise in AppDaemon. Or even just as MQTT sensors with jinja2 templates to parse out the payload into temperature, humidity and battery values.

I ended up choosing to use pyscript to do this. It was a great excuse to learn how to use this custom component with a real application. I liked being able to write this code in python, too.

Code

Here’s the code! This is pretty much the first variation that actually works, there’s plenty of opportunity to improve it now that I’m more comfortable with how pyscript integrates…

#  Parse BLE temperature/humidity sensor announcements
#  Louis Mamakos
#  [email protected]
#
#  These service announcements have been relayed into MQTT
#  by an ESP32 firmware image that receives any and all BLE
#  service announcements.
#

app_name = 'ble_temperature'

UNKNOWNS = []
TRIGGERS = []
DEVICES = dict()
STATS = {
    "dups":      0,
    "wrong_len": 0,
    "missing":   0,
    "payload_error": 0,
    "unknown":   0,
}

def update_ble_entities(mac, name, temperature, humidity,
                        battery_percent, battery_v, counter):
    if DEVICES[mac]['attributes_initialized']:
        state.set(DEVICES[mac]['temperature_entity'], temperature, counter=counter)
        state.set(DEVICES[mac]['humidity_entity'], humidity, counter=counter)
        state.set(DEVICES[mac]['battery_entity'], battery_percent,
                      battery_voltage=battery_v, counter=counter)
    else:
        # if static attributes haven't been set, create them now
        log.info(f"Create entity attributes for {mac}/{name}")
        DEVICES[mac]['attributes_initialized'] = True

        temperature_attributes = {
            "friendly_name": f"{name} Temperature",
            "device_class": "temperature",
            "unit_of_measurement": "°F",
            "icon": "mdi:thermometer",
            "mac": mac,
            "counter": counter
        }
        state.set(DEVICES[mac]['temperature_entity'], temperature, temperature_attributes)        

        humidity_attributes = {
            "friendly_name": f"{name} Humidity",
            "device_class": "humidity",
            "unit_of_measurement": "%",
            "icon": "mdi:water-percent",
            "mac": mac,
            "counter": counter
        }
        state.set(DEVICES[mac]['humidity_entity'], humidity, humidity_attributes)
        
        battery_attributes = {
            "friendly_name": f"{name} Battery",
            "device_class": "battery",
            "unit_of_measurement": "%",
            "icon": "mdi:battery-bluetooth-variant",
            "mac": mac,
            "counter": counter            
        }
        state.set(DEVICES[mac]['battery_entity'], battery_percent, battery_attributes)


# called when an MQTT message is received on one of the subscribed topics.
def ble_message(**mqttmsg):
    """MQTT message from BLE relay"""
    
    # message looks like  { "mac": "A4:C1:38:DE:90:CE", "service_data": "18:1A", "len": 13, "data": "A4C138DE90CE00E3255A0BCE7B" }
    
    if not 'payload_obj' in mqttmsg:
        log.warning("payload not valid JSON")
        STATS['payload_error'] += 1
        return

    m = mqttmsg['payload_obj']
    mac = m['mac'].lower()

    data = m['data']
    if len(data) < 26:
        STATS['wrong_len'] += 1
        return

    h_mac         = data[0:12]
    h_temp        = round(int(data[12:16], 16)/10*1.8 + 32.0, 2)
    h_humidity    = int(data[16:18], 16)
    h_battery_pct = int(data[18:20], 16)
    h_battery_v   = int(data[20:24], 16)/1000.0
    h_counter     = int(data[24:26], 16)

    log.debug(f'MAC: {mac} mac={h_mac} h_temp={h_temp:.1f} h_humidity={h_humidity} h_battery_pct={h_battery_pct} h_battery_mv={h_battery_v} h_counter={h_counter}')
    if mac in DEVICES:
        # probably futile attempt to minimize race condition to avoid duplicate entity updates
        old_counter = DEVICES[mac]['counter']
        DEVICES[mac]['counter'] = h_counter
        if h_counter != old_counter:
            update_ble_entities(mac, DEVICES[mac]['name'], 
                                h_temp, h_humidity, h_battery_pct, h_battery_v, h_counter)
        else:
            STATS['dups'] += 1
            DEVICES[mac]['dups'] += 1
    else:
        STATS['unknown'] += 1
        if not mac in UNKNOWNS:
            log.info(f"Unknown MAC {mac}")
            UNKNOWNS.append(mac)

# generate an MQTT topic trigger 
def mqttTrigger(topic):
    log.debug(f'Subscribing to topic {topic}')

    @mqtt_trigger(topic)
    def mqtt_message_fun(**kwargs):
        ble_message(**kwargs)

    return mqtt_message_fun

def initialize(cfg):
    import re

    for mac in cfg['devices']:
        name = cfg['devices'][mac]['name']
        # contruct an entity name, all lower case and replacing illegal characters with '_'
        ent_name = re.sub(r'[^0-9A-Za-z]', '_', name.lower())
        mac = mac.lower()
        DEVICES[mac] = {
            'temperature_entity':   f'sensor.ble_{ent_name}_temperature',
            'humidity_entity':      f'sensor.ble_{ent_name}_humidity',            
            'battery_entity':       f'sensor.ble_{ent_name}_battery',
            'attributes_initialized': False,
            'mac':                  mac,
            'name':                 name,
            'counter':              -1,
            'dups':                 0,
            'missing':              0
        }
        log.debug(f'config - {mac} as {ent_name} / {name}')

    # subscribe topics
    for topic in cfg['topics']:
        TRIGGERS.append(mqttTrigger(topic))


if 'apps' in pyscript.config and app_name in pyscript.config['apps']:
    initialize(pyscript.config['apps'][app_name])
else:
    logger.warning(f'No {app_name} configuration found')

there’s some statistics counters in there which are not yet exposed, I’ll get around to doing something about that as I add another ESP32.

This bit of code is dropped into pyscript as an “app”, meaning there is external YAML configuration that’s passed to it. My configuration looks like this:

apps:
  ble_temperature:
    topics:
      - '19916/ble_relay/service_adv/+/18:1A'
      - '19916/ble_relay2/service_adv/+/18:1A'
    devices:
      A4:C1:38:A5:90:5C:
        name: 'Master Bath'
      A4:C1:38:D6:59:31:
        name: '1st Floor Bath'
      A4:C1:38:DE:90:CE:
        name: 'Master Bedroom'
      A4:C1:38:6A:4D:B6:
        name: 'Blue Bedroom'
      A4:C1:38:C1:A6:51:
        name: '2nd Floor Office'
      A4:C1:38:03:DB:59:
        name: 'Kitchen'
      A4:C1:38:FC:C9:F9:
        name: 'Laundry Room'
      A4:C1:38:58:E6:1D:
        name: 'Attic'

As new devices are added, you can add their MAC address here, and as the service announcements are received, the python script will create and populate the entities for temperature, humidity, battery percent (with battery voltage attribute.)

You’ll note this fragment of code:

    h_mac         = data[0:12]
    h_temp        = round(int(data[12:16], 16)/10*1.8 + 32.0, 2)
    h_humidity    = int(data[16:18], 16)
    h_battery_pct = int(data[18:20], 16)
    h_battery_v   = int(data[20:24], 16)/1000.0
    h_counter     = int(data[24:26], 16)

that parses out the values from the payload. I use the int() function with the radix specified as 16 to parse the hex characters in the payload part of the message. This is also described in the github page for that custom firmware if you want to adapt this to a different solution.

Summary

So far, so good!

I’ve had it working for a whole 2 days now! And it’s working well enough that I ordered another 8 of these sensors to scatter around into other places. At less than $5 each, it’s quite a good deal. And they have an LCD display you can look at, and are attractive enough that my wife doesn’t hate that I’ve stuck them to the walls in various places.

I’m not suggesting this is a better solution than the in-built capability in ESPHome. It scratches my itch of treating these sensors differently. And in particular, in treating the ESP32/ESPHome part of this solution as a generic, dumb BLE service announcement relay that really has no per-device configuration state.

By making the ESP32 “dumb”, this allows me to freely spread the ESP32 “relay” devices around my home to get coverage without worrying about binding a specific sensor to a specific ESP32. And I’m already using the MQTT intergration method with ESPHome vs. using the API method and component because I like being able to monitor the MQTT traffic when debugging stuff.

I hope you found this of interest.

10 Likes

Excellent work. I have a number of these sensors and two esp32’s set up. It is a pain to hardwire the MAC addresses for each sensor. My scripting skills are very average. Are you able to post this all to gitbhub or similar with a bit more intel about setup?

Thanks! I’m still tweaking the pyscript code a bit. There’s some things I want to do in there:

  • I want to check for gaps in the counter value that the sensors transmit and count missing updates. This seems like it might be useful to understand if the sensors are close enough to one of the ESP32 “relay” devices, or more need to be added.
  • I want to expose the stats counters for duplicate updates (received by more than one ESP32, or transmitted more than once)
  • I want to persist the sensor values. Right now, when you restart Home Assistant, the sensors do not appear until the first message is sent. I didn’t want to create the sensors ahead of having a value I can plug in there. There is a way that I can create another pyscript.foo entity and store all the states in there as part of it’s attributes, and have Home Assistant persist (and restore) that when restarted. The goal here to avoid having ugly errors in Lovelace dashboards that refer to sensor entities that don’t yet exist.

This thing exists in two distinct pieces - the ESPHome configuration for the ESP32; you should be able to plug in the one code fragment (really, in your case @kiwipaul, the on_ble_advertise: section and lambda within the device tracker).

The other bit is the pyscript code and “app” configuration, which I didn’t do a great job of explaining. There are some very good docs on the pyscript integration that should fill in the context of how to install it and where the configuration would go.

Roughly speaking, in my Home Assistant configuration.yaml file, I have this:

# https://github.com/custom-components/pyscript
pyscript: !include pyscript/config.yaml

The pyscript code I posted above lives in /config/pyscript/apps/ble_temperature.py
and then in my /config/pyscript/config.yaml file, this:

hass_is_global: true
allow_all_imports: true
apps:
  ble_temperature:
    topics:
      - '19916/ble_relay/service_adv/+/18:1A'
      - '19916/ble_relay2/service_adv/+/18:1A'  
      - '19916/freezer/service_adv/+/18:1A'
      - '19916/ble_observatory/service_adv/+/18:1A'
    devices:
      # MAC prefix is A4:C1:38:
      A4:C1:38:A5:90:5C:
        name: 'Master Bath'
      A4:C1:38:D6:59:31:
        name: '1st Floor Bath'
      A4:C1:38:DE:90:CE:
        name: 'Master Bedroom'
      A4:C1:38:6A:4D:B6:
        name: 'Blue Bedroom'
      A4:C1:38:C1:A6:51:
        name: '2nd Floor Office'
      A4:C1:38:03:DB:59:
        name: 'Kitchen'
      A4:C1:38:FC:C9:F9:
        name: 'Laundry Room'
      A4:C1:38:58:E6:1D:
        name: 'Attic'
      A4:C1:38:3E:D7:97:
        name: 'Obs Warm Room'
      A4:C1:38:07:B3:1E:
        name: 'Obs Telescope Room'

and that’s just about it. You should, of course, season to taste with your own MAC addresses and topic names. There’s nothing special about the 19916 prefix; that’s just a local convention I use for all my Home Automation MQTT topics, and should match whatever you use in your ESPHome configuration.

The pyscript code takes the names and constructs entity_ids like this:

Master Bath

  • sensor.ble_master_bath_battery
  • sensor.ble_master_bath_humidity
  • sensor.ble_master_bath_temperature

It’s up to you to ensure there are not any name conflicts. Similar to Home Assistant, the name has any character that’s not 0-9, A-Z, a-z turned into and _ character.

As far as publishing this… the ESPHome configuration is in an entirely different place than my Home Assistant configuration (for which I don’t publish the repo publically.) So in any case, I’d need to do some sort of ad-hoc publishing scheme, somewhere.

I really recommend that you play with pyscript. It’s really quite fun to use, especially if you use the jupyter notebook extension described in the pyscript docs. You can then interactively play with code using an interactive jupyter notebook web page… this is just an amazing and productive way to learn how the use the code. Including watching triggers on automations fire. Highly recommended!

By the way, I’d also love to be able to do something like this in ESPHome with Dallas 1-wire bus temperature sensors. It would be great if it would just read all the sensors that it found and published them to individual topics. I hate that so much of the configuration in ESPHome is compiled into the image… well all of the configuration, I suppose… This is annoying because for this project, I have 3 instances, all configured the same except for some top-level name for an MQTT prefix.

In my use case, I generally always use MQTT and manual entity configuration; this could be at odds with the style of integration using the “api” method to talk to the Home Assistant ESPHome integration (which I don’t use.) I build all my ESPHome deployments outside of Home Assistant since I already have platformio installed for other purposes.

This is great functionality, a great use of pyscript, and very nice code. Thanks for sharing!!

1 Like

Just a heads up for others. If you happen to be trying to use this with the pvvx firmware, make sure to set the advertising type to Atc1441.

The Custom type uses a different format than the original atc1441 firmware.

I hacked in support for LYWSDCGQ sensors, and I do mean hacked. This is not some of my prettiest work.

#  Parse BLE temperature/humidity sensor announcements
#  Louis Mamakos
#  [email protected]
#
#  These service announcements have been relayed into MQTT
#  by an ESP32 firmware image that receives any and all BLE
#  service announcements.
#

app_name = 'ble_temperature'

UNKNOWNS = []
TRIGGERS = []
DEVICES = dict()
STATS = {
    "dups":      0,
    "wrong_len": 0,
    "missing":   0,
    "payload_error": 0,
    "unknown":   0,
}

def update_ble_entities(mac, name, temperature, humidity, battery_percent, battery_v, counter):
    if DEVICES[mac]['attributes_initialized']:
        log.info(f"Updating entity attributes for {mac}/{name}")
        if temperature is not None:
            state.set(DEVICES[mac]['temperature_entity'], temperature, counter=counter)
        if humidity is not None:
            state.set(DEVICES[mac]['humidity_entity'], humidity, counter=counter)
        if battery_percent is not None:
            if battery_v is not None:
                state.set(DEVICES[mac]['battery_entity'], battery_percent, battery_voltage=battery_v, counter=counter)
            else:
                state.set(DEVICES[mac]['battery_entity'], battery_percent, counter=counter)
    else:
        # if static attributes haven't been set, create them now
        log.info(f"Create entity attributes for {mac}/{name}")
        DEVICES[mac]['attributes_initialized'] = True

        temperature_attributes = {
            "friendly_name": f"{name} Temperature",
            "device_class": "temperature",
            "unit_of_measurement": "°F",
            "icon": "mdi:thermometer",
            "mac": mac,
            "counter": counter
        }
        state.set(DEVICES[mac]['temperature_entity'], temperature, temperature_attributes)        

        humidity_attributes = {
            "friendly_name": f"{name} Humidity",
            "device_class": "humidity",
            "unit_of_measurement": "%",
            "icon": "mdi:water-percent",
            "mac": mac,
            "counter": counter
        }
        state.set(DEVICES[mac]['humidity_entity'], humidity, humidity_attributes)
        
        battery_attributes = {
            "friendly_name": f"{name} Battery",
            "device_class": "battery",
            "unit_of_measurement": "%",
            "icon": "mdi:battery-bluetooth-variant",
            "mac": mac,
            "counter": counter
        }
        state.set(DEVICES[mac]['battery_entity'], battery_percent, battery_attributes)


# called when an MQTT message is received on one of the subscribed topics.
def ble_message(**mqttmsg):
    """MQTT message from BLE relay"""
    
    # message looks like  { "mac": "A4:C1:38:DE:90:CE", "service_data": "18:1A", "len": 13, "data": "A4C138DE90CE00E3255A0BCE7B" }
    
    if not 'payload_obj' in mqttmsg:
        log.warning("payload not valid JSON")
        STATS['payload_error'] += 1
        return

    m = mqttmsg['payload_obj']
    mac = m['mac'].lower()

    data = m['data']
    service_type = m['service_data']
    
    if '18:1A' in service_type or '0x181A' in service_type:
        if len(data) < 26:
            STATS['wrong_len'] += 1
            return
        h_mac         = data[0:12]
        h_temp        = round(int(data[12:16], 16)/10*1.8 + 32.0, 1)
        h_humidity    = int(data[16:18], 16)
        h_battery_pct = int(data[18:20], 16)
        h_battery_v   = int(data[20:24], 16)/1000.0
        h_counter     = int(data[24:26], 16)
        log.debug(f'MAC: {mac} mac={h_mac} h_temp={h_temp:.1f} h_humidity={h_humidity} h_battery_pct={h_battery_pct} h_battery_mv={h_battery_v} h_counter={h_counter}')
        if mac in DEVICES:
            # probably futile attempt to minimize race condition to avoid duplicate entity updates
            old_counter = DEVICES[mac]['counter']
            DEVICES[mac]['counter'] = h_counter
            if h_counter != old_counter:
                update_ble_entities(mac, DEVICES[mac]['name'], 
                                    h_temp, h_humidity, h_battery_pct, h_battery_v, h_counter)
            else:
                STATS['dups'] += 1
                DEVICES[mac]['dups'] += 1
        else:
            STATS['unknown'] += 1
            if not mac in UNKNOWNS:
                log.info(f"Unknown MAC {mac}")
                UNKNOWNS.append(mac)

    elif 'FE:95' in service_type or '0xFE95' in service_type:
        if len(data) < 30:
            STATS['wrong_len'] += 1
            return
        h_mac = reverse_bytes(data[10:22])
        value_type = data[22:24]
        if value_type == '0D':
            value_length = int(data[26:28])
            if value_length != 4:
                log.warning(f'Value type {value_type} does not match value length {value_length}')
                return
            value_data = data[28:(28+value_length*2)]
            h_temp        = round(int(reverse_bytes(value_data[0:4]), 16)/10*1.8 + 32.0, 1)
            h_humidity    = int(reverse_bytes(value_data[4:8]), 16)/10
            h_counter     = int(data[8:10], 16)
            log.debug(f'MAC: {mac} mac={h_mac} h_temp={h_temp:.1f} h_humidity={h_humidity} h_counter={h_counter}')
            if mac in DEVICES:
                # probably futile attempt to minimize race condition to avoid duplicate entity updates
                old_counter = DEVICES[mac]['counter']
                DEVICES[mac]['counter'] = h_counter
                if h_counter != old_counter:
                    update_ble_entities(mac, DEVICES[mac]['name'], 
                                        h_temp, h_humidity, None, None, h_counter)
                else:
                    STATS['dups'] += 1
                    DEVICES[mac]['dups'] += 1
            else:
                STATS['unknown'] += 1
                if not mac in UNKNOWNS:
                    log.info(f"Unknown MAC {mac}")
                    UNKNOWNS.append(mac)
        elif value_type == '0A':
            value_length = int(data[26:28])
            if value_length != 1:
                log.warning(f'Value type {value_type} does not match value length {value_length}')
                return
            value_data = data[28:(28+value_length*2)]
            h_battery_pct = int(value_data, 16)
            h_counter     = int(data[8:10], 16)
            log.debug(f'MAC: {mac} mac={h_mac} h_battery_pct={h_battery_pct} h_counter={h_counter}')
            if mac in DEVICES:
                # probably futile attempt to minimize race condition to avoid duplicate entity updates
                old_counter = DEVICES[mac]['counter']
                DEVICES[mac]['counter'] = h_counter
                if h_counter != old_counter:
                    update_ble_entities(mac, DEVICES[mac]['name'], 
                                        None, None, h_battery_pct, None, h_counter)
                else:
                    STATS['dups'] += 1
                    DEVICES[mac]['dups'] += 1
            else:
                STATS['unknown'] += 1
                if not mac in UNKNOWNS:
                    log.info(f"Unknown MAC {mac}")
                    UNKNOWNS.append(mac)
        else:
            log.info(f'Unknown value_type {value_type}')
    else:
        log.info(f'Unknown data format {data}')

def reverse_bytes(bstring):
    bstring = ''.join(reversed([bstring[i:i+2] for i in range(0, len(bstring), 2)]))
    return bstring

# generate an MQTT topic trigger 
def mqttTrigger(topic):
    log.debug(f'Subscribing to topic {topic}')

    @mqtt_trigger(topic)
    def mqtt_message_fun(**kwargs):
        ble_message(**kwargs)

    return mqtt_message_fun

def initialize(cfg):
    import re

    for mac in cfg['devices']:
        name = cfg['devices'][mac]['name']
        # contruct an entity name, all lower case and replacing illegal characters with '_'
        ent_name = re.sub(r'[^0-9A-Za-z]', '_', name.lower())
        mac = mac.lower()
        DEVICES[mac] = {
            'temperature_entity':   f'sensor.ble_{ent_name}_temperature',
            'humidity_entity':      f'sensor.ble_{ent_name}_humidity',            
            'battery_entity':       f'sensor.ble_{ent_name}_battery',
            'attributes_initialized': False,
            'mac':                  mac,
            'name':                 name,
            'counter':              -1,
            'dups':                 0,
            'missing':              0
        }
        log.debug(f'config - {mac} as {ent_name} / {name}')

    # subscribe topics
    for topic in cfg['topics']:
        TRIGGERS.append(mqttTrigger(topic))

if 'apps' in pyscript.config and app_name in pyscript.config['apps']:
    initialize(pyscript.config['apps'][app_name])
else:
    log.warning(f'No {app_name} configuration found')

Also, to keep my pyscript config concise, I reduced my topics block to the following…

    topics:
      - '+/service_adv/+/0x181A'
      - '+/service_adv/+/0xFE95'

The 0xFE95 topic is what the older, unencrypted sensors use. The tricky part was that these sensors send different payloads depending on if they’re reporting battery, or temperature with humidity. They don’t send all of the information at the same time.

Also, for some reason, my topics are slightly differently than OPs, perhaps due to differences in the type of MQTT broker used. Some tweakage may be needed for others.

My topics are different than default due to local circumstances; I have that 19916/ prefix on topic strings just because.

Glad you were able to find this useful. Having multiple ESP32 systems forwarding the BLE announcements into MQTT make arranging for adequate coverage around the house really easy, and you don’t have to figure out what sensor is heard by what ESP32.

Thank you Louis, this is great! I set up this configuration with an ESP32 and 10 Xiaomi BLE temp sensors, and it’s working nicely. Really appreciate you taking the time to share your work!

One thing I noticed after setting this up was that the logbook integration was showing a ton of file_ble_temperature_mqtt_message_fun has been triggered by mqtt messages. I think an event is being recorded for every mqtt event / ble broadcast.

To understand more, these queries show how many events are from the pyscript:

$ sqlite3 home-assistant_v2.db "select count(*) from events where event_data like '%pyscript.file_ble_temperature_mqtt_message_fun%';"
59123
$ sqlite3 home-assistant_v2.db "select count(*) from events;"
165063
$ 

For me, over a third! And this is after <24hr of setting up the system.

To stop recording these events (I don’t think they’re useful, and am worried about the load on my Raspberry Pi’s SD card), I added this to my configuration.yaml:

recorder:
    exclude:
        entities:
          - "pyscript.file_ble_temperature_mqtt_message_fun"

Hopefully this is useful to others also.

1 Like

That’s a great suggestion to exclude those events from the recorder database! Thanks for passing that along.

Here’s the section of pyscript needed to parse the Custom format. Might be a better way to do this in python. I left it in Celcius, you will have to update the attributes if that’s the case for you.

    h_mac         = data[0:12]
    h_temp        = int(data[14:16] + data[12:14], 16) / 100.0
    h_humidity    = int(data[18:20] + data[16:18], 16) / 100.0
    h_battery_v   = int(data[22:24] + data[20:22], 16) / 1000.0
    h_battery_pct = int(data[24:26], 16)
    h_counter     = int(data[26:28], 16)

Hi, Looking at the for something I wan’t to do with BLE distance… where is this coming from?

Figured this out id of the mqtt: