Background
I bought 8 of the newer Xiaomi bluetooth low energy temperature and humidty sensors. These are the versions that have encrypted payloads. These identify themselves as LYWSD03MMC flavor devices.
I did re-flash these with the custom firmware project without too much trouble, and they worked great. Just for clarity, after these devices got flashed with the custom firmware, I used the “Custom” advertising type, rather than the “Mi Like” alternative.
I knew that ESPHome on the ESP32 had specific support for these temperature/humidity devices, but I took a slightly different approach. I wanted to emulate the same arrangement I already use with Sonoff RF bridges, where the messages are they are received are just relayed to MQTT. My goal here was twofold:
- As I add new devices, I don’t need to rebuild the ESP32 firmware image as the devices are added or replace.
- I can add additional ESP32 devices in different parts of the house to get increased coverage areas. I don’t need to associate any particular sensor to one ESP32; any or all of the ESP32 devices can receive and relay the BLE announcements to MQTT. Code at the far end can drop duplicate messages if they’re present.
This has worked out pretty well! I have only a single ESP32 at the moment, though will be updating another one already in the house with some other sensors on it, to also act as a relay.
How does this work?
I use the esp32_ble_tracker component in ESPHome to listen for the BLE service announcements. Within that component, I use the on_ble_advertise “hook” to capture these service advertisement events and trigger running some code in a lambda.
Here’s an excerpt of the code. I admit this is a little ugly, and if I knew C++ and the standard library better, I could come up with something more elegant. But you’ll get a sense of it…
esp32_ble_tracker:
on_ble_advertise:
- then:
- lambda: |-
{
const char topicstr[] = "%s/%s_adv/%s/%s";
static char jsonstr[1024];
static char topic[100];
static char hexdata[150];
for (auto data : x.get_service_datas()) {
int i;
for (i = 0; i < 2 * data.data.size(); i+=2) {
sprintf(hexdata+i, "%02X", data.data[i/2]);
}
sprintf(jsonstr,
"{ \"mac\": \"%s\", \"service_data\": \"%s\", \"len\": %2d, \"data\": \"%s\" }",
x.address_str().c_str(), /* MAC address */
data.uuid.to_string().c_str(), /* UUID */
data.data.size(), /* length */
hexdata); /* hex representation of data */
sprintf(topic, topicstr,
id(mqtt_client).get_topic_prefix().c_str(),
"service",
x.address_str().c_str(), data.uuid.to_string().c_str());
id(mqtt_client).publish(topic, jsonstr);
}
for (auto data : x.get_manufacturer_datas()) {
int i;
for (i = 0; i < 2 * data.data.size(); i+=2) {
sprintf(hexdata+i, "%02X", data.data[i/2]);
}
sprintf(jsonstr,
"{ \"mac\": \"%s\", \"manufacturer_data\": \"%s\", \"len\": %2d, \"data\": \"%s\" }",
x.address_str().c_str(), /* MAC address */
data.uuid.to_string().c_str(), /* UUID */
data.data.size(), /* length */
hexdata); /* hex representation of data */
sprintf(topic, topicstr,
id(mqtt_client).get_topic_prefix().c_str(),
"manufacturer",
x.address_str().c_str(), data.uuid.to_string().c_str());
id(mqtt_client).publish(topic, jsonstr);
}
}
The general idea is that MQTT messages are published to a topic that ends with the MAC address and the service announcement UUID as the last two component of the MQTT topic. The payload is a chunk of JSON which include a hex encoding of the service announcement payload.
Example
Here’s what an MQTT message looks like, both topic and payload:
topic: 19916/ble_relay/service_adv/A4:C1:38:FC:C9:F9/18:1A
payload: { "mac": "A4:C1:38:FC:C9:F9", "service_data": "18:1A", "len": 13, "data": "A4C138FCC9F900AD224E0B5D1F" }
Home Assistant
On the Home Assistant side of things, you can subscribe to an MQTT topic (or topics) to receive these messages. Note that all of the BLE service announcements are relayed to MQTT; there’s quite a collection of these floating around from various device!
I found it useful to subscribe to a topic like this: 19916/ble_relay/service_adv/+/18:1A with a single MQTT path component wildcard for the MAC address, and looking for the 18:1A service type (which is what this custom firmware generates).
How
There’s a variety of ways these MQTT messages can be received and processed. You could imagine building a Node RED flow to receive these messages, parse the payloads and create entities within Home Assistant. Likewise in AppDaemon. Or even just as MQTT sensors with jinja2 templates to parse out the payload into temperature, humidity and battery values.
I ended up choosing to use pyscript to do this. It was a great excuse to learn how to use this custom component with a real application. I liked being able to write this code in python, too.
Code
Here’s the code! This is pretty much the first variation that actually works, there’s plenty of opportunity to improve it now that I’m more comfortable with how pyscript integrates…
# Parse BLE temperature/humidity sensor announcements
# Louis Mamakos
# [email protected]
#
# These service announcements have been relayed into MQTT
# by an ESP32 firmware image that receives any and all BLE
# service announcements.
#
app_name = 'ble_temperature'
UNKNOWNS = []
TRIGGERS = []
DEVICES = dict()
STATS = {
"dups": 0,
"wrong_len": 0,
"missing": 0,
"payload_error": 0,
"unknown": 0,
}
def update_ble_entities(mac, name, temperature, humidity,
battery_percent, battery_v, counter):
if DEVICES[mac]['attributes_initialized']:
state.set(DEVICES[mac]['temperature_entity'], temperature, counter=counter)
state.set(DEVICES[mac]['humidity_entity'], humidity, counter=counter)
state.set(DEVICES[mac]['battery_entity'], battery_percent,
battery_voltage=battery_v, counter=counter)
else:
# if static attributes haven't been set, create them now
log.info(f"Create entity attributes for {mac}/{name}")
DEVICES[mac]['attributes_initialized'] = True
temperature_attributes = {
"friendly_name": f"{name} Temperature",
"device_class": "temperature",
"unit_of_measurement": "°F",
"icon": "mdi:thermometer",
"mac": mac,
"counter": counter
}
state.set(DEVICES[mac]['temperature_entity'], temperature, temperature_attributes)
humidity_attributes = {
"friendly_name": f"{name} Humidity",
"device_class": "humidity",
"unit_of_measurement": "%",
"icon": "mdi:water-percent",
"mac": mac,
"counter": counter
}
state.set(DEVICES[mac]['humidity_entity'], humidity, humidity_attributes)
battery_attributes = {
"friendly_name": f"{name} Battery",
"device_class": "battery",
"unit_of_measurement": "%",
"icon": "mdi:battery-bluetooth-variant",
"mac": mac,
"counter": counter
}
state.set(DEVICES[mac]['battery_entity'], battery_percent, battery_attributes)
# called when an MQTT message is received on one of the subscribed topics.
def ble_message(**mqttmsg):
"""MQTT message from BLE relay"""
# message looks like { "mac": "A4:C1:38:DE:90:CE", "service_data": "18:1A", "len": 13, "data": "A4C138DE90CE00E3255A0BCE7B" }
if not 'payload_obj' in mqttmsg:
log.warning("payload not valid JSON")
STATS['payload_error'] += 1
return
m = mqttmsg['payload_obj']
mac = m['mac'].lower()
data = m['data']
if len(data) < 26:
STATS['wrong_len'] += 1
return
h_mac = data[0:12]
h_temp = round(int(data[12:16], 16)/10*1.8 + 32.0, 2)
h_humidity = int(data[16:18], 16)
h_battery_pct = int(data[18:20], 16)
h_battery_v = int(data[20:24], 16)/1000.0
h_counter = int(data[24:26], 16)
log.debug(f'MAC: {mac} mac={h_mac} h_temp={h_temp:.1f} h_humidity={h_humidity} h_battery_pct={h_battery_pct} h_battery_mv={h_battery_v} h_counter={h_counter}')
if mac in DEVICES:
# probably futile attempt to minimize race condition to avoid duplicate entity updates
old_counter = DEVICES[mac]['counter']
DEVICES[mac]['counter'] = h_counter
if h_counter != old_counter:
update_ble_entities(mac, DEVICES[mac]['name'],
h_temp, h_humidity, h_battery_pct, h_battery_v, h_counter)
else:
STATS['dups'] += 1
DEVICES[mac]['dups'] += 1
else:
STATS['unknown'] += 1
if not mac in UNKNOWNS:
log.info(f"Unknown MAC {mac}")
UNKNOWNS.append(mac)
# generate an MQTT topic trigger
def mqttTrigger(topic):
log.debug(f'Subscribing to topic {topic}')
@mqtt_trigger(topic)
def mqtt_message_fun(**kwargs):
ble_message(**kwargs)
return mqtt_message_fun
def initialize(cfg):
import re
for mac in cfg['devices']:
name = cfg['devices'][mac]['name']
# contruct an entity name, all lower case and replacing illegal characters with '_'
ent_name = re.sub(r'[^0-9A-Za-z]', '_', name.lower())
mac = mac.lower()
DEVICES[mac] = {
'temperature_entity': f'sensor.ble_{ent_name}_temperature',
'humidity_entity': f'sensor.ble_{ent_name}_humidity',
'battery_entity': f'sensor.ble_{ent_name}_battery',
'attributes_initialized': False,
'mac': mac,
'name': name,
'counter': -1,
'dups': 0,
'missing': 0
}
log.debug(f'config - {mac} as {ent_name} / {name}')
# subscribe topics
for topic in cfg['topics']:
TRIGGERS.append(mqttTrigger(topic))
if 'apps' in pyscript.config and app_name in pyscript.config['apps']:
initialize(pyscript.config['apps'][app_name])
else:
logger.warning(f'No {app_name} configuration found')
there’s some statistics counters in there which are not yet exposed, I’ll get around to doing something about that as I add another ESP32.
This bit of code is dropped into pyscript as an “app”, meaning there is external YAML configuration that’s passed to it. My configuration looks like this:
apps:
ble_temperature:
topics:
- '19916/ble_relay/service_adv/+/18:1A'
- '19916/ble_relay2/service_adv/+/18:1A'
devices:
A4:C1:38:A5:90:5C:
name: 'Master Bath'
A4:C1:38:D6:59:31:
name: '1st Floor Bath'
A4:C1:38:DE:90:CE:
name: 'Master Bedroom'
A4:C1:38:6A:4D:B6:
name: 'Blue Bedroom'
A4:C1:38:C1:A6:51:
name: '2nd Floor Office'
A4:C1:38:03:DB:59:
name: 'Kitchen'
A4:C1:38:FC:C9:F9:
name: 'Laundry Room'
A4:C1:38:58:E6:1D:
name: 'Attic'
As new devices are added, you can add their MAC address here, and as the service announcements are received, the python script will create and populate the entities for temperature, humidity, battery percent (with battery voltage attribute.)
You’ll note this fragment of code:
h_mac = data[0:12]
h_temp = round(int(data[12:16], 16)/10*1.8 + 32.0, 2)
h_humidity = int(data[16:18], 16)
h_battery_pct = int(data[18:20], 16)
h_battery_v = int(data[20:24], 16)/1000.0
h_counter = int(data[24:26], 16)
that parses out the values from the payload. I use the int() function with the radix specified as 16 to parse the hex characters in the payload part of the message. This is also described in the github page for that custom firmware if you want to adapt this to a different solution.
Summary
So far, so good!
I’ve had it working for a whole 2 days now! And it’s working well enough that I ordered another 8 of these sensors to scatter around into other places. At less than $5 each, it’s quite a good deal. And they have an LCD display you can look at, and are attractive enough that my wife doesn’t hate that I’ve stuck them to the walls in various places.
I’m not suggesting this is a better solution than the in-built capability in ESPHome. It scratches my itch of treating these sensors differently. And in particular, in treating the ESP32/ESPHome part of this solution as a generic, dumb BLE service announcement relay that really has no per-device configuration state.
By making the ESP32 “dumb”, this allows me to freely spread the ESP32 “relay” devices around my home to get coverage without worrying about binding a specific sensor to a specific ESP32. And I’m already using the MQTT intergration method with ESPHome vs. using the API method and component because I like being able to monitor the MQTT traffic when debugging stuff.
I hope you found this of interest.