ESP32 based BLE tracking for mqtt_room

Share your Projects!
112

The config you posted has a max distance setting of 4m. This is the cut off distance, so the ESP will not report any devices it thinks are further away. If you want it to report everything, set that value to zero and it will report every advertising packet it sees in the scan.

The idea of the max distance setting is to ensure that your ESP isn’t picking up advertising packets that are outside the room in which the device is located. For small rooms, I’ve found that a setting of 3m is good, while larger rooms will need a higher setting. Play around with the max distance, as well as the transmit power and measured power on your phone and see what results you can get.

I’ve tested with a Mi Band 2, and it does respond to a scan with a useable message, so it should work. It’s not strictly an iBeacon format, but since it doesn’t change its bluetooth hardware MAC address in the scan responses (the way phones do), it should work fine. Note that the configuration will be slightly different, the device ID in your sensors.yaml will be just the hardware MAC address, without a major or minor version number.

113

Okay, so i’ve tried to get this working for a couple of days.
I’ve read the documentation several times, upside down even.
I see the ESP’s in my ARP in the router, I see it in the unifi controller aswell.
They get IP and sends packets…

I got this in my hassio MQTT Broker addon:

1573655959: New client connected from 192.168.1.xx as esp32_2 (p2, c1, k60, u'username').
1573655962: New connection from 192.168.1.xx on port 1883.
1573655962: Client esp32_2 already connected, closing old connection.
1573655962: New client connected from 192.168.1.xx as esp32_2 (p2, c1, k60, u'username').
1573655985: New connection from 192.168.1.xx on port 1883.
1573655985: New client connected from 192.168.1.xx as nvr2112 (p1, c1, k60, u'username').
1573655985: Client nvr2112 disconnected.
1573655996: New connection from 192.168.1.xx on port 1883.
1573655996: New client connected from 192.168.1.xx as nvr2125 (p1, c1, k60, u'username').
1573655996: Client nvr2125 disconnected.
1573656098: New connection from 192.168.1.xx on port 1883.
1573656098: Client esp32_1 already connected, closing old connection.
1573656098: New client connected from 192.168.1.xx as esp32_1 (p2, c1, k60, u'username').
1573656130: New connection from 192.168.1.xx on port 1883.
[INFO] found username on local database
1573656130: Client esp32_1 already connected, closing old connection.
1573656130: New client connected from 192.168.1.xx as esp32_1 (p2, c1, k60, u'username').
1573656137: New connection from 192.168.1.xx on port 1883.
1573656137: Client esp32_2 already connected, closing old connection.
1573656137: New client connected from 192.168.1.xx as esp32_2 (p2, c1, k60, u'username').
1573656173: New connection from 192.168.1.xx on port 1883.
1573656173: Client esp32_1 already connected, closing old connection.
1573656173: New client connected from 192.168.1.xx as esp32_1 (p2, c1, k60, u'username').
1573656180: New connection from 192.168.1.xx on port 1883.
1573656180: Client esp32_1 already connected, closing old connection.
1573656180: New client connected from 192.168.1.xx as esp32_1 (p2, c1, k60, u'username').
1573656212: New connection from 192.168.1.xx on port 1883.
1573656212: Client esp32_3 already connected, closing old connection.
1573656212: New client connected from 192.168.1.xx as esp32_3 (p2, c1, k60, u'username').

The binary sensor is disconnected. Why? In Atom it says:

devices_discovered: 14
devices_reported: 2
Telemetry sent
Scanning...

More from Atom serial monitor:

Connecting to WiFi...
[WiFi-event] event: 0
Wifi Ready
restarted
[WiFi-event] event: 2
STA Start
WiFi Time remaining: 1999
[WiFi-event] event: 4
[WiFi-event] event: 7
IP address:     192.168.1.xx
Hostname:       esp32_3
Connecting to MQTT
Stopping wifi reconnect timer
Scanning...     Connected to MQTT.
Success sending message to topic:       presence_nodes/Sovrummet
Telemetry sent
Scan done! Devices found: 11

binary_sensor.yaml

platform: mqtt
name: ESP32 3
state_topic: "presence_nodes/esp32_3"
json_attributes_topic: "presence_nodes/esp32_3/tele"
payload_on: "CONNECTED"
payload_off: "DISCONNECTED"
device_class: connectivity

sensor.yaml

platform: mqtt_room
device_id: "2F234454-CF6D-4A0F-ADF2-F4911BA9FFA6-1-0"
name: 'Oskars_bt_mqtt'
state_topic: 'room_presence'
timeout: 10
away_timeout: 15
114

You’ll see this regularly in the MQTT logs, as due to the need to use antenna sharing, the WiFi has to be temporarily put into a lower priority than the bluetooth for the duration of the scan. The MQTT connection can suffer. The time is not long enough to trigger a disconnect, but when the connection gets re-established, I see this show in my log. You’ll notice other devices crop up in there with less regularity.

…or that’s what I’ve convinced myself is the case.

What’s concerning to me is the frequency at which this happens. What is your config for scan duration and wait time? Can you paste your config (minus any secret info) for the module or modules. Currently I don’t do any linting to check that they’re not misconfigured in such a way as to cause a repeated scan loop.

115

Hi,

this is my Settings_local.h:
https://pastebin.com/xxUX7xbV

I hope it was the settings file you wanted to see.

1 Like
116

That was, and it all looks fine to me.

I just noticed in your sensor.yaml configuration, you’ve included dashes in the device id. You should strip those out and leave in only the ones separating the ID from the major and minor version numbers, like this:

platform: mqtt_room
device_id: 2F234454CF6D4A0FADF2F4911BA9FFA6-1-0
name: 'Oskars_bt_mqtt'
state_topic: 'room_presence'
timeout: 10
away_timeout: 15

Is the binary sensor showing DISCONNECTED the only issue you’re encountering, or are there other problems as well?

117

The problem I have is that the binary sensor says DISCONNECTED and that the sensor shows not_home.

Even when I changed the dashes. :pensive:

Edit: nevermind, the only problem now is the DISCONNECTED part. Now it shows distance and what esp Im connected to.

Edit2: I have 4 meters in the settings file, but it will not connect if Im more then 1 meter away. Is there any tips for that?

118

Okay, glad we’ve got partial success.

Next, can you subscribe to the mqtt topic presence_nodes/# and paste the results? It woultd show us the payloads for all the esp32s and their telemetry as well.

I’m currently working on a fix for distance calculations for generic BLE devices, as the default used in the code isn’t very accurate.

119

Thank you for your patience. :slight_smile:

Here is some of the output:

presence_nodes/esp32_2/tele {"room":"Koket","ip":"192.168.1.x","hostname":"esp32_2","scan_dur":5,"wait_dur":5,"max_dist":4,"disc_ct":26,"rept_ct":3}
presence_nodes/Köket CONNECTED
presence_nodes/Koket CONNECTED
presence_nodes/Kallaren CONNECTED
presence_nodes/esp32_1/tele {"room":"Kallaren","ip":"192.168.1.x","hostname":"esp32_1","scan_dur":5,"wait_dur":5,"max_dist":4,"disc_ct":10,"rept_ct":2}
presence_nodes/Sovrummet CONNECTED

EDIT: I think I can see the problem myself. I’m changing the state_topic to "presence_nodes/Sovrummet" instead of "presence_nodes/esp32_1"

EDIT2: Yepp, that was it. Thank you for your help @ptrsnja! :slight_smile:

1 Like
120

Glad it’s working now! How are the distance calculations looking? I’ve found that it gets less accurate the further I get, and seems to grow exponentially. I’m currently working on a fix for that, but no idea how long until I’m ready to release a new version.

121

I experience the same as you. I had to increase “measured power” in the BeaconScope app to -72 instead of -59 to get some connection.

I will try different settings and update the thread if I get some better results.

122

Hi, I have a problem. when I run the build I see the following in the status window

Compiling .pio\build\esp32\lib480\BLE\BLEAdvertising.cpp.o
Compiling .pio\build\esp32\lib480\BLE\BLEBeacon.cpp.o
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino: In function 'void WiFiEvent(system_event_id_t)':
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_SCAN_DONE' not handled in switch [-Wswitch]
   switch(event) {
         ^
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_STA_CONNECTED' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_STA_AUTHMODE_CHANGE' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_STA_LOST_IP' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_STA_WPS_ER_SUCCESS' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_STA_WPS_ER_FAILED' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_STA_WPS_ER_TIMEOUT' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_STA_WPS_ER_PIN' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_STA_WPS_ER_PBC_OVERLAP' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_AP_START' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_AP_STOP' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_AP_STACONNECTED' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_AP_STADISCONNECTED' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_AP_STAIPASSIGNED' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_AP_PROBEREQRECVED' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_GOT_IP6' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_ETH_START' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_ETH_STOP' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_ETH_CONNECTED' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_ETH_DISCONNECTED' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_ETH_GOT_IP' not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:207:9: warning: enumeration value 'SYSTEM_EVENT_MAX' 
not handled in switch [-Wswitch]
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino: In lambda function:
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:493:50: warning: unknown conversion type character 0xa in format [-Wformat=]
       Serial.printf("Progress: %u% \n\r", percent);
Compiling .pio\build\esp32\lib480\BLE\BLECharacteristic.cpp.o
                                                  ^
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino: In function 'bool handleMqttDisconnect()':
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:173:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino: In function 'bool reportDevice(BLEAdvertisedDevice)':
Compiling .pio\build\esp32\lib480\BLE\BLECharacteristicMap.cpp.o
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:409:18: warning: 'distance' may be used uninitialized in this function [-Wmaybe-uninitialized]
     Serial.printf("%s exceeded distance threshold %.2f\n\r", mac_address.c_str(), distance);
                  ^
C:/Users/shvedko/Desktop/ESP32-mqtt-room-master/ESP32-mqtt-room.ino:283:8: note: 'distance' was declared here
  float distance;
        ^
Compiling .pio\build\esp32\lib480\BLE\BLEClient.cpp.o

Compiling .pio\build\esp32\lib480\BLE\BLESecurity.cpp.o
C:\Users\shvedko\.platformio\packages\framework-arduinoespressif32\libraries\BLE\src\BLERemoteService.cpp: In member function 'void BLERemoteService::getCharacteristics(std::map<short unsigned int, BLERemoteCharacteristic*>*)':
C:\Users\shvedko\.platformio\packages\framework-arduinoespressif32\libraries\BLE\src\BLERemoteService.cpp:246:89: warning: parameter 'pCharacteristicMap' set but not used [-Wunused-but-set-parameter]
 void BLERemoteService::getCharacteristics(std::map<uint16_t, BLERemoteCharacteristic*>* pCharacteristicMap) {

The settings file is full. tell me how to solve my problem

123

Those appear to be compilation warnings, but I do not see an error entry. Does it fail to compile or upload?

124

Eventually compiles successfully. I thought that these were errors)))) thanks for the clarification

125

I have another question.
is it possible to configure esp 32 so that it sends an MQTT message only about my BLE devices (register MAC addresses directly in the code)?

126

Not currently, but that’s an interesting idea. Why would you want to restrict the data being collected?

If you’re concerned about performance, the messages shouldn’t be much of an issue for your mqtt server, and unless you configure a sensor for it, Home Assistant doesn’t check for it.

127

Great work here! I’ve skimmed through the posts and can’t see how the initial struggles that were occurring when simultaneously trying to do wi-fi and bluetooth were resolved?

128

It was a lot of trial-and-error, testing, and use of different libraries to manage connections to WiFi and MQTT. Essentially, connection failures to MQTT or WiFi are ignored during an active bluetooth scan, since the antenna is not available. There is a reconnection timer set for either/each of these, in order to re-establish connection when it fails. Finally, I’m allowing WiFi to disconnect when needed, and recovering once the antenna is available.

I haven’t documented any of that here, but if you’re interested, take a look at the source code in the GitHub repo. It’ll allow you to see all the event handlers and async libraries used. This is one of the main reasons why I moved toPlatformIO - I can set specific dependencies and it will auto-install them for the user.

Curious why you ask - are you working on an ESP32 BLE project?

129

Heh no. If I were to, I don’t know why I wouldn’t use yours! :slight_smile:
I’m interested for two reasons:

  • I’m an electrical engineer so I try to keep an eye on what hardware can do what
  • I’ve toyed with the idea of having BLE trackers in the house, and I assume that ESP32’s would be the most economical (but honestly don’t have much of a use case, so it isn’t high on my list)

Has anyone noticed any battery impacts from the android iBeacon apps?

1 Like
130

I’ve had some trouble with keeping the background service running in Android 9 on my OnePlus 5. There are bluetooth issues on the current build of my phone’s firmware, and it will regularly kill the process somehow. I use bluetooth for other things quite often, so it’s always enabled on my phone. If you’re always running bluetooth, I would imagine that it’s not much extra, as it’s a simple low-energy advertisement. However, if you’re a battery miser and you turn everything off when you’re not using it, you may find an impact.

I picked up a load of Trackr Pixel BLE beacons on clearance for 4 bucks a piece, and they work reliably for tracking internally (although the app is terrible). I now know where my dog is when I’m not home, although I haven’t come up with anything interesting to do with that data. Maybe ensure lights are on for him after dark.

132

Hi, I’m having trouble Building this. I don’t think it;s ESP_MQTT_room itself, but I’ve tried on MAC OS and Ubunto and get the same errors.

PACKAGES: toolchain-xtensa32 2.50200.80 (5.2.0), framework-arduinoespressif32 2.10004.191002 (1.0.4), tool-esptoolpy 1.20600.0 (2.6.0)
Converting ESP32-mqtt-room.ino
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Looking for ArduinoJSON library in registry
Found: https://platformio.org/lib/show/64/ArduinoJson
LibraryManager: Installing id=64 @ ^6

ArduinoJson @ 6.14.1 is already installed
Found 31 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <ESP32 BLE Arduino> 1.0.1
|-- <AsyncMqttClient> 0.8.2
|   |-- <AsyncTCP> 1.1.1
|-- <AsyncTCP> 1.1.1
|-- <ArduinoOTA> 1.0
|   |-- <Update> 1.0
|   |-- <WiFi> 1.0
|   |-- <ESPmDNS> 1.0
|   |   |-- <WiFi> 1.0
|-- <WiFi> 1.0
Building in release mode
Compiling .pio/build/esp32/src/ESP32-mqtt-room.ino.cpp.o
Generating partitions .pio/build/esp32/partitions.bin
xtensa-esp32-elf-g++: error: ESP32-mqtt-room.ino.cpp: No such file or directory
xtensa-esp32-elf-g++: fatal error: no input files
compilation terminated.
Compiling .pio/build/esp32/lib17b/BLE/BLE2902.cpp.o
Compiling .pio/build/esp32/lib17b/BLE/BLE2904.cpp.o
Compiling .pio/build/esp32/lib17b/BLE/BLEAddress.cpp.o
Compiling .pio/build/esp32/lib17b/BLE/BLEAdvertisedDevice.cpp.o
Compiling .pio/build/esp32/lib17b/BLE/BLEAdvertising.cpp.o
*** [.pio/build/esp32/src/ESP32-mqtt-room.ino.cpp.o] Error 1
Compiling .pio/build/esp32/lib17b/BLE/BLEBeacon.cpp.o
========================== [FAILED] Took 2.78 seconds ==========================

Has anyone seen this before, and have a sugested solution. I’m banging my head against the wall trying to sort this out.

Thanks, T