I’m not sure what’s going wrong there. Try removing the .piolibdeps folder again and building the project once more.
I can’t reproduce the problem here.
Just wanted to say thanks for this. I wanted to expand my room presence, originally done with happy bubbles (RIP), and this is easy and reliable (not to mention cheap, ~$10 per room). Haven’t given OTA a shot yet, but it should make messing with stuff far easier. The only issue I ran into was #include <ArduinoJSON.h> needed changed to #include <ArduinoJson.h>. Thanks again!
1 Like
So this is a pretty great project. I have an old rpi 3 that used to run HA before i moved away from it, and a RpiZeroW, both running room assistant, and i wanted to use this to supplement as a i have a few esp32s laying around (who does that?)
Ive got it working as expected, except that it justeither not detect some of the devices that room assistant does, or report significantly different distance values.
Any hints ? Or do they just not play nicely together (ie do i need all rpis, or all esp32s?)
Also – can anyone give me some insight into how mqtt_room actually works ?
Is it all based on first detection and then timeout ? Or does smallest distance win ? I seem to be getting things move all over the place when detected my multiple sensors, which happens a bit.
@sh00t2kill I think the main difference between the ESP32s and the rPi boards would be the antenna, which would explain the different distance calculations. I was previously using a combination, but I found that due to the inconsistency of the boards, it was causing erroneous reporting. I’ve moved my rPi 3 to be a media server, and my rPi Zero Ws are running volumio in my kids’ rooms.
If you’re comfortable with writing code, you could potentially implement a distance offset (either a constant or a percent increase/decrease). The distance calculation is exactly the same as exists in the room assistant project, so I can only imagine that hardware differences would be causing the variation you’re seeing.
As for detection of devices, try playing with the scan duration parameter. Very short scans can miss devices, but longer scans means a longer delay between detection and reporting. You can also play with the max distance parameter (the units are meters, but it’s a best guess anyway).
My best understanding is that mqtt_room uses a combination of last seen and nearest distance. You can set a couple of parameters in the sensor entry, which can help keep the random room jumps to a minimum. I’m using these values, stuck in a local variable so I only have to modify it in one place. As far as I can tell, timeout is used to determine how long the last measured distance from a given sensor should remain valid, while away_timeout sets how long from the last time the beacon was seen until it is reported as not_home.
- platform: mqtt_room
device_id: "2f234454cf6d4a0fadf2f4911ba9ffa6-1-0"
name: 'Nexus 5 bt_room'
<<: &ROOM_PARAMS
state_topic: 'room_presence'
timeout: 15
away_timeout: 120
- platform: mqtt_room
device_id: "fda50693a4e24fb1afcfc6eb07647825-5-4"
name: 'Keys bt_room'
<<: *ROOM_PARAMS
1 Like
I’ve been playing with the away timeout, I guess it’s just about finding the sweet spot.
I was playing around with that value last night and it’s definitely better than it was.
It’s entirely possible that my dogs just move around A LOT during the night - the main driver for this project is to track their movements when I’m not at home, as well as normal presence tracking for automation etc
Thanks!
I’m doing the same with my dog, and I’ve set up a couple automations to notify me if I’ve forgotten to leave the radio on for him. I can respond to the notification and it will automatically start it up. I’ve also got a voice integration through the dialog flow component so I can tell it what station I want to hear, and where. I’ll eventually put together a video series going through each part of the system, as it’s somewhat cross-platform integration heavy.
I had the timeout at 10 seconds. Ive bumped it up to 15 and it seems to have stopped the random state changes.
RE the ESP32 scan itself, i had better luck with more frequent but longer scans, rather than shorter but less frequent.
I also think there may be a logic issue with the distance limit … ive currently got this functionality disabled so i get all results, and will troubleshoot later – i physically seperated my scanners more too which has helped.
As i said earlier, its all about finding the sweet spot!
Hi, just stumbled across this project today and it appears to be just what I’m looking for. Was able to get it installed on a spare ESP32 I had lying around and it seems to be kinda working.
Telemetry is rock-solid. It reports to my MQTT broker periodically every time.
However, the actual BLE scan seems to be flaky somehow. I un-commented a number of your Serial.print statements and can confirm that it always sees a bunch of bluetooth devices. However, I can’t seem to detect the simulated BLE beacon from my phone (using Beacon Simulator on a Samsung S7). For example, here is some output from your code:
Advertised Device: Name: , Address: 35:34:27:31:8f:48, manufacturer data: 060001092002385eec7be52cce4bd4bb85dd7a29d8ee4bb989062bfa85
Advertised Device: Name: , Address: 7a:13:1e:bb:85:9c, serviceUUID: 0000feaa-0000-1000-8000-00805f9b34fb
Advertised Device: Name: , Address: 01:c8:2c:d4:b8:54, manufacturer data: 060001092002f270905ecd34b83cd02351a9bb1444cd0972d9f7804272
Advertised Device: Name: , Address: 5e:8c:be:71:bb:c8, serviceUUID: 0000feaa-0000-1000-8000-00805f9b34fb
Scan done! Devices found: 5\
However, after running continuously for over 4 hours, I’ve only had three scan cycles where it actually detected the beacon on my phone and reported it over MQTT. This is frustrating because it’s obviously CAPABLE of working, but I just can’t seem to get it to detect BLE reliably (or even close to reliably).
I know this isn’t much to go on, but I thought I’d see if you’ve experienced anything like this. I’m going to test using other phones/Android devices and will report back if any of that testing reveals anything helpful.
This is a really useful project. Thanks for developing it!
tl;dr at bottom
There’s some information you can provide that can help us solve the problem:
- Can you copy/paste your configuration for home assistant ( not to be confused with the config_local.h file from this project) as well as
- the results you get in the mqtt room_presence/# topic? I use
mosquitto_subon another device to get the values. - It can also help to see a screen grab of the telemetry, just to see how many devices are detected vs how many are reported (the difference coming from the distance threshold).
- If you’re feeling charitable, you can also include the results from subscribing to the telemetry topic, which will show us values over time, where the screen grab in #3 is just a point-in-time.
How did you assess the success of the report? By subscribing to the mqtt topic, or by reading the sensor state in the home assistant frontend? Does the device get reported successfully in Home Assistant if you put your phone right up next to the ESP32?
The problem sounds like it could be a misconfiguration of home assistant, meaning the flow of information is interrupted between MQTT and the front end, rather than between the beacon and the mqtt server. However, if you’ve seen the actual home assistant sensor values show up 4 times, this couldn’t be the case, so skip troubleshooting this part of the equation.
If it’s not a problem with the configuration, and the ESP32 is running and reporting what it thinks is accurate information, it might be down to the app you’re using. There’s a lot that can potentially go wrong with the configuration of a broadcast beacon app. The “distance” is actually a best-guess, based on the received signal strength and the value your beacon sends as its expected signal strength at a distance of one meter in open air. If that value is misconfigured, the detector (I really need to come up with a name for a flashed ESP32 device) will miscalculate the distance, and think that the device is further than in reality, and exclude it by setting.
I find it useful to use a second phone with another app (I like NRF Connect) to scan for the beacon. I couldn’t get Beacon Simulator to work reliably. I found the best results for broadcasting as a beacon come from “BeaconScope” which will scan for, as well as transmit ibeacons. It’s also important to turn off battery optimisation for the broadcasting app, and even with that, I need to periodically open the app again to restart the broadcast process.
Do you have any other trackable devices? Beacon hardware, or some fitness trackers are compatible, and a lot of Xiaomi bluetooth low energy devices (like the flower sensors and from what I’ve read, the temperature/humidity sensors as well) can be tracked by entering their MAC address as the device ID in your home assistant config (without the colons).
I’ve also just this evening updated the documentation. You can read what I’ve written here and if you’ve got suggestions for improvements, I’d love to update it. I’m not the most thorough of documentation writers.
There are also some cases where the device will get itself into a state where it thinks it’s disconnected from MQTT, so it fails to publish any new settings, but the server still sees an active connection, and the onMqttDisconnect callback never fires. I’m working on patching it for the time being, as there’s a bigger piece of work to update the OTA functions to the newest version of the espressif32 platform.
It’s not the most stable right now, but it’s pre-beta, written in a language foreign to the author (c++ I mean, not English), and the environment is built on shifting sands. The code is “hot” so hopefully I’ll have something new to release soon.
Stick with it, I hope it’ll be worth it. I need to put together a troubleshooting section of the documentation, and your experience here will help me with that.
tl;dr it’s as stable as I can get it, but it’s not perfect. Expect reliability problems, but not the one you’ve described.
Thanks for the quick reply, And sorry I didn’t explain my issue as well as I should have. At this point, I haven’t done anything with Home Assistant, so the configuration there can’t be the problem.
Right now, I’m just trying to ensure that:
- The ESP32 can communicate with my MQTT broker. To verify this, I’ve set up the ESP32 config file per your instructions. I am able to receive telemetry signals from the ESP32, for example:
{"room":"front_entry","ip":"192.168.0.11","hostname":"esp32_entry_presence","scan_dur":5,"wait_dur":5,"max_dist":100}
This telemetry is communicated regularly to my MQTT broker. I’m using MQTTLens to monitor these communications, by the way. Again, no HA in the picture yet. From this, I conclude that the ESP32 is connecting over Wifi and communicating as expected with the MQTT broker.
- I also need to verify whether the ESP32 is able to receive BLE signals from something. To check this, I’m monitoring the topic “room_presence” on the MQTT broker. I get (almost) nothing on this topic. However, 3 times in the last 5 or 6 hours I’ve gotten a message on this topic. Unfortunately, I rebooted and lost the message, but it seemed like a plausible message from the ESP32. However, besides these times I get nothing. So, then I went to the serial monitor and un-commented some parts of your code to try to see if the ESP32 was seeing any bluetooth devices. The answer is yes, it is. I posted the serial output in my last e-mail. So far, I’ve concluded that the ESP32 is able to see bluetooth signals, although BLE beacons are very sporadic. Almost non-existent. So, it could be a problem with my ESP32 (although I have now tried this on two different test units that have worked on other projects) OR with my Bluetooth transmitted. At your suggestion, I have installed both BeaconScope and Beacon Simulator on two different Android devices. I can confirm that I am able to detect a beacon generated on one device with Beacon Simulator on another device with BeaconScope. At a distance of about 0.5 meter I’m getting an RSSI reading of -90 dBm or so from one Android device to the other in this setup. Transmit power is set at -75 dBm.
So, at this point I think I have a device capable of transmitting a BLE beacon signal. I know that my ESP32 is able to transmit to the MQTT broker, however I am not seeing anything on the “room_presence” topic except for a few sporadic messages a couple of hours ago.
I feel like I’m close, but obviously not there yet. I’ll read the additions you made to the documentation tonight and let you know if I figure out anything else. Obviously, if you have any other ideas, they’re welcome!
I’m signing off for the night, but I’ve just pushed my latest to the develop branch of the repo. One of the changes is to allow you to set the maxDistance parameter to zero in your config_local.h file. If it’s zero, the device will ignore distance and report EVERYTHING it sees. Could be helpful in troubleshooting.
It could also be totally borked, but you can always revert to the master branch and get the stable(ish) code.
OK. Thanks for your help tonight. As you’ll probably get this in the morning, here’s my update before I go to bed…
I’m starting to understand what you mean about the Android “simulators” being a bit flaky. I now have three different Android devices (a Samsung S7, a Fire tablet, and an ASUS tablet) all running both beaconScope and Beacon Simulator, and the results are bizarre. When I set one to transmit, sometimes both of the other two can see the signal, sometimes one, and sometimes neither. The S7 seems to transmit better than the other two, but even then it’s not reliable. All three on next to each other on my desk, so range is not the issue, even though the calculated distance varies from 0m to ~80m. Clearly, the RSSI-based distance calculation isn’t very good unless the calibration value is pretty accurate.
With 3 devices each transmitting several different beacons, I’ve been able to generate ~10 different sets of MQTT messages from the ESP32. This is better than the 3 in 6 hours from one device earlier, so at least I’m making progress. Based on this I’m starting to become more confident that this is not an ESP32 hardware problem or an issue with your code. I think it has to do with the unreliable Android devices I’m using. I tried an old Polar fitness tracker that I had lying around but this couldn’t be detected by any of the Android devices, so I don’t think it will work.
I think I need to invest in an actual physical beacon of some kind. Any suggestions on what you’ve found to be the most reliable? It’s exciting to get this “almost” working. Hopefully a visit to Amazon to get a proper tracker will get things moving on my end.
One final note, here’s an example of the MQTT transmission I was able to receive on the room_presence topic (so, from Android device running BeaconScope to ESP32 to MQTT broker to MQTTLens…I haven’t even attempted to set up HA yet):
{"id":"48c8cb230078","uuid":"48c8cb230078","rssi":-85,"distance":15.13}
It certainly looks like this is coming from one of the beacons, but this is one of only a few transmissions successfully received over ~30 minutes.
I hate to be the bearer of bad news, but that’s not a beacon advertisement - it’s just a plain BLE advertisement. It looks like it’s not detecting the presence of beacon data.
Try using the NRF Connect app. It will tell you the device type, and let you find the beacons more easily.
Here’s a capture of two advertisements reported by my device, the first is a BLE device and the second is an emulated beacon.
Jul 11 08:15:12 room_presence/livingroom {"id":"f079590f9b22","uuid":"f079590f9b22","rssi":-76,"distance":6.45}
Jul 11 08:15:32 room_presence/livingroom {"id":"2ec29f7da28e45eeadf491713bbe7948-1-0","uuid":"2ec29f7da28e45eeadf491713bbe7948","rssi":-87,"major":1,"minor":0,"txPower":-79,"distance":2}
Note the additional information in the beacon packet - txPower, major and minor are reported by the device. Here’s how I’ve configured beacon scope:
1 Like
Thanks for the recommendation on the NRF app. That might work. When I get home tonight I’ll see if I can make any headway. In the meantime, I’ve ordered a “real” beacon which hopefully will eliminate this variable and allow easier troubleshooting.
Version 0.0.6 has been released!
This is a minor update, which addresses some stability issues I’ve faced after an update to PlatformIO’s espressif32 platform, as well as some libraries getting updated.
There’s also a whole lot of documentation now.. I spent some time learning to use GitHub pages, so now you should have a good way of figuring out how it’s all meant to work. I’m astonished at how this project is growing into something useful for the community, and decided it was time to invest in some good documentation as a matter of principle. I’m keen to continue developing this project, and I hope to have a video made by the end of the year (yes, it’s a long timeline, but I’m trying to be realistic).
Keep those questions coming if you have any difficulties with it. It’s the best way to grow the project and improve the documentation.
Cheers,
James
I have been using this one and it works with mqtt pretty well on my pi w https://github.com/jxlarrea/ha-bt-proximity
That looks interesting, although it’s a significant departure from integration with the mqtt_room component in Home Assistant. It would also end up being quite a bit more costly than using ESP32s as sensor nodes (~10 bucks for an ESP32 versus ~30 for a Pi Zero W and SD card). However, it looks interesting, in that it’s a more fully-featured solution than the built-in mqtt room component.
A quick note about the update:
If you want to update from a previous version to v0.0.6, you may have trouble connecting to the ESP32 over the air. If you see the error message in the console that looks like this:
13:53:51 [DEBUG]: Options: {‘timeout’: 10, ‘esp_ip’: ‘192.168.1.209’, ‘host_port’: 27562, ‘image’: ‘.pio\build\esp32\firmware.bin’, ‘host_ip’: ‘0.0.0.0’, ‘auth’: ‘’, ‘esp
_port’: 3232, ‘spiffs’: False, ‘debug’: True, ‘progress’: True}
13:53:51 [INFO]: Starting on 0.0.0.0:27562
13:53:51 [INFO]: Upload size: 1568912
Sending invitation to 192.168.1.209 …
13:55:31 [ERROR]: No response from the ESP
*** [upload] Error 1
then you can add the following to your platformio.ini file:
upload_flags =
--port=3232
Save the file and run the upload command again. Once you’ve successfully uploaded v0.0.6 to the device, you’ll need to remove this modification, as v0.0.6 listens on the default port 8266