Passive BLE Monitor integration (Xiaomi Mijia BLE MiBeacon monitor)

Great that they work.

Just some explanation, sensors show up after a first BLE message that is received, but without a value (unknown). After the set period (60 seconds by default) it will average the measurements during that minute, and will start to display the value. So, this takes 1 minute normally.

However, some sensors send data very rarely, like the LYWSD03MMC with stock firmware, which shows only one message in 5 minutes. So, after a restart, it can take up to 5 minutes to show up. That’s why a lot of people like the ATC firmware, it sends several messages per minute, making it way more responsive.

Not sure why debug logging isn’t working for you. To enable debug logging, add the following lines to configuration.yaml:

logger:
  default: warn
  logs:
    custom_components.ble_monitor: debug

Hello all,

Thanks for the work done so far, you did a great job!
The information i’ve got reading the messages within this product thread helped me a lot to go further with my own raspberry - xiaomi/qingping sensors setup.

However there is still an open question with Qingping CGDK2 sensor.
Could you say how it is possible to read the events/notifications/messages from OS Linux level for this sensor?
I was able to got encryption key, and was able to read battery status directly using bluetoothctl, but still struggling with reading temp, hum data from it.
For example, i’ve LYWSD03MMC sensor which events are easily readable out of the box and the data can be obtained with the following python script: https://github.com/JsBergbau/MiTemperature2/blob/7d6ff64fe17e0c53e7f67e3cfa26e740dd9c699c/LYWSD03MMC.py
But how to read the same for Qingping CGDK2?

I have a python script that does the decoding of BLE data of all sensors that we have added to BLE monitor. I use it for adding new devices and it’s a very ugly script with a lot of intermediate debugging messages. You have to manually add the data of one message and the encryption key, so it isn’t doing the BLE connection stuff.

It is basically the same script as what is done in __init__.py at the end of the file. May I ask what you need it for? Why not use BLE monitor? It supports the CGDK2 sensor.

Anyway, if you want the script, I can send it to you. Just send me a private message.

Thanks for feedback , Ernst.

My idea was to read the data from CGDK2 from “clean” OS level, so manually, not using Home Assistant or other products.
Will try to find the relevant part in init.py
Basically trying to get the values to pass it to the other script.

To explain a bit more in detail (for those who are interested :wink: )
BLE monitor does the communication on “OS level” in the file aioblescan_ext.py. This tool is slightly modified, but is normally also available as pypi package (aioblescan). It is basically doing a HCIDUMP of all Bluetooth messages that are received and returns back a long string with numbers for each message.

Next step is the filtering of this data, which is done in __init__.py, starting at line 692. The function parse_raw_message(self, data): will do the magic. It first checks that the correct service data is present for Xiaomi, Qingping or ATC is available (CGDK2 uses the Xiaomi format). Other messages are ignored.

If the message is in the correct format, it will do several checks on length, checks that data is available in the message, checks which device has send the message, but also will do the encrypting for encrypted sensor messages (CGDK2 is encrypted). After encryption, the data is readable (after converting the right numbers from hex to decimals).

Also check this page on our github for an explation of the message structure.

1 Like

Issue with modifying settings in the UI and/or install the component?
There is currently was an issue with BLE monitor, which doesn’t allow you to modify the options of BLE monitor anymore and doesn’t allow you to configure the component after a new installation. This started after the update to HA 2021.3. The component continues to work normally, but you aren’t able to modify options.

Fix available in 1.3.0

2 Likes

Newbie to both Raspberry Pi and Home Assistant.

I am using HA 2021.3.3 and Passive BLE Monitor 1.3.0 on a Pi 4b

I have added 4 CGDK2’s and obtain both the humidity and temperature values.

My problem is that each device individually updates at roughly 11 / 12 minute intervals.

It would appear that the device is not recognised except for the 12mn updates.
The log sample is for a successful update.

2021-03-10 19:44:44 DEBUG (Thread-8) [custom_components.ble_monitor] HCIdump thread: Scanning will be restarted
2021-03-10 19:44:44 DEBUG (Thread-8) [custom_components.ble_monitor] 1060 HCI events processed for previous period.
2021-03-10 19:44:44 DEBUG (Thread-8) [custom_components.ble_monitor] HCIdump thread: Run
2021-03-10 19:44:44 DEBUG (Thread-8) [custom_components.ble_monitor] HCIdump thread: connected to hci0
2021-03-10 19:44:44 DEBUG (Thread-8) [custom_components.ble_monitor] HCIdump thread: start main event_loop
2021-03-10 19:44:46 DEBUG (Thread-8) [custom_components.ble_monitor] Invalid data: Device unkown
2021-03-10 19:44:50 DEBUG (Thread-8) [custom_components.ble_monitor] Invalid data: Device unkown
2021-03-10 19:44:54 DEBUG (Thread-8) [custom_components.ble_monitor] Invalid data: Device unkown
2021-03-10 19:44:58 DEBUG (Thread-8) [custom_components.ble_monitor] Invalid data: Device unkown
2021-03-10 19:45:02 DEBUG (Thread-8) [custom_components.ble_monitor] Invalid data: Device unkown
2021-03-10 19:45:10 DEBUG (Thread-8) [custom_components.ble_monitor] Invalid data: Device unkown
2021-03-10 19:45:14 DEBUG (Thread-8) [custom_components.ble_monitor] Invalid data: Device unkown
2021-03-10 19:45:18 DEBUG (Thread-8) [custom_components.ble_monitor] Invalid data: Device unkown
2021-03-10 19:45:22 DEBUG (Thread-8) [custom_components.ble_monitor] Invalid data: Device unkown
2021-03-10 19:45:26 DEBUG (Thread-8) [custom_components.ble_monitor] Invalid data: Device unkown
2021-03-10 19:45:30 DEBUG (Thread-8) [custom_components.ble_monitor] Invalid data: Device unkown
2021-03-10 19:45:38 DEBUG (Thread-8) [custom_components.ble_monitor] Invalid data: Device unkown
2021-03-10 19:45:42 DEBUG (Thread-8) [custom_components.ble_monitor] Invalid data: Device unkown
2021-03-10 19:45:44 DEBUG (MainThread) [custom_components.ble_monitor.binary_sensor] 1 MiBeacon BLE ADV messages processed for 1 binary sensor device(s) total. Priority queue = 0
2021-03-10 19:45:44 DEBUG (MainThread) [custom_components.ble_monitor.sensor] 4 MiBeacon BLE ADV messages processed for 4 measuring device(s).

Normally after the 60 second sampling the BLE ADV count will be 0.

The update rate for an individual device is constant and correct.

Have you any idea why this is happening ?

My first suspicion would be a bad reception. Did you check the faq with tips. If that doesn’t help, please open an issue on github and I’ll help you with further debugging.

Hello all,
does anyone has problems with bluetooth on a raspberry pi ? My problem is that after enabling the integration with the ble, my system freezes after some hours and the only way to restore it , is to reboot the raspberry. I’ve had the same problem with another intergration that uses the ble , it was room assistant. When i disabled it, my system was up and running for a week or more. Some days before when i enabled hte BLE Monitor integration the problem comes back. Has anyone see the same behavior ?

Edit : CPU normal, Memory normal, Temperature normal, all seems to be normal but the hole system freezes or runs very slowly.

Edit 2 : Almost 10 hours since the last restart, system was laggy again. Waiting for 2 minutes to login and load the user interface. Afterter that going to integrations and disabled the integration. System is back to life again and runs normal. I can’t find something similar with the integration in log file, only errors about connection and data from my devices (TV, Router, cameras etc)

with latest ota 1.1.1_0224 all lag problems are gone and qingping motion sensor triggers almost as fast as lumi.

I am still getting periodic issues with supervisor removing and then failing to add back in the BT dongle. The only solution is to physically remove and reinsert. Otherwise, the integration works impeccably.

I do need to monitor this and see when it has failed - most of the heating decisions in the house are made on the basis of these sensors.

I first created a binary sensor using this template.

{{ states.sensor  
  | selectattr('attributes.unit_of_measurement', '==', '°C')
  | selectattr('attributes.sensor type', '==', 'LYWSDCGQ')
  | selectattr('attributes.last median of', '<', 5)
  | list | count > 0}}

This worked for a while but I hit an incident where the BLE sensors were not being updated so this template still showed a value for last_median_of.

I have now created a Node-Red check for either one of the BLE sensors not being updated or the numbers of readings dropping below 5.

[{"id":"715fca80.becbc4","type":"ha-get-entities","z":"8ec6d5b3.077b58","server":"9bb65ffe.1998d","name":"","rules":[{"property":"attributes.sensor type","logic":"is","value":"LYWSDCGQ","valueType":"str"}],"output_type":"array","output_empty_results":false,"output_location_type":"msg","output_location":"data","output_results_count":1,"x":370,"y":480,"wires":[["61e3beda.49f06"]]},{"id":"61e3beda.49f06","type":"function","z":"8ec6d5b3.077b58","name":"Check for stale","func":"msg.payload = msg.data.filter(function(data){\n    return (data.attributes.device_class == \"temperature\") && ((data.attributes.last_median_of < 5) || (Date.parse(data.last_updated) + 300000) < Date.now());\n}).length > 0;\n\nreturn msg;\n","outputs":1,"noerr":0,"initialize":"","finalize":"","x":588,"y":480,"wires":[["63d3fe43.e747e"]]},{"id":"9bb65ffe.1998d","type":"server","name":"HassIO","legacy":false,"addon":true,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open","connectionDelay":true,"cacheJson":true}]

The function code is quite elegant;

msg.payload = msg.data.filter(function(data){
    return (data.attributes.device_class == "temperature") && ((data.attributes.last_median_of < 5) || (Date.parse(data.last_updated) + 300000) < Date.now());
}).length > 0;

I then send a message to telegram to say there is a problem.

HTH someone else :slight_smile:

1 Like

Yes, I have exactly the same issue. Not sure where to begin to help identify it. I have been meaning to dig into the logs.

Yes, loads. Are you on Supervised or HA OS.

I’m on Supervised.

If you do an hciconfig -a does it say UP, DOWN or do you get nothing?

One of the OS updates seemed to uninstall some packages - reinstalling them seems to fix it.

Second problem is my error above where the Bluetooth adapter just seems to be removed by Supervisor. Only solution is to remove and reinsert the Bluetooth device.

Finally there is a kernel bug in RaspberryOS in that the internal and external BT devices do not play nicely together.

I decided to disable BT entirely on my Pi4. There are some hickups, that I don’t have the time to solve or even check further at the moment. If I disable this integration entirely, my BT runs for at least a week without any hazzle, if I enable it, BT is going down within an hour. I’m not saying this has something to do with this intergation, it is just that BT seems to be bugged on Pis.

If I do as @baz123 suggested, this is what I’m getting… 170 (!!!) new installed packages??? Nope, that would go against “never touch a running system”.

sudo apt install bluetooth pi-bluetooth bluez blueman bluez-hcidump --dry-run
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
bluetooth ist schon die neueste Version (5.50-1.2~deb10u1+rpt2).
bluez ist schon die neueste Version (5.50-1.2~deb10u1+rpt2).
bluez-hcidump ist schon die neueste Version (5.50-1.2~deb10u1+rpt2).
Die folgenden zusätzlichen Pakete werden installiert:
  adwaita-icon-theme at-spi2-core bluez-obexd dbus-user-session
  dconf-gsettings-backend dconf-service fontconfig fontconfig-config
  fonts-dejavu-core gir1.2-appindicator3-0.1 gir1.2-atk-1.0 gir1.2-freedesktop
  gir1.2-gdkpixbuf-2.0 gir1.2-gtk-3.0 gir1.2-notify-0.7 gir1.2-pango-1.0
  glib-networking glib-networking-common glib-networking-services
  gnome-icon-theme gsettings-desktop-schemas gtk-update-icon-cache
  hicolor-icon-theme libaom0 libappindicator3-1 libasound2-plugins libasyncns0
  libatk-bridge2.0-0 libatk1.0-0 libatk1.0-data libatspi2.0-0 libavahi-client3
  libavcodec58 libavresample4 libavutil56 libcairo-gobject2 libcairo2
  libcodec2-0.8.1 libcolord2 libcroco3 libcups2 libdatrie1 libdbusmenu-glib4
  libdbusmenu-gtk3-4 libdconf1 libdrm-amdgpu1 libdrm-common libdrm-nouveau2
  libdrm-radeon1 libdrm2 libepoxy0 libflac8 libfontconfig1 libgdk-pixbuf2.0-0
  libgdk-pixbuf2.0-bin libgdk-pixbuf2.0-common libgl1 libgl1-mesa-dri
  libglapi-mesa libglvnd0 libglx-mesa0 libglx0 libgraphite2-3 libgsm1
  libgtk-3-0 libgtk-3-bin libgtk-3-common libharfbuzz0b libical3 libice6
  libindicator3-7 libjack-jackd2-0 libjbig0 libjson-glib-1.0-0
  libjson-glib-1.0-common liblcms2-2 libllvm9 libltdl7 libmp3lame0 libnotify4
  libogg0 libopenjp2-7 libopus0 liborc-0.4-0 libpango-1.0-0
  libpangocairo-1.0-0 libpangoft2-1.0-0 libpangoxft-1.0-0 libpixman-1-0
  libproxy1v5 libpulse-mainloop-glib0 libpulse0 libpulsedsp librest-0.7-0
  librsvg2-2 librsvg2-common libsbc1 libsensors-config libsensors5 libshine3
  libsm6 libsnappy1v5 libsndfile1 libsoup-gnome2.4-1 libsoup2.4-1 libsoxr0
  libspeex1 libspeexdsp1 libswresample3 libtdb1 libthai-data libthai0
  libtheora0 libtiff5 libtwolame0 libva-drm2 libva-x11-2 libva2
  libvdpau-va-gl1 libvdpau1 libvorbis0a libvorbisenc2 libvpx5 libwavpack1
  libwayland-client0 libwayland-cursor0 libwayland-egl1 libwebp6 libwebpmux3
  libwebrtc-audio-processing1 libx11-xcb1 libx264-155 libx265-165
  libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0
  libxcb-shm0 libxcb-sync1 libxcb-xfixes0 libxcomposite1 libxcursor1
  libxdamage1 libxfixes3 libxft2 libxi6 libxinerama1 libxkbcommon0 libxrandr2
  libxrender1 libxshmfence1 libxtst6 libxvidcore4 libxxf86vm1 libzvbi-common
  libzvbi0 mesa-va-drivers mesa-vdpau-drivers notification-daemon pulseaudio
  pulseaudio-module-bluetooth pulseaudio-utils python3-cairo python3-gi-cairo
  rtkit va-driver-all vdpau-driver-all x11-common
Vorgeschlagene Pakete:
  colord cups-common gvfs jackd2 liblcms2-utils opus-tools librsvg2-bin
  lm-sensors speex pavumeter pavucontrol paman paprefs
Die folgenden NEUEN Pakete werden installiert:
  adwaita-icon-theme at-spi2-core blueman bluez-obexd dbus-user-session
  dconf-gsettings-backend dconf-service fontconfig fontconfig-config
  fonts-dejavu-core gir1.2-appindicator3-0.1 gir1.2-atk-1.0 gir1.2-freedesktop
  gir1.2-gdkpixbuf-2.0 gir1.2-gtk-3.0 gir1.2-notify-0.7 gir1.2-pango-1.0
  glib-networking glib-networking-common glib-networking-services
  gnome-icon-theme gsettings-desktop-schemas gtk-update-icon-cache
  hicolor-icon-theme libaom0 libappindicator3-1 libasound2-plugins libasyncns0
  libatk-bridge2.0-0 libatk1.0-0 libatk1.0-data libatspi2.0-0 libavahi-client3
  libavcodec58 libavresample4 libavutil56 libcairo-gobject2 libcairo2
  libcodec2-0.8.1 libcolord2 libcroco3 libcups2 libdatrie1 libdbusmenu-glib4
  libdbusmenu-gtk3-4 libdconf1 libdrm-amdgpu1 libdrm-common libdrm-nouveau2
  libdrm-radeon1 libdrm2 libepoxy0 libflac8 libfontconfig1 libgdk-pixbuf2.0-0
  libgdk-pixbuf2.0-bin libgdk-pixbuf2.0-common libgl1 libgl1-mesa-dri
  libglapi-mesa libglvnd0 libglx-mesa0 libglx0 libgraphite2-3 libgsm1
  libgtk-3-0 libgtk-3-bin libgtk-3-common libharfbuzz0b libical3 libice6
  libindicator3-7 libjack-jackd2-0 libjbig0 libjson-glib-1.0-0
  libjson-glib-1.0-common liblcms2-2 libllvm9 libltdl7 libmp3lame0 libnotify4
  libogg0 libopenjp2-7 libopus0 liborc-0.4-0 libpango-1.0-0
  libpangocairo-1.0-0 libpangoft2-1.0-0 libpangoxft-1.0-0 libpixman-1-0
  libproxy1v5 libpulse-mainloop-glib0 libpulse0 libpulsedsp librest-0.7-0
  librsvg2-2 librsvg2-common libsbc1 libsensors-config libsensors5 libshine3
  libsm6 libsnappy1v5 libsndfile1 libsoup-gnome2.4-1 libsoup2.4-1 libsoxr0
  libspeex1 libspeexdsp1 libswresample3 libtdb1 libthai-data libthai0
  libtheora0 libtiff5 libtwolame0 libva-drm2 libva-x11-2 libva2
  libvdpau-va-gl1 libvdpau1 libvorbis0a libvorbisenc2 libvpx5 libwavpack1
  libwayland-client0 libwayland-cursor0 libwayland-egl1 libwebp6 libwebpmux3
  libwebrtc-audio-processing1 libx11-xcb1 libx264-155 libx265-165
  libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0
  libxcb-shm0 libxcb-sync1 libxcb-xfixes0 libxcomposite1 libxcursor1
  libxdamage1 libxfixes3 libxft2 libxi6 libxinerama1 libxkbcommon0 libxrandr2
  libxrender1 libxshmfence1 libxtst6 libxvidcore4 libxxf86vm1 libzvbi-common
  libzvbi0 mesa-va-drivers mesa-vdpau-drivers notification-daemon pulseaudio
  pulseaudio-module-bluetooth pulseaudio-utils python3-cairo python3-gi-cairo
  rtkit va-driver-all vdpau-driver-all x11-common
Die folgenden zurßckgehaltenen Pakete werden verändert:
  pi-bluetooth
Die folgenden Pakete werden aktualisiert (Upgrade):
  pi-bluetooth
1 aktualisiert, 170 neu installiert, 0 zu entfernen und 0 nicht aktualisiert.

libgtk is clearly a desktop package, I can’t see why I’d need something like that on a headless Pi, just to achieve BT functionality…

For me that’s a no-go, because I only have three BT devices that I check with this integration. It’s simply not worth the time for me. I’m changing these three devices to Zigbee based ones and sell the BTs to a neighbour or someboby else. :slight_smile:

How do I find the hci device numbers on HA OS?

The ‘hardware’ output from Supervisor does not list the hci devices as such. I should have 2x BT dongles attached but only one seems to work.

To run hcitool dev on Home Assistant OS, to show which device is on which hci, you will need to follow these instructions first.

This will allow you to get access to the actual system.

1 Like

I need some help in adding RTCGQ02LM Mi Motion Sensor to my HA setup.

The device shows up as binary_sensor.ble_light_54ef44e2**** but is showing as state unknown. I assume because I do not have an encryption key. I have BLE_monitor the latter I assume is recognising the device but not the data. I do not have a Xiaomi hub but have ZigbeeMqtt setup and working for Ikea and Xiaomi switches.

Any assistance in setup of this device would be much appreciated.

RTCGQ02LM indeed needs an encryption key. The sensor is recognized and will show up as unknown, till you enter the key. Recognition is needed to show the sensor in the drop down menu at the bottom of the option menu of the BLE monitor integration, where you can add your encryption key.

To get the encryption key, you will need to add the sensor to the MiHome app first. I’m not sure this is possible without an Xiaomi Hub to be honest, you should try. Adding it to MiHome is needed to make sure the key is exchanged. There are a number of ways to get the key, this is described in the FAQ. But the easiest way is to get the key from the Xiaomi cloud with Xiaomi Cloud Token Extractor.

Thanks Ernst for your reply. I was trying to get away with not having to go down the path of an Xiaomi Hub just to get the encryption key but it is now looking as I have little/no choice unless someone has an alternative. Thanks again for your input.

Some one know how to extract the encryption key of GCPR1? i cant include it in Mi Home app too

Thans in advice