Passive BLE Monitor integration

beaj · February 2, 2022, 10:30am

Interestingly, I don’t think this is an iBeacon, Looking at the sensors the dB number looks like it is very close to the pi3. There is other kit near it (switches, nas, x64 servers) but I have no iBeacons.

It also only seems to be the uuid sensor that is noisy.

This is the config to create the table. I apologise in advance as it was thrown together to get some stats. It needs optimising which is why I set a scan interval of 10 minutes.

~B

#
# This is pants but only way currently to get top n sql query results
#
  - platform: sql
    db_url: !secret db_url
    scan_interval: 600
    queries:
      - name: HA Sensor2 Database Size
        query: "SELECT ROUND(page_count * page_size,1)/1000000 as size FROM pragma_page_count(), pragma_page_size();"
        column: "size"
        unit_of_measurement: MB
        
      - name: HA Sensor2 Database events
        query: 'SELECT count(*) as number_events FROM events;'
        column: number_events
        
      - name: HA Sensor2 Database states
        query: 'SELECT count(*) as number_states FROM states;'
        column: number_states

      - name: HA Sensor2 entity 01
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 1;"
        column: "entity_id"

      - name: HA Sensor2 entity 02
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 2;"
        column: "entity_id"

      - name: HA Sensor2 entity 03
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 3;"
        column: "entity_id"

      - name: HA Sensor2 entity 04
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 4;"
        column: "entity_id"

      - name: HA Sensor2 entity 05
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 5;"
        column: "entity_id"

      - name: HA Sensor2 entity 06
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 6;"
        column: "entity_id"

      - name: HA Sensor2 entity 07
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 7;"
        column: "entity_id"

      - name: HA Sensor2 entity 08
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 8;"
        column: "entity_id"

      - name: HA Sensor2 entity 09
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 9;"
        column: "entity_id"

      - name: HA Sensor2 entity 10
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 10;"
        column: "entity_id"
        
      - name: HA Sensor2 entity 11
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 11;"
        column: "entity_id"

      - name: HA Sensor2 entity 12
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 12;"
        column: "entity_id"

      - name: HA Sensor2 entity 13
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 13;"
        column: "entity_id"

      - name: HA Sensor2 entity 14
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 14;"
        column: "entity_id"

      - name: HA Sensor2 entity 15
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 15;"
        column: "entity_id"

      - name: HA Sensor2 entity 16
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 16;"
        column: "entity_id"

      - name: HA Sensor2 entity 17
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 17;"
        column: "entity_id"

      - name: HA Sensor2 entity 18
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 18;"
        column: "entity_id"

      - name: HA Sensor2 entity 19
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 19;"
        column: "entity_id"

      - name: HA Sensor2 entity 20
        query: "SELECT entity_id, COUNT(*) as count FROM states GROUP BY entity_id ORDER BY count DESC LIMIT 20;"
        column: "entity_id"

Markdown Card to display

type: markdown
title: Top 20 DB Entities
content: >+

  | Statistics | |

  |:---|---:|

  | Database size:  | {{ '%3.3f' | format(
  states('sensor.ha_sensor2_database_size') | float ) }} Mb |

  | States table count: | {{ '%6.0f' | format(
  states('sensor.ha_sensor2_database_states') | float ) }} |

  | Events table count:  | {{ '%6.0f' | format(
  states('sensor.ha_sensor2_database_events') | float ) }} |

  | | |

  | | |

  | | |


  | Entity | States |  {{'&nbsp;' * 10}}% |

  |:----|----:|---:|

  | {{states('sensor.ha_sensor2_entity_01') }} |
  {{state_attr('sensor.ha_sensor2_entity_01','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_01','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_02') }} |
  {{state_attr('sensor.ha_sensor2_entity_02','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_02','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_03') }} |
  {{state_attr('sensor.ha_sensor2_entity_03','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_03','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_04') }} |
  {{state_attr('sensor.ha_sensor2_entity_04','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_04','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_05') }} |
  {{state_attr('sensor.ha_sensor2_entity_05','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_05','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_06') }} |
  {{state_attr('sensor.ha_sensor2_entity_06','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_06','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_07') }} |
  {{state_attr('sensor.ha_sensor2_entity_07','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_07','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_08') }} |
  {{state_attr('sensor.ha_sensor2_entity_08','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_08','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_09') }} |
  {{state_attr('sensor.ha_sensor2_entity_09','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_09','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_10') }} |
  {{state_attr('sensor.ha_sensor2_entity_10','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_10','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_11') }} |
  {{state_attr('sensor.ha_sensor2_entity_11','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_11','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_12') }} |
  {{state_attr('sensor.ha_sensor2_entity_12','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_12','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_13') }} |
  {{state_attr('sensor.ha_sensor2_entity_13','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_13','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_14') }} |
  {{state_attr('sensor.ha_sensor2_entity_14','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_14','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_15') }} |
  {{state_attr('sensor.ha_sensor2_entity_15','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_15','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_16') }} |
  {{state_attr('sensor.ha_sensor2_entity_16','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_16','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_17') }} |
  {{state_attr('sensor.ha_sensor2_entity_17','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_17','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_18') }} |
  {{state_attr('sensor.ha_sensor2_entity_18','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_18','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_19') }} |
  {{state_attr('sensor.ha_sensor2_entity_19','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_19','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

  |  {{states('sensor.ha_sensor2_entity_20') }} |
  {{state_attr('sensor.ha_sensor2_entity_20','count') }} | {{  '%2.2f' | format(
  state_attr('sensor.ha_sensor2_entity_20','count') /
  (states('sensor.ha_sensor2_database_states') ) | float * 100 ) }} |

Ernst · February 2, 2022, 3:00pm

I’ve created a new issue on github, where I will further investigate this, see

github.com/custom-components/ble_monitor

Load from iBeacon sensors on database

opened 02:53PM - 02 Feb 22 UTC

Ernst79

As reported on the forum, the iBeacon is still giving a high load on the databas…e. See https://community.home-assistant.io/t/passive-ble-monitor-integration/303583/361?u=ernst In this issue, I try to reproduce this behavior and make some notes. Step 1. Deleted the Database to start with a new fresh empty DB Step 2. Restarted HA Step 3. After 25 minutes I have the following number of number of state updates. Seems to be the `uuid` sensor only. ![image](https://user-images.githubusercontent.com/24375847/152177905-68d6f66e-ea0e-4aec-b52a-502bce3e47f3.png) Step 4. Check if it is continuously increasing, or only occurring in the first period (= 1 minute).....In progress

So far, I can confirm that the uuid sensor is indeed updating quite fast, which it shouldn’t. Bug confirmed.

beaj · February 2, 2022, 4:34pm

Thanks Ernst

Let me know if you want me to test anything.

~B

Ernst · February 2, 2022, 4:58pm

@dmamontov already found a fix. I’ll do a final test tonight, and will release it later tonight. The PR can be found here

fokcuk · February 3, 2022, 1:15am

Hi.
Is it possible to add Qingping Air Monitor lite? It has both WIFI and BLE. I can see that it is chaning BLE UUID on rotation, so I would assume that it sends similar info as it does via WIFI. Happy to help out with what I can (with my limited knowledge on the topic) =)
I have capured pcap and hcidump data

Ernst · February 3, 2022, 10:05pm

Just create an issue on the github repository with the data, and I will have a look

cpc1027 · February 9, 2022, 2:48pm

I have several Govee H101 Thermo-Hygrometers devices added, and I have one that is reporting the incorrect temperature. After trying to troubleshoot for a while, I finally realized that I had to calibrate the temperature in the Govee app to correct it for that particular sensor. It displays correctly on the sensor screen and in the Govee app, but I think the un-calibrated temp is being reported in HA. Is is possible to pull the calibrated temp from the sensor? Or maybe to add an offset via the device configuration menu in the integration?

dersgniw · February 10, 2022, 1:17am

Should I expect the device tracker part of this to track an Android phone? It doesn’t seem to work, but I’m not sure that it’s supposed to.

WhimsySpoon · February 10, 2022, 6:39am

I use the iBeacon feature and have the companion app broadcast on BLE.

baz123 · February 10, 2022, 11:08am

Can anyone confirm which model this sensor is and if the firmware needs flashing? I have some of the older version that ‘just worked’.

Ernst · February 10, 2022, 11:59am

Its a qingping CGDK2. Its supported, but you will need an encryption key

Ernst · February 11, 2022, 7:04pm

Its a bit difficult to add an offset to ble monitor, as a device can have multiple sensor (e.g. temperature and humidity). This would mean we have to make such an option for every possible sensor measurement. You can use a template sensor instead, which should do the trick as well.

unmesh59 · February 12, 2022, 1:53am

I’m accustomed to using Zigbee sensors but wanted to try out BLE sensors and am exploring various passive collector options. I have a few old BT USB dongles in a drawer and am looking for recommendations for a modern one to use on my Intel server running hassio so I don’t go blaming the hardware when things don’t work!

Any suggestions for specific devices or even just underlying chips that I should consider?

P.S.: I already have Tasmota-on-ESP32 up and running as a collector

Ernst · February 12, 2022, 8:02am

Bluetooth 4 is generally better supported than Bluetooth 5 on Linux systems (less driver issues). I personally use a ASUS BT400, which is cheap and works perfect. For better range, you could look for a adapter with antenna.

unmesh59 · February 12, 2022, 7:37pm

Looks like the BT400 uses a Broadcom chip.

I’m running a VM installed from hassio.ova. Is there a straightforward way to figure out what USB Bluetooth drivers it has short of plugging a USB adapter in?

unmesh59 · February 13, 2022, 9:00pm

I ordered a BT400 but thought I might try out an older Broadcom based USB adapter in the interim. It shows up as hci0 in the Integration GUI but no devices have shown up yet even though the Bluetooth sensor is only a few feet away. Maybe I need the newer adapter.

In the meantime, I see that hciconfig and hcidump and btmon don’t seem to be installed in the VM though bluetoothctl is. Do these need to be installed separately? If so, how?

Added: I got the BT400, swapped dongles and the sensors showed up!

unmesh59 · February 15, 2022, 1:08am

Looks like I ordered and received a UB400 and not a BT400 and am therefore using a CSR chip instead of Broadcom. Sorry for the misinformation earlier!

I’m collecting transmissions from a couple of ATCs flashed with pvvx firmware using this integration as well as a couple of ESP32s running Tasmota. Once a day, the Passive BLE monitor reports room temperature at around 160 degrees F for one sample and then goes back to 70-ish. It is a different time for each of the ATCs and I only have two samples so far. Neither of the ESPs report this. At other times, the reports are the same though the collectors are not always capturing the same samples.

Any thoughts on why this might be happening?

bruturb · February 26, 2022, 5:48am

I have a Govee H5178 too. How the heck did you find its MAC address?

Ernst · February 26, 2022, 8:04am

It should be discover automatically, when discovery is enabled in BLE monitor settings. But Govee sensor require active scan to be turned on (also a setting in bLE monitor.)

bruturb · February 26, 2022, 8:00pm

Thanks. I figured out why Passive BLE Monitor couldn’t find it: module 'socket' has no attribute 'AF_BLUETOOTH'. I saw instructions in FAQ but I can’t install PyBluez because: error in PyBluez setup command: use_2to3 is invalid.
It’s annoying and I need a brake.