BLE custom component


#1

Hi all,
for those who have the problems with BLE tracking, the component included in Home Assistant has been reported to have some issues.
So I tried to fix the component creating a custom one.
Just create a file bluetooth_le_tracker.py in the custom_components/device_tracker folder and copy the following content into it:

"""
Tracking for bluetooth low energy devices.

"""
import logging

import voluptuous as vol
from homeassistant.helpers.event import track_point_in_utc_time
from homeassistant.components.device_tracker import (
    YAML_DEVICES, CONF_TRACK_NEW, CONF_SCAN_INTERVAL, DEFAULT_SCAN_INTERVAL,
    PLATFORM_SCHEMA, load_config, SOURCE_TYPE_BLUETOOTH_LE
)
import homeassistant.util.dt as dt_util
import homeassistant.helpers.config_validation as cv

_LOGGER = logging.getLogger(__name__)

REQUIREMENTS = ['pygatt==3.2.0']

BLE_PREFIX = 'BLE_'
MIN_SEEN_NEW = 5
CONF_SCAN_DURATION = 'scan_duration'
CONF_BLUETOOTH_DEVICE = 'device_id'

PLATFORM_SCHEMA = PLATFORM_SCHEMA.extend({
    vol.Optional(CONF_SCAN_DURATION, default=10): cv.positive_int,
    vol.Optional(CONF_BLUETOOTH_DEVICE, default='hci0'): cv.string
})


def setup_scanner(hass, config, see, discovery_info=None):
    """Set up the Bluetooth LE Scanner."""
    # pylint: disable=import-error
    import pygatt
    new_devices = {}

    def see_device(address, name, new_device=False):
        """Mark a device as seen."""
        if new_device:
            if address in new_devices:
                _LOGGER.debug(
                    "Seen %s %s times", address, new_devices[address])
                new_devices[address] += 1
                if new_devices[address] >= MIN_SEEN_NEW:
                    _LOGGER.debug("Adding %s to tracked devices", address)
                    devs_to_track.append(address)
                else:
                    return
            else:
                _LOGGER.debug("Seen %s for the first time", address)
                new_devices[address] = 1
                return

        see(mac=BLE_PREFIX + address, host_name=name.strip("\x00"),
            source_type=SOURCE_TYPE_BLUETOOTH_LE)

    def discover_ble_devices():
        """Discover Bluetooth LE devices."""
        _LOGGER.debug("Discovering Bluetooth LE devices")
        try:
            adapter = pygatt.GATTToolBackend()
            devs = adapter.scan()
            devices = {}
            for x in devs:
                devices[x['address']] = x['name']
            _LOGGER.debug("Bluetooth LE devices discovered = %s", devices)
        except RuntimeError as error:
            _LOGGER.error("Error during Bluetooth LE scan: %s", error)
            devices = []
        return devices

    yaml_path = hass.config.path(YAML_DEVICES)
    duration = config.get(CONF_SCAN_DURATION)
    ble_dev_id = config.get(CONF_BLUETOOTH_DEVICE)
    devs_to_track = []
    devs_donot_track = []

    # Load all known devices.
    # We just need the devices so set consider_home and home range
    # to 0
    for device in load_config(yaml_path, hass, 0):
        # check if device is a valid bluetooth device
        if device.mac and device.mac[:4].upper() == BLE_PREFIX:
            if device.track:
                _LOGGER.debug("Adding %s to BLE tracker", device.mac)
                devs_to_track.append(device.mac[4:])
            else:
                _LOGGER.debug("Adding %s to BLE do not track", device.mac)
                devs_donot_track.append(device.mac[4:])

    # if track new devices is true discover new devices
    # on every scan.
    track_new = config.get(CONF_TRACK_NEW)

    if not devs_to_track and not track_new:
        _LOGGER.warning("No Bluetooth LE devices to track!")
        return False

    interval = config.get(CONF_SCAN_INTERVAL, DEFAULT_SCAN_INTERVAL)

    def update_ble(now):
        """Lookup Bluetooth LE devices and update status."""
        devs = discover_ble_devices()
        for mac in devs_to_track:
            _LOGGER.debug("Checking %s", mac)
            result = mac in devs
            _LOGGER.debug("Checking %s", result)
            
            if not result:
                # Could not lookup device name
                continue
            if devs[mac] is None:
                devs[mac] = mac
            see_device(mac, devs[mac])

        if track_new:
            for address in devs:
                if address not in devs_to_track and \
                        address not in devs_donot_track:
                    _LOGGER.info("Discovered Bluetooth LE device %s", address)
                    see_device(address, devs[address], new_device=True)

        track_point_in_utc_time(hass, update_ble, dt_util.utcnow() + interval)

    update_ble(dt_util.utcnow())

    return True


Is there a good way to use Bluetooth LE tracking in HA?
Asus Tinker Board Support for hass.io?
#2

Can this be used with bluetooth usb stick as the scanner device on home assistant ?


#3

You should open a pull request and include this in the main repo!

Please beforehand test and make sure it actually doesn’t cause major load on a Raspberry Pi, which is the main issue with the current BLE tracker.


Bluetooth LE dropping with new HA 82 version
#4

Will this work on Ubuntu NUC?


#5

Hi,

I think so, yes.
Just make sure to specify your device in your config file:

device_tracker:
  - platform: bluetooth_le_tracker
  - device_id: YOUR_DEVICE_ID

#6

I tested on my RPi 3 and it works pretty well. No major load.


#7

Hi,
I saw that you closed the PR. Will your work be implemented in some new version? I think it would be great if this device tracker would be finally fixed.


#8

Your module still uses pygatt - but the problem I have been having is it doesn’t seem possible to install pygatt on Ubuntu, in my case with HA running on an Intel Nuc, in Docker. Any tips there?


#9

Hello,

This is my first post here.
I’m running home-assistant within hassio and also wanted to use BLE tracker.
I also couldn’t make use of pygatt with hassio.

I got inspired by luca-angemiMar python code and some other snippet on the internet. I’ve updated the custom tracker without using pygatt. It works well within hassio.

For what it worth, I thought it could be nice to share it with the community.
Please be kind. It’s the first time I write Python and I had not programmed since 10 years.
I’ve just tested for a few hours. Seems to work but it likely contains bugs and is not optmized.
Would be great if someone could improve it.

"""
Tracking for bluetooth low energy devices.

"""
import logging

import voluptuous as vol
from homeassistant.helpers.event import track_point_in_utc_time
from homeassistant.components.device_tracker import (
    YAML_DEVICES, CONF_TRACK_NEW, CONF_SCAN_INTERVAL, DEFAULT_SCAN_INTERVAL,
    PLATFORM_SCHEMA, load_config, SOURCE_TYPE_BLUETOOTH_LE
)
import homeassistant.util.dt as dt_util
import homeassistant.helpers.config_validation as cv
import struct
import bluetooth._bluetooth as bluez
import time

_LOGGER = logging.getLogger(__name__)

REQUIREMENTS = ['PyBluez==0.22']

BLE_PREFIX = 'BLE_'
MIN_SEEN_NEW = 5
CONF_SCAN_DURATION = 'scan_duration'
CONF_BLUETOOTH_DEVICE = 'device_id'

OGF_LE_CTL=0x08
OCF_LE_SET_SCAN_ENABLE=0x000C
LE_META_EVENT = 0x3e
EVT_LE_CONN_COMPLETE=0x01
EVT_LE_ADVERTISING_REPORT=0x02
COMPLETE_LOCAL_NAME=0x09
SHORTENED_LOCAL_NAME=0x08

PLATFORM_SCHEMA = PLATFORM_SCHEMA.extend({
    vol.Optional(CONF_SCAN_DURATION, default=10): cv.positive_int,
    vol.Optional(CONF_BLUETOOTH_DEVICE, default='0'): cv.string
})


def setup_scanner(hass, config, see, discovery_info=None):
    def packed_bdaddr_to_string(bdaddr_packed):
        return ':'.join('%02x'%i for i in struct.unpack("<BBBBBB", bdaddr_packed[::-1]))

    def returnnumberpacket(pkt):
        myInteger = 0
        multiple = 256
        for c in pkt:
            myInteger +=  int(c) * multiple
            multiple = 1
        return myInteger 

    def returnstringpacket(pkt):
        myString = "";
        for c in pkt:
            myString +=  "%02x" % (c,)
        return myString

    """Set up the Bluetooth LE Scanner."""
    new_devices = {}

    def see_device(address, name, new_device=False):
        """Mark a device as seen."""
        if new_device:
            if address in new_devices:
                _LOGGER.debug(
                    "Seen %s %s times", address, new_devices[address])
                new_devices[address] += 1
                if new_devices[address] >= MIN_SEEN_NEW:
                    _LOGGER.debug("Adding %s to tracked devices", address)
                    devs_to_track.append(address)
                else:
                    return
            else:
                _LOGGER.debug("Seen %s for the first time", address)
                new_devices[address] = 1
                return

        see(mac=BLE_PREFIX + address, host_name=name.strip("\x00"),
            source_type=SOURCE_TYPE_BLUETOOTH_LE)

    duration = config.get(CONF_SCAN_DURATION)
    hciId = int(config.get(CONF_BLUETOOTH_DEVICE)[3:])
    sock = bluez.hci_open_dev(hciId)
    _LOGGER.debug('Connected to bluetooth adapter hci%i',hciId)

    cmd_pkt = struct.pack("<BB", 0x01, 0x00)
    bluez.hci_send_cmd(sock, OGF_LE_CTL, OCF_LE_SET_SCAN_ENABLE, cmd_pkt)
    _LOGGER.debug('Activated BLE scan on bluetooth adapter hci%i',hciId)                
            
    def discover_ble_devices():
        """Discover Bluetooth LE devices."""
        _LOGGER.debug("Discovering Bluetooth LE devices")
        try:
            #[{'name': None, 'address': 'A4:77:33:C2:D0:5F'}, {'name': 'BF PPG Project', 'address': 'A0:E6:F8:75:50:84'}]
            devices = {}

            startTime = time.time()
            while time.time() < startTime + duration:

                old_filter = sock.getsockopt( bluez.SOL_HCI, bluez.HCI_FILTER, 14)
                flt = bluez.hci_filter_new()
                bluez.hci_filter_all_events(flt)
                bluez.hci_filter_set_ptype(flt, bluez.HCI_EVENT_PKT)
                sock.setsockopt( bluez.SOL_HCI, bluez.HCI_FILTER, flt )

                pkt = sock.recv(255)
                ptype, event, plen = struct.unpack("BBB", pkt[:3])

                if event == LE_META_EVENT:
                    subevent, = struct.unpack("B", pkt[3:4])
                    pkt = pkt[4:]
                    if subevent == EVT_LE_ADVERTISING_REPORT:
                        num_reports = struct.unpack("B", pkt[0:1])[0]
                        report_pkt_offset = 0
                        report_event_type = struct.unpack("B", pkt[report_pkt_offset + 1: report_pkt_offset + 1 + 1])[0]
                        
                        for i in range(0, num_reports):
                            macAdressSeen = packed_bdaddr_to_string(pkt[3:9])
                            rssi, = struct.unpack("b", pkt[report_pkt_offset-1:])

                            #Look for 0x09 - "Complete Local Name" or 0x08 - "Shortened Local Name"                    
                            #Read AD type and length after mac address
                            report_pkt_offset = report_pkt_offset + 10
                            len, adType = struct.unpack("BB", pkt[report_pkt_offset : report_pkt_offset + 2])
                            name = ""

                            while(adType != COMPLETE_LOCAL_NAME and adType != SHORTENED_LOCAL_NAME and report_pkt_offset + len + 2 < plen - 1):
                                report_pkt_offset = report_pkt_offset + len + 1
                                len, adType = struct.unpack("BB", pkt[report_pkt_offset : report_pkt_offset + 2])

                            
                            if adType == COMPLETE_LOCAL_NAME:
                                name = pkt[report_pkt_offset + 2 : report_pkt_offset + len + 1].decode('UTF-8')
                            elif adType == SHORTENED_LOCAL_NAME:
                                name = pkt[report_pkt_offset + 2 : report_pkt_offset + len + 1].decode('UTF-8')

                            devices[macAdressSeen.upper()] = name
                sock.setsockopt( bluez.SOL_HCI, bluez.HCI_FILTER, old_filter )

            _LOGGER.debug("Bluetooth LE devices discovered = %s", devices)
        except RuntimeError as error:
            _LOGGER.error("Error during Bluetooth LE scan: %s", error)
            devices = []
        return devices

    yaml_path = hass.config.path(YAML_DEVICES)
    devs_to_track = []
    devs_donot_track = []
    
    # Load all known devices.
    # We just need the devices so set consider_home and home range
    # to 0
    for device in load_config(yaml_path, hass, 0):
        # check if device is a valid bluetooth device
        if device.mac and device.mac[:4].upper() == BLE_PREFIX:
            if device.track:
                _LOGGER.debug("Adding %s to BLE tracker", device.mac)
                devs_to_track.append(device.mac[4:])
            else:
                _LOGGER.debug("Adding %s to BLE do not track", device.mac)
                devs_donot_track.append(device.mac[4:])

    # if track new devices is true discover new devices
    # on every scan.
    track_new = config.get(CONF_TRACK_NEW)

    if not devs_to_track and not track_new:
        _LOGGER.warning("No Bluetooth LE devices to track!")
        return False

    interval = config.get(CONF_SCAN_INTERVAL, DEFAULT_SCAN_INTERVAL)

    def update_ble(now):
        """Lookup Bluetooth LE devices and update status."""
        devs = discover_ble_devices()
        for mac in devs_to_track:
            _LOGGER.debug("Checking %s", mac)
            result = mac in devs
            _LOGGER.debug("Checking %s", result)
            
            if not result:
                # Could not lookup device name
                continue
            if devs[mac] is None:
                devs[mac] = mac
            see_device(mac, devs[mac])

        if track_new:
            for address in devs:
                if address not in devs_to_track and \
                        address not in devs_donot_track:
                    _LOGGER.info("Discovered Bluetooth LE device %s", address)
                    see_device(address, devs[address], new_device=True)

        track_point_in_utc_time(hass, update_ble, dt_util.utcnow() + interval)

    update_ble(dt_util.utcnow())

    return True

I use it with this configuration:

device_tracker:
  - platform: bluetooth_le_tracker
    device_id: hci0
    scan_duration: 20
    interval_seconds: 10

Regards,


#10

I really wanted to try it on my hassio, however, I get error:

ERROR (MainThread) [homeassistant.components.device_tracker] Error setting up platform bluetooth_le_tracker_mod

Do I need to do something in addition to adding the modified file in the custom component folder and configuration?


#11

The component is working great, but there’s one flaw if you’re not running as root. If you happen to restart while scanning, you get:

pygatt.exceptions.BLEError: BLE adapter requires reset after a scan as root- call adapter.reset()

Best solution would be to call a reset on shutdown or on loading the component or call

ExecStartPre=/bin/hciconfig hci0 reset

in your home-assistant.service right before ExecStart


#12

Thanks so much for this. I was looking for a way to get my BLE beacons to work for presence detection.


#13

Can anyone advise on the issue I am having with the component? It will run fine for a couple of hours and then eventually die with the following errors. I am running hassos latest version on a RPi B 3. I have also tried running the other version of code that @rodrig.toni added but it will also die for me after a number of hours


2018-09-11 04:38:53 ERROR (MainThread) [homeassistant.core] Error doing job: Future exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pexpect/spawnbase.py", line 166, in read_nonblocking
    s = os.read(self.child_fd, size)
OSError: [Errno 5] I/O error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pexpect/expect.py", line 111, in expect_loop
    incoming = spawn.read_nonblocking(spawn.maxread, timeout)
  File "/usr/local/lib/python3.6/site-packages/pexpect/pty_spawn.py", line 485, in read_nonblocking
    return super(spawn, self).read_nonblocking(size)
  File "/usr/local/lib/python3.6/site-packages/pexpect/spawnbase.py", line 171, in read_nonblocking
    raise EOF('End Of File (EOF). Exception style platform.')
pexpect.exceptions.EOF: End Of File (EOF). Exception style platform.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pygatt/backends/gatttool/gatttool.py", line 315, in scan
    scan.expect('foooooo', timeout=timeout)
  File "/usr/local/lib/python3.6/site-packages/pexpect/spawnbase.py", line 341, in expect
    timeout, searchwindowsize, async_)
  File "/usr/local/lib/python3.6/site-packages/pexpect/spawnbase.py", line 369, in expect_list
    return exp.expect_loop(timeout)
  File "/usr/local/lib/python3.6/site-packages/pexpect/expect.py", line 117, in expect_loop
    return self.eof(e)
  File "/usr/local/lib/python3.6/site-packages/pexpect/expect.py", line 63, in eof
    raise EOF(msg)
pexpect.exceptions.EOF: End Of File (EOF). Exception style platform.
<pexpect.pty_spawn.spawn object at 0x7fab8d87b8>
command: /usr/bin/hcitool
args: ['/usr/bin/hcitool', '-i', 'hci0', 'lescan']
buffer (last 100 chars): b''
before (last 100 chars): b'Set scan parameters failed: I/O error\r\n'
after: <class 'pexpect.exceptions.EOF'>
match: None
match_index: None
exitstatus: 1
flag_eof: True
pid: 3801
child_fd: 44
closed: False
timeout: 30
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_re:
    0: re.compile(b'foooooo')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/device_tracker/bluetooth_le_tracker.py", line 103, in update_ble
    devs = discover_ble_devices()
  File "/config/custom_components/device_tracker/bluetooth_le_tracker.py", line 62, in discover_ble_devices
    devs = adapter.scan()
  File "/usr/local/lib/python3.6/site-packages/pygatt/backends/gatttool/gatttool.py", line 326, in scan
    raise BLEError(message)
pygatt.exceptions.BLEError: Unexpected error when scanning: Set scan parameters failed: I/O error

#14

Something going wrong with the bluetooth stack/driver/subsystem.
Could you get a dmesg of the system when this happens?

I have no experience with hassos, so could be Hassos also.
Pi3b test system used to run on raspbian and had no issues.
Could be interesting to try if you can’t find a solution. (Hassos bug maybe?)


#15

This looks like it could be the issue. I have reverted back to just using bluetooth for the moment with my phone as it is more reliable. It would be realy nice to get my BLE fobs working though

[45856.788473] Bluetooth: hci0: Frame reassembly failed (-84)
[45862.694066] Bluetooth: hci0: Frame reassembly failed (-84)
[45863.516528] Bluetooth: hci0 command 0x0419 tx timeout
[45868.572557] Bluetooth: hci0 command 0x0419 tx timeout
[45871.641685] Bluetooth: hci0: Frame reassembly failed (-84)
[45871.643537] Bluetooth: hci0: Frame reassembly failed (-84)
[45873.564681] Bluetooth: hci0 command 0x0419 tx timeout
[45876.763025] Bluetooth: hci0: Frame reassembly failed (-84)
[45876.764781] Bluetooth: hci0: Frame reassembly failed (-84)
[45876.766403] Bluetooth: hci0: Frame reassembly failed (-84)
[45889.268746] Bluetooth: hci0: Frame reassembly failed (-84)
[45889.270316] Bluetooth: hci0: Frame reassembly failed (-84)
[45889.271754] Bluetooth: hci0: Frame reassembly failed (-84)
[45890.245405] Bluetooth: hci0: Frame reassembly failed (-84)
[45890.246780] Bluetooth: hci0: Frame reassembly failed (-84)
[45890.248039] Bluetooth: hci0: Frame reassembly failed (-84)
[45891.228652] Bluetooth: hci0 command 0x0401 tx timeout
[45893.276669] Bluetooth: hci0 command 0x0419 tx timeout
[45896.284687] Bluetooth: hci0 command 0x0419 tx timeout
[45899.282138] Bluetooth: hci0: Frame reassembly failed (-84)
[45901.340711] Bluetooth: hci0 command 0x0419 tx timeout

I was thinking about moving to another installation method so I would have more control. I have been kind of putting this off for a while


#16

There has been an update to BLE with V78.0 so hopefully, that resolves the issue. I have updated now and will see if I get any errors.


#17

Does the script recognize ble devices automatically and save them to known_devices? I see in the code that yes, but file is empty… Do I need to activate Bluetooth BCM43xx component?


#18

I believe that the BLE tracker is broken in 0.78, see https://github.com/home-assistant/home-assistant/issues/16698


#20

@majkim did you check if your bluetooth is detecting your device with hcitool/hciconfig ?
you can check if it detects your devices with:

hcitool lescan

if that works, you can try manually adding 1 to see if it reacts.

@molodax this thread is about the custom_component that @luca-angemi made. Your bug is about the official component? But ty for pointing out the official component should be fixed as off 0.78! Will try :stuck_out_tongue: :love_you_gesture:


#21

Not sure that I fully understand you, however, the update to the official component in 0.78.0 was done based on this custom one. Perhaps, it would be good if someone would contribute to the official tracker to fix it.