Some BT Smart Hub 2 + Netgear Nighthawk X6 R8000 + Google Nest Wifi Pro Sensors I have identified and created

beecho01 · July 13, 2024, 10:12am

Hi All,

I am a bit of a tinkerer and wanted some more information out of my network devices, but I found the existing integrations lacking in the detail that I hoped for. To that end, I went on a bit of a dig through the Web UIs and was able to identify how some of the information was being delivered to the Web UI via APIs and XML files.

BT Smart Hub 2

Sensors for gathering the data from my device at address 192.168.1.254:

Rest Sensors:

sensor:
  - platform: rest
    name: "BT Smart Hub 2 Raw Data"
    resource: "http://192.168.1.254/nonAuth/wan_conn.xml"
    headers:
      Referer: "http://192.168.1.254/basic_-_status.htm"
    value_template: "OK"
    scan_interval: 60
    json_attributes:
      - status

  - platform: rest
    name: "BT Smart Hub 2 Firmware Version"
    resource_template: "http://192.168.1.254/cgi/cgi_basicStatus.js?t={{ (now().timestamp() * 1000) | int }}"
    headers:
      Referer: "http://192.168.1.254/basic_-_status.htm"
    value_template: >
      {% set fw_ver = value.split('fw_ver=')[1].split(';')[0].replace('"', '') %}
      {{ fw_ver | regex_replace(find="%2E", replace=".") | regex_replace(find="%2D", replace="-") | regex_replace(find="%20", replace=" ") | regex_replace(find="%28", replace="(") | regex_replace(find="%29", replace=")") | regex_replace(find="%3A", replace=":") }}
    scan_interval: 60

  - platform: rest
    name: "BT Smart Hub 2 Firmware Update Time"
    resource_template: "http://192.168.1.254/cgi/cgi_basicStatus.js?t={{ (now().timestamp() * 1000) | int }}"
    headers:
      Referer: "http://192.168.1.254/basic_-_status.htm"
    value_template: >
      {% set fw_update_time = value.split('fw_update_time=')[1].split(';')[0].replace("'", '') %}
      {% set formatted_time = fw_update_time | regex_replace(find="%20", replace=" ") | regex_replace(find="%3A", replace=":") %}
      {% set dt = strptime(formatted_time, '%a %b %d %H:%M:%S %Y') %}
      {{ (dt.timestamp() * 1000) | int }}
    scan_interval: 60

  - platform: rest
    name: "BT Smart Hub 2 Serial Number"
    resource_template: "http://192.168.1.254/cgi/cgi_basicStatus.js?t={{ (now().timestamp() * 1000) | int }}"
    headers:
      Referer: "http://192.168.1.254/basic_-_status.htm"
    value_template: >
      {% set serial_no = value.split('serial_no=')[1].split(';')[0].replace('"', '') %}
      {{ serial_no | regex_replace(find="%2B", replace="+") | regex_replace(find="%2E", replace=".") }}
    scan_interval: 60

  - platform: rest
    name: "BT Smart Hub 2 Online Devices"
    resource_template: "http://192.168.1.254/cgi/cgi_home.js?t={{ (now().timestamp() * 1000) | int }}"
    headers:
      Referer: "http://192.168.1.254/"
    value_template: >
      {% set raw_response = value | default('') %}
      {% set clients = raw_response | regex_findall("activity:'1'") %}
      {% set clients_count = clients | length - 1 %}
      {{ [clients_count, 0] | max | int }}

Template Sensors:

template:
  - sensor:
      - name: "BT Smart Hub 2 WAN Total Traffic"
        state: >
          {% set wan_conn_volume_list = state_attr('sensor.bt_smart_hub_2_raw_data', 'status').get('wan_conn_volume_list', {}).get('@value', '') %}
          {% set decoded_list = wan_conn_volume_list.replace("%3B", ";").replace("%2E", ".").replace("%3A", ":").replace("%2F", "/").replace("[", "").replace("]", "").replace("'", "") %}
          {% set items = decoded_list.split(',') %}
          {% set wan_total = items[0].split(';')[0] | int + items[1].split(';')[0] | int + items[2].split(';')[0] | int %}
          {{ wan_total | trim }}
        device_class: data_size
        unit_of_measurement: B

      - name: "BT Smart Hub 2 WAN Download Traffic"
        state: >
          {% set wan_conn_volume_list = state_attr('sensor.bt_smart_hub_2_raw_data', 'status').get('wan_conn_volume_list', {}).get('@value', '') %}
          {% set decoded_list = wan_conn_volume_list.replace("%3B", ";").replace("%2E", ".").replace("%3A", ":").replace("%2F", "/").replace("[", "").replace("]", "").replace("'", "") %}
          {% set items = decoded_list.split(',') %}
          {% set wan_download = items[0].split(';')[1] | int + items[1].split(';')[1] | int + items[2].split(';')[1] | int %}
          {{ wan_download | trim }}
        device_class: data_size
        unit_of_measurement: B

      - name: "BT Smart Hub 2 WAN Upload Traffic"
        state: >
          {% set wan_conn_volume_list = state_attr('sensor.bt_smart_hub_2_raw_data', 'status').get('wan_conn_volume_list', {}).get('@value', '') %}
          {% set decoded_list = wan_conn_volume_list.replace("%3B", ";").replace("%2E", ".").replace("%3A", ":").replace("%2F", "/").replace("[", "").replace("]", "").replace("'", "") %}
          {% set items = decoded_list.split(',') %}
          {% set wan_upload = items[0].split(';')[2] | int + items[1].split(';')[2] | int + items[2].split(';')[2] | int %}
          {{ wan_upload | trim }}
        device_class: data_size
        unit_of_measurement: B

      - name: "BT Smart Hub 2 Status Rate"
        state: >
          {% set status_rate = state_attr('sensor.bt_smart_hub_2_raw_data', 'status').get('status_rate', {}).get('@value', '') %}
          {% set decoded_list = status_rate.replace("%3B", ";").replace("%2E", ".").replace("%3A", ":").replace("%2F", "/") %}
          {% set parsed_list = decoded_list | regex_findall('\'(.*?)\'') %}
          {% set first_part = parsed_list[0] if parsed_list else '' %}
          {% set first_value = first_part.split(';')[0] | int(0) if first_part else 0 %}
          {{ (first_value / 1000000) | round(0) | default(0) }}
        device_class: data_rate
        unit_of_measurement: Mbit/s

      - name: "BT Smart Hub 2 IPv6 Link-Local Address"
        state: >
          {% set ip6_lla_list = state_attr('sensor.bt_smart_hub_2_raw_data', 'status').get('ip6_lla_list', {}).get('@value', '') %}
          {% set decoded_list = ip6_lla_list.replace("%3B", ";").replace("%2E", ".").replace("%3A", ":").replace("%2F", "/") %}
          {% set ip6_lla = decoded_list | regex_findall('\'(.*?)\'') %}
          {{ ip6_lla[0] if ip6_lla | length > 0 else none }}

      - name: "BT Smart Hub 2 IPv6 Global Unicast Address"
        state: >
          {% set ip6_gua_list = state_attr('sensor.bt_smart_hub_2_raw_data', 'status').get('ip6_gua_list', {}).get('@value', '') %}
          {% set decoded_list = ip6_gua_list.replace("%3B", ";").replace("%2E", ".").replace("%3A", ":").replace("%2F", "/") %}
          {% set split_list = decoded_list | regex_findall("'(.*?)'") %}
          {% set ip6_gua = split_list[0].split(";")[0] | trim %}
          {{ ip6_gua | trim if ip6_gua | length > 0 else none }}

      - name: "BT Smart Hub 2 IPv4 Public Address"
        state: >
          {% set ip4_info_list = state_attr('sensor.bt_smart_hub_2_raw_data', 'status').get('ip4_info_list', {}).get('@value', '') %}
          {% set decoded_list = ip4_info_list.replace("%3B", ";").replace("%2E", ".").replace("%3A", ":").replace("%2F", "/") %}
          {% set ip4_public = decoded_list.split("', '")[0].replace('[','').replace(']','').replace("'",'').split(';') %}
          {{ ip4_public[0] | trim if ip4_public | length > 0 else none }}

      - name: "BT Smart Hub 2 IPv4 Primary DNS"
        state: >
          {% set ip4_info_list = state_attr('sensor.bt_smart_hub_2_raw_data', 'status').get('ip4_info_list', {}).get('@value', '') %}
          {% set decoded_list = ip4_info_list.replace("%3B", ";").replace("%2E", ".").replace("%3A", ":").replace("%2F", "/") %}
          {% set ip4_dns = decoded_list.split("', '")[0].replace('[','').replace(']','').replace("'",'').split(';') %}
          {{ ip4_dns[3] | trim if ip4_dns | length > 0 else none }}

      - name: "BT Smart Hub 2 IPv4 Secondary DNS"
        state: >
          {% set ip4_info_list = state_attr('sensor.bt_smart_hub_2_raw_data', 'status').get('ip4_info_list', {}).get('@value', '') %}
          {% set decoded_list = ip4_info_list.replace("%3B", ";").replace("%2E", ".").replace("%3A", ":").replace("%2F", "/") %}
          {% set dns_list = decoded_list.split("','")[0].replace('[','').replace(']','').replace("'",'').split(';') %}
          {% set ip4_dns = dns_list[0].split(',') %}
          {{ ip4_dns[0] | trim if ip4_dns | length > 0 else none }}

      - name: "BT Smart Hub 2 IPv4 Subnet Mask"
        state: >
          {% set ip4_info_list = state_attr('sensor.bt_smart_hub_2_raw_data', 'status').get('ip4_info_list', {}).get('@value', '') %}
          {% set decoded_list = ip4_info_list.replace("%3B", ";").replace("%2E", ".").replace("%3A", ":").replace("%2F", "/") %}
          {% set ip4_subnet = decoded_list.split("', '")[0].replace('[','').replace(']','').replace("'",'').split(';') %}
          {{ ip4_subnet[1] | trim if ip4_subnet | length > 0 else none }}

      - name: "BT Smart Hub 2 IPv4 Default Gateway"
        state: >
          {% set ip4_info_list = state_attr('sensor.bt_smart_hub_2_raw_data', 'status').get('ip4_info_list', {}).get('@value', '') %}
          {% set decoded_list = ip4_info_list.replace("%3B", ";").replace("%2E", ".").replace("%3A", ":").replace("%2F", "/") %}
          {% set ip4_gateway = decoded_list.split("', '")[0].replace('[','').replace(']','').replace("'",'').split(';') %}
          {{ ip4_gateway[2] | trim if ip4_gateway | length > 0 else none }}

Maybe this can be included somehow in the BT Smart Hub integration @typhoon2099?

Netgear Nighthawk X6 R8000

I had some issues identifying the Netgear Nighthawk X6 R8000 attached devices. The integration appears to show all devices on my network/VLAN and there doesn’t appear to be a way to use rest sensors or multiscrape. So I was only able to discern devices that were directly attached via Wi-Fi from the existing integration. This does require all devices provided by the Netgear integration to have the “Link Type” sensor enabled.

Template Sensor:

name: "Nighthawk X6 R8000 Wifi Devices"
state: >
  {{ 
    expand(integration_entities('netgear'))
    | selectattr('domain', 'eq', 'sensor')
    | selectattr('entity_id', 'search', '_link_type')
    | map(attribute='state')
    | select('match', 'wireless')
    | list
    | length
  }}
state_class: Total

Google Nest Wifi Pro

Last but not least, I was able to use the still exposed v1 API provided by the Google Nest Wifi Pro. It would seem the more detailed information is hidden in their v2 APIs which now require OAuth. See below the sensor to gather the v1 API.

Sensors for gathering the data from my device at address 192.168.0.45:

Rest Sensor:

sensor:
  - platform: rest
    name: "Google Nest Wifi Pro Raw Data"
    resource: "http://192.168.0.45/api/v1/status"
    value_template: "OK"
    scan_interval: 60
    json_attributes:
      - dns
      - setupState
      - software
      - system
      - vorlonInfo
      - wan

Template Sensors:

This exposed a fair number of returns including the model name, uptime, update progress to name a few.

template:
  sensors:
      - name: "Google Nest Wifi Pro DNS Mode"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'dns').mode | capitalize }}

      - name: "Google Nest Wifi Pro Setup State"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'setupState') }}

      - name: "Google Nest Wifi Pro Software Version"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'software').softwareVersion }}

      - name: "Google Nest Wifi Pro Update Channel"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'software').updateChannel }}

      - name: "Google Nest Wifi Pro Update New Version"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'software').updateNewVersion }}

      - name: "Google Nest Wifi Pro Update Progress"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'software').updateProgress | float }}
        unit_of_measurement: %
        state_class: measurement

      - name: "Google Nest Wifi Pro Update Required"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'software').updateRequired | lower }}

      - name: "Google Nest Wifi Pro Update Status"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'software').updateStatus | capitalize }}

      - name: "Google Nest Wifi Pro Country Code"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'system').countryCode | upper }}

      - name: "Google Nest Wifi Pro Group Role"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'system').groupRole }}

      - name: "Google Nest Wifi Pro Hardware ID"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'system').hardwareId }}

      - name: "Google Nest Wifi Pro LAN0 Link"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'system').lan0Link | lower }}

      - name: "Google Nest Wifi Pro LED Animation"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'system').ledAnimation | capitalize }}

      - name: "Google Nest Wifi Pro LED Brightness"
        state: >
          {% set led_intensity = state_attr('sensor.google_nest_wifi_pro_raw_data', 'system').ledIntensity | int %}
          {{ {0: 'Off', 50: 'Low', 100: 'High'}.get(led_intensity, 'Unknown') }}

      - name: "Google Nest Wifi Pro Model ID"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'system').modelId | capitalize }}

      - name: "Google Nest Wifi Pro OOBE Detailed Status"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'system').oobeDetailedStatus }}

      - name: "Google Nest Wifi Pro Uptime"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'system').uptime | int }}
        unit_of_measurement: s
        device_class: duration
        state_class: measurement

      - name: "Google Nest Wifi Pro Vorlon Info Migration Mode"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'vorlonInfo').migrationMode }}

      - name: "Google Nest Wifi Pro WAN Captive Portal"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'wan').captivePortal | lower }}

      - name: "Google Nest Wifi Pro WAN Ethernet Link"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'wan').ethernetLink | lower }}

      - name: "Google Nest Wifi Pro WAN Gateway IPv4 Address"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'wan').gatewayIpAddress }}

      - name: "Google Nest Wifi Pro WAN Invalid Credentials"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'wan').invalidCredentials | lower }}

      - name: "Google Nest Wifi Pro WAN IPv4 Address"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'wan').ipAddress | lower }}

      - name: "Google Nest Wifi Pro WAN IPv4 Method"
        state: >
          {% set ip_Method = state_attr('sensor.google_nest_wifi_pro_raw_data', 'wan').ipMethod %}
          {{ {'dhcp': 'DHCP', 'static': 'Static'}.get(ip_Method, 'Unknown') }}

      - name: "Google Nest Wifi Pro WAN IPv4 Prefix Length"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'wan').ipPrefixLength | int }}

      - name: "Google Nest Wifi Pro WAN Lease Duration"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'wan').leaseDurationSeconds | int }}
        unit_of_measurement: s
        device_class: duration
        state_class: measurement

      - name: "Google Nest Wifi Pro WAN Local IPv4 Address"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'wan').localIpAddress }}

      - name: "Google Nest Wifi Pro WAN Online"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'wan').online | lower }}

      - name: "Google Nest Wifi Pro WAN PPPoE Detected"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'wan').pppoeDetected | lower }}

      - name: "Google Nest Wifi Pro WAN VLAN Scan Attempt Count"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'wan').vlanScanAttemptCount | int }}
        state_class: measurement

      - name: "Google Nest Wifi Pro WAN VLAN Scan Complete"
        state: >
          {{ state_attr('sensor.google_nest_wifi_pro_raw_data', 'wan').vlanScanComplete | lower }}

I hope this is of use to some!

typhoon2099 · July 13, 2024, 10:52am

Hi @beecho01, thanks for having a look at the BT Smart Hub 2, I’ve been meaning to get back to the integration and update things, I’ll have to add those endpoints and data to the list! There’s a Python package that I’m currently the maintainer for, I’ll add the data there and then update the Home Assistant integration.

I’m on holiday at the moment so won’t be back to my laptop until next weekend at the earliest, but I’ll try and get something dinebwoth it once I’m back. There’s also an issue with ipv6 devices not correctly being marked as offline, something I need to fix to get automations to work with my work machine.

Thank you for looking into those endpoints, your investigation should make it pretty straightforward to update!

beecho01 · July 13, 2024, 12:55pm

Brilliant! Glad to have helped. If you need me to do any tests, drop me a message on here or Github.

beecho01 · July 13, 2024, 1:09pm

If its of any use @typhoon2099, the cgi/cgi_home.js address contains a number of IPv6 details. Specifically activity_ipv6_ll and activity_ipv6 might be of use?

{
  mac:'XX%3AXX%3AXX%3AXX%3AXX%3AXX',
  hostname:'SERIESX',
  ip:'192%2E168%2E1%2E253',
  ipv6:'',
  name:'SERIESX',
  activity:'0',
  os:'Windows%2010',
  device:'PC',
  time_first_seen:'2024%2F06%2F07%2021%3A27%3A50',
  time_last_active:'2024%2F06%2F07%2021%3A27%3A53',
  dhcp_option:'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
  port:'eth0',
  ipv6_ll:'XXXX%3A%3AXXX%3AXXXX%3AXXXX%3AXXXX',
  activity_ip:'0',
  activity_ipv6_ll:'0',
  activity_ipv6:'0',
  device_oui:'NA',
  device_serial:'NA',
  device_class:'NA',
  reconnected:'0'
},
{
  mac:'XX%3AXX%3AXX%3AXX%3AXX%3AXX',
  hostname:'',
  ip:'192%2E168%2E1%2E105',
  ipv6:'XXXX%3AXXXX%3AXXXX%3AXXXX%3AXXXX%3AXXXX%3AXXXX%3AXXXX%2CXXXX%3AXXXX%3AXXXX%3AXXXX%3AXXXX%3AXXXX%3AXXX%3AXXXX',
  name:'XX%3AXX%3AXX%3AXX%3AXX%3AXX',
  activity:'1',
  os:'iOS',
  device:'iPhone',
  time_first_seen:'1981%2F10%2F01%2000%3A00%3A35',
  time_last_active:'2024%2F07%2F13%2013%3A58%3A24',
  dhcp_option:'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
  port:'wl0',
  ipv6_ll:'XXXX%3A%3AXXX%3AXXXX%3AXXXX%3AXXXX',
  activity_ip:'1',
  activity_ipv6_ll:'1',
  activity_ipv6:'1',
  device_oui:'NA',
  device_serial:'NA',
  device_class:'NA',
  reconnected:'0'
},

typhoon2099 · July 13, 2024, 2:53pm

Thanks, those are the values that are being read by the Python package. Try doing a scan with the package from GitHub - jxwolstenholme/btsmarthub_devicelist: Retrieve a list of devices from a bt smarthub or bt smarthub 2 on a local network, you should see those values being returned. There’s an example of how to use it in the readme.

typhoon2099 · July 13, 2024, 2:54pm

My general experience is that the Smart Hub reports IPv6 devices as still being online, even after they’ve been offline for a while. I think it’s something to do with DHCP leases, but I haven’t had time to track the issue down to the root cause.

EDIT: I raised an issue for this aong to.e ago, feel free to expand details there: IPV6 addresses can lock a device in an active state · Issue #25 · jxwolstenholme/btsmarthub_devicelist · GitHub

beecho01 · July 13, 2024, 4:57pm

I’ll see what I can find. I am not sure if it is to do with the default “stateless” IPv6 allocation. I too will have a read up about that too more.

I have found some more addresses that might be of use to you. Not sure if you have these already:

Home Settings

Referer: http://192.168.1.254/basic_-_my_devices.htm
Address: http://192.168.1.254/cgi/cgi_basicMyDevice.js?t=1720889584951
Referer: http://192.168.1.254/basic_-_hub_light_control.htm
Address: http://192.168.1.254/cgi/cgi_basicBrightness.js?t=1720889584951
Referer: http://192.168.1.254/basic_-_mobile_status.htm
Address: http://192.168.1.254/cgi/cgi_basicMobile.js?t=1720889584951

Advanced Settings

Referer: http://192.168.1.254/broadband.htm
Address: http://192.168.1.254/cgi/cgi_broadband.js?t=1720889584951
Referer: http://192.168.1.254/wifi.htm
Address: http://192.168.1.254/cgi/cgi_wifi.js?t=1720889584951
Referer: http://192.168.1.254/firewall.htm
Address: http://192.168.1.254/cgi/cgi_firewall.js?t=1720889584951
Referer: http://192.168.1.254/my_network.htm
Address: http://192.168.1.254/cgi/cgi_myNetwork.js?t=1720889584951
Referer: Multiple Addresses
Address: http://192.168.1.254/cgi/cgi_owl.js?t=1720889584951
Referer: http://192.168.1.254/helpdesk.htm
Address: http://192.168.1.254/cgi/cgi_helpdesk.js?t=1720889584951
Referer: http://192.168.1.254/bt_access_control.htm
Address: http://192.168.1.254/cgi/cgi_btAccessControl.js?t=1720889584951
Referer: http://192.168.1.254/IPv6.htm
Address: http://192.168.1.254/cgi/cgi_IPv6.js?t=1720889584951