Technicolor TG-789vac V2 DSL Stats

One thing that I’d wanted to do since starting with HA is to get my line stats from my modem into HA so that I don’t have to use that horrid interface that Technicolor modems have. Here in Australia, there is a mixture of different ways to connect to the online world. For me, I’m connected to the NBN via FTTN. This is basically a VDSL2 connection and I’m fortunate enough to get close to the maximum 100/40 speeds on offer.

Unfortunately Technicolor doesn’t make it easy to scrape their modems, and the scrapping tools available for HA didn’t seem to work. Fortunately, I was lucky to stumble upon someone that had written their own scraper just for to specific modems, the TG-1 and the TG-789, which scraps the broadband page and gathers the data for the physical line. This is exactly what I wanted.

After some learning about Linux, and the awk command, I was able to put together a bunch of sensors that get the data for me. The first outputs the data to a file every 30 minutes, and then the other sensors read the relevant values from that file and convert from Kbps to Mbps. All in all, I’m pretty happy with this, and I learnt a little more about awk and Linux.

Here is the end result.

dslstats

If anyone is interested in the script that scrapes the modem, it is https://github.com/mkj/tgiistat.

1 Like

Thanks for this post @cjsimmons

I used your work to create a HA sensor for my TG799vac modem. Code is saved at the location below:

1 Like

Not exactly my work. I just used something I found and made it work with HA. Just a command to run the script, and then a bunch of command line sensors using both tail and awk commands to find the data for each sensor.

Thank you @htpc-helper for turning the original work from the tgiistat project on GitHub into a custom HA sensor for TG799vac modems.

However, upon installing there were a few problems:

  1. async_get_last_state which seems to have been replaced by RestoreEntity.
  2. The uptime strings from my modem don’t contain a space between the digits, hence the s.isdigit() function returned empty. Solved the problem by using regex re.findall(r'\b\d+', uptime)].

@htpc-helper would you consider creating a repository on your GitHub instead of a Gist? Would make contributing easier as would have created a PR for these changes.

Instructions

  1. Create a folder inside custom_components called technicolormodem.

  2. Copy the contents of the below code block and save it to a file called sensor.py inside custom_components/technicolormodem.

    sensor.py code
    # Home Assistant sensor plugin for Technicolor TG799vac modem
    # Modem has been modified with https://github.com/davidjb/technicolor-tg799vac-hacks
    # May work with other similar modems
    
    # Based on the work of Matt Johnston with https://github.com/mkj/tgiistat
    
    # htpc-helper (c) 2018
    # MIT license, see bottom of file
    
    from datetime import timedelta
    import logging
    import voluptuous as vol
    import sys, re, datetime, time
    
    import requests
    import re
    from bs4 import BeautifulSoup
    
    from homeassistant.components.sensor import DOMAIN, PLATFORM_SCHEMA
    from homeassistant.const import (CONF_HOST, CONF_MONITORED_VARIABLES, CONF_NAME)
    import homeassistant.helpers.config_validation as cv
    from homeassistant.helpers.entity import Entity
    from homeassistant.exceptions import PlatformNotReady
    from homeassistant.util import Throttle
    from homeassistant.helpers.restore_state import RestoreEntity
    
    REQUIREMENTS = ['beautifulsoup4==4.6.0', 'requests==2.18.4']
    
    _LOGGER = logging.getLogger(__name__)
    
    DEFAULT_NAME = 'TechnicolorModem'
    
    SENSOR_TYPES = {
       'up_speed': ['Upload Speed', 'Mbit/s'],
       'down_speed': ['Download Speed', 'Mbit/s'],
       'up_maxspeed': ['Upload Max Speed', 'Mbit/s'],
       'down_maxspeed': ['Download Max Speed', 'Mbit/s'],
       'up_power': ['Upload Power', 'dBm'],
       'down_power': ['Download Power', 'dBm'],
       'up_noisemargin': ['Up Noise Margin', 'dB'],
       'down_noisemargin': ['Down Noise Margin', 'dB'],
       'up_attenuation1': ['Up Attenuation 1', 'dB'],
       'up_attenuation2': ['Up Attenuation 2', 'dB'],
       'up_attenuation3': ['Up Attenuation 3', 'dB'],
       'down_attenuation1': ['Down Attenuation 1', 'dB'],
       'down_attenuation2': ['Down Attenuation 2', 'dB'],
       'down_attenuation3': ['Down Attenuation 3', 'dB'],
       'dsl_uptime': ['DSL Uptime', 'seconds'],
       'dsl_mode': ['DSL Mode', None],
       'dsl_type': ['DSL Type', None],
       'dsl_status': ['DSL Status', None],
       'product_vendor': ['Product Vendor', None],
       'product_name': ['Product Name', None],
       'software_version': ['Software Version', None],
       'firmware_version': ['Firmware Version', None],
       'hardware_version': ['Hardware Version', None],
       'serial_number': ['Serial Number', None],
       'mac_address': ['MAC Address', None],
       'uptime': ['Uptime', 'seconds']
    }
    
    SCAN_INTERVAL = timedelta(minutes=1)
    
    PLATFORM_SCHEMA = PLATFORM_SCHEMA.extend({
       vol.Required(CONF_HOST): cv.string,
       vol.Optional(CONF_MONITORED_VARIABLES, default=['dsl_status']):
             vol.All(cv.ensure_list, [vol.In(SENSOR_TYPES)]),
       vol.Optional(CONF_NAME, default=DEFAULT_NAME): cv.string,
    })
    
    def setup_platform(hass, config, add_devices, discovery_info=None):
       try:
             modem = ModemData(config)
       except:
             _LOGGER.warning("Unable to connect to Technicolor Modem")
             raise PlatformNotReady
    
       dev = []
       for sensor in config[CONF_MONITORED_VARIABLES]:
             dev.append(TechnicolorModemSensor(modem, sensor))
       add_devices(dev, True)
    
    class TechnicolorModemSensor(RestoreEntity):
       """Representation of a Technicolor Modem sensor."""
    
       def __init__(self, modem, sensor):
             """Initialize the sensor."""
             self.modem = modem
             self.type = sensor
             self._name = SENSOR_TYPES[sensor][0]
             self._unit_of_measurement = SENSOR_TYPES[sensor][1]
             self._state = None
    
       @property
       def name(self):
             """Return the name of the sensor."""
             return '{} {}'.format(self.modem.config.get(CONF_NAME), self._name)
    
       @property
       def state(self):
             """Return the state of the sensor."""
             return self._state
    
       @property
       def unit_of_measurement(self):
             """Return the unit of measurement of this entity, if any."""
             return self._unit_of_measurement
    
       def update(self):
             """Get the latest data from Technicolor modem and updates the state."""
             self.modem.update()
             if self.modem.data != {}:
             self._state = self.modem.data[self.type]
    
       async def async_added_to_hass(self):
             """Handle all entity which are about to be added."""
             state = await self.async_get_last_state()
             if not state:
                return
             self._state = state.state
    
    class ModemData(object):
       """Get the latest data from the modem"""
       REQUEST_TIMEOUT = 30
    
       def __init__(self, config):
             self.data = {}
             self.config = config
             self.__session = None
             self.__soup = None
    
       def __connect(self):
             """ Authenticates with the modem.
             Returns a session on success or throws an exception
             """
             session = requests.Session()
             return session
    
       def update(self):
             if not self.__session:
                self.__session = self.__connect()
    
             # Process broadband page
             broadband_url = '%s/modals/broadband-modal.lp' % self.config.get(CONF_HOST)
             broadband_data = self.__session.get(broadband_url, timeout = self.REQUEST_TIMEOUT, verify=False)
             self.__soup = BeautifulSoup(broadband_data.text, 'html.parser')
    
             self.data['up_speed'], self.data['down_speed'] = self.__fetch_pair("Line Rate", 'Mbps')
             self.data['up_maxspeed'], self.data['down_maxspeed'] = self.__fetch_pair("Maximum Line rate", 'Mbps')
             self.data['up_power'], self.data['down_power'] = self.__fetch_pair("Output Power", 'dBm')
             self.data['up_noisemargin'], self.data['down_noisemargin'] = self.__fetch_pair("Noise Margin", 'dB')
             self.__fetch_line_attenuation()
             self.data['dsl_uptime'] = self.__fetch_uptime('DSL Uptime')
             self.data['dsl_mode'] = self.__fetch_string('DSL Mode')
             self.data['dsl_type'] = self.__fetch_string('DSL Type')
             self.data['dsl_status'] = self.__fetch_string('DSL Status')
    
             # Change to Mbit/s
             for n in 'down_speed', 'up_speed', 'down_maxspeed', 'up_maxspeed':
                self.data[n] = round(self.data[n], 2)
    
             # Process Gateway
             gateway_url = '%s/modals/gateway-modal.lp' % self.config.get(CONF_HOST)
             gateway_data = self.__session.get(gateway_url, timeout = self.REQUEST_TIMEOUT, verify=False)
             self.__soup = BeautifulSoup(gateway_data.text, 'html.parser')
             names = [
                'Product Vendor',
                'Product Name',
                'Software Version',
                'Firmware Version',
                'Hardware Version',
                'Serial Number',
                'MAC Address',
             ]
             for n in names:
                self.data[n.lower().replace(' ', '_')] = self.__fetch_string(n)
             self.data['uptime'] = self.__fetch_uptime('Uptime')
    
       def __fetch_string(self, title):
             lr = self.__soup.find_all(string=title)
             return lr[0].parent.parent.find_next('span').text
    
       def __fetch_pair(self, title, unit):
             # Find the label
             lr = self.__soup.find_all(string=title)
             # Traverse up to the parent div that also includes the values.
             # Search that div for text with the units (Mbps, dB etc)
             updown = lr[0].parent.parent.find_all(string=re.compile(unit))
             # Extract the float out of eg "4.85 Mbps"
             return (float(t.replace(unit,'').strip()) for t in updown)
    
       def __fetch_line_attenuation(self):
             """ Special case since VDSL has 3 values each for up/down
                eg "22.5, 64.9, 89.4 dB"
                (measuring attenuation in 3 different frequency bands?)
                we construct {up,down}_attenuation{1,2,3}
             """
             title = "Line Attenuation"
             unit = "dB"
             lr = self.__soup.find_all(string=title)
             updown = lr[0].parent.parent.find_all(string=re.compile(unit))
             for dirn, triple in zip(("up", "down"), updown):
                # [:3] to get rid of N/A from the strange "2.8, 12.8, 18.9,N/A,N/A dB 7.8, 16.7, 24.3 dB"
                vals = (v.strip() for v in triple.replace(unit, '').split(',')[:3])
                for n, t in enumerate(vals, 1):
                   self.data['%s_attenuation%d' % (dirn, n)] = float(t)
    
       def __fetch_uptime(self, name):
             """ Returns uptime in seconds """
             uptime = self.__fetch_string(name)
             uptime = [int(s) for s in re.findall(r'\b\d+', uptime)]
             ftr = [86400,3600,60,1]
             uptime = sum([a*b for a,b in zip(ftr[-len(uptime):], uptime)])
             return uptime
    
    # Copyright (c) 2018 htpc-helper
    # All rights reserved.
    #
    # Permission is hereby granted, free of charge, to any person obtaining a copy
    # of this software and associated documentation files (the "Software"), to deal
    # in the Software without restriction, including without limitation the rights
    # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    # copies of the Software, and to permit persons to whom the Software is
    # furnished to do so, subject to the following conditions:
    #
    # The above copyright notice and this permission notice shall be included in all
    # copies or substantial portions of the Software.
    #
    # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    # SOFTWARE.
    
  3. Define a new sensor platform with the below:

    platform: technicolormodem
    host: IP_ADDRESS
    monitored_variables:
      - dsl_status
      - dsl_uptime
      - uptime
      - up_speed
      - down_speed
      - up_maxspeed
      - down_maxspeed
    
    All available monitored_variables
    up_speed
    down_speed
    up_maxspeed
    down_maxspeed
    up_power
    down_power
    up_noisemargin
    down_noisemargin
    up_attenuation1
    up_attenuation2
    up_attenuation3
    down_attenuation1
    down_attenuation2
    down_attenuation3
    dsl_uptime
    dsl_mode
    dsl_type
    dsl_status
    product_vendor
    product_name
    software_version
    firmware_version
    hardware_version
    serial_number
    mac_address
    uptime
    
  4. Restart HA.

Optional:

If you’d like have a human/friendly version of the uptime sensors copy the code from the respective blocks below. Template sensor code courtesy of @SupahNoob:

technicolormodem_uptime_friendly template sensor
platform: template
sensors:
  technicolormodem_uptime_friendly:
    value_template: >-
      {%- set uptime  = states.sensor.technicolormodem_uptime.state | round -%}
                {%- set sep     = ', ' -%}
                {%- set TIME_MAP = {
                    'week': (uptime / 604800) % 10080,
                    'day': (uptime / 86400) % 7,
                    'hour': (uptime / 3600) % 24,
                  'minute': (uptime % 60)
                }
                -%}

                {%- for unit, duration in TIME_MAP.items() if duration >= 1 -%}
                  {%- if not loop.first -%}
                    {{ sep }}
                  {%- endif -%}

                  {{ (duration | string).split('.')[0] }} {{ unit }}

                  {%- if duration >= 2 -%}
                    s
                  {%- endif -%}
                {%- endfor -%}

                {%- if uptime < 1 -%}
                  just now
                {%- endif -%}
technicolormodem_dsl_uptime_friendly template sensor
platform: template
sensors:
  technicolormodem_dsl_uptime_friendly
    value_template: >-
      {%- set uptime  = states.sensor.technicolormodem_dsl_uptime.state | round -%}
                {%- set sep     = ', ' -%}
                {%- set TIME_MAP = {
                    'week': (uptime / 604800) % 10080,
                    'day': (uptime / 86400) % 7,
                    'hour': (uptime / 3600) % 24,
                  'minute': (uptime % 60)
                }
                -%}

                {%- for unit, duration in TIME_MAP.items() if duration >= 1 -%}
                  {%- if not loop.first -%}
                    {{ sep }}
                  {%- endif -%}

                  {{ (duration | string).split('.')[0] }} {{ unit }}

                  {%- if duration >= 2 -%}
                    s
                  {%- endif -%}
                {%- endfor -%}

                {%- if uptime < 1 -%}
                  just now
                {%- endif -%}

Enjoy!

1 Like

Nice work. I’ll give it a go later on and see how it goes. I’m still using the original method from my first post at the top in this thread.

I was going to go down a similar route to you initially as I’m using DSLstats which can be setup to run a webserver to display stats. The issue is trying to get values out of a single pre tag is next to impossible.

Funnily enough after some time and effort I actually succeeded using Node-RED with a many string nodes manipulating the data.

So, as we’ve both down the hard road I think an easier and more elegant way of accomplishing the task is well deserved.

I’ve just given your custom component a go, and it’s not working for me. I really have no idea if this is your component, or the fact that I’ve just rooted my modem and installed some other firmware so I could get IPv6 working with my new ISP. I’m currently running firmware from UNO in the UK. They have the same model as I have here in AU, so no issues there.

I’ve also installed a modded theme for the modem that adds a bucket load of information to the web interface. Maybe this has changed something and thrown the whole tgiistat script from working. But as it stands now, I have no line stats in HA. :sob:

Hi @cryptelli

Thanks for your interest in this project. As you suggested I’ve upgraded the Gist to a repository and the link is below:

I’ve added you as a contributor and look forward to receiving your PR

Thanks for doing the groundwork! :star:

Cheers, PR submitted. I also have another one ready to go for the readme which I’ve updated with a heap of information including instructions and configuration options.

I believe you’ve answered your own question here :wink:. The script uses a python package which scrapes the web interface, if the firmware/theme you’ve installed has changed what it’s expecting then it might not return and if it does the data might be out of order.

Yep, seems to be the case. In the code for the components it looks for broadband-bridge-modal.lp, where as mine is now at broadband-modal.lp.

The only other issue I have is that this doesn’t seem to be logging into my modem. Using the original tgiistat script I can get it to work, but with the component, it doesn’t authenticate.

Yep, that is going to affect things.

As it works with tgiistat the only difference in the authentication section between the two are the missing lines between session = requests.Session() and return session. I’m not too experienced with Python so not sure what needs to be done, @htpc-helper any ideas?

I’m not sure, but I was using a old custom component I found on GitHub that would log in. Not sure if it helps, but there might be something there you guys can use.

Well, I got it working again with authentication. I just modified where the data is pulled from, and removed the snr, and power value lines as they were causing some sort of issue. I don’t need them, so no need for me to worry.

Not sure what your component offers over this one I’m using. Maybe you can combine both into one.

@cryptelli - thanks for your PR, I’ve approved and merged it into master. Please submit your doco PR when you are ready.

@cjsimmons - yes I noticed the difference in the URL when I worked on your code. I thought this may have been because my modem has been modified with this, or possibly due to a different model number. Mine is 799vac and I believe yours is 789vac.

Thank you, have another one!

Thought I should add for information stake that mine is stock (running in bridge) apart from being rooted using the autoflashgui tool and installing telnet.

The hacks you posted @htpc-helper look interesting… :thinking:

I hope you enjoy the hacks @cryptelli. It seems that they have no impact on compatibility with the sensor and the url difference identified by @cjsimmons is purely due to the model difference.

Also thinking back to last year when I wrote this I remember removing the auth code that @cjsimmons had in his solution which is another reason this isn’t working for him. My router doesn’t require username and password.

Thanks again @cryptelli for the PR with the updated documentation. Really appreciate your hard work to polish this and make it useful to more people!

I’ll look at using them for sure, since I’m already rooted.

I wondered why the auth code was missing, tried adding it back but it baulked when trying to connect since as with you I don’t require a user/pass to login.

I’m glad I decided to contribute, what started as a fix for my own use has turned into a improvement which can benefit anyone who stumbles across this custom sensor.

Suggestion, would it be worth while starting a separate post (with reference to this)? Depending on the search term used your initial post doesn’t come up and if people don’t go digging might think it only applies to a TG-789vac

@htpc-helper actually, it less to do with the model and more to do with the fact that I have a modded GUI running on mine. Before the modded GUI, everything was working fine, and the URL you use is the same one I was using. I installed this as the firmware I’m using from UNO in the UK doesn’t show all the line stats that this component requires. This fixes that but changed a few things around

The GUI I’m running now is tch-nginx-gui. This is installed after rooting the modem with autoflashgui.

As for my modem, I have the TG789vac V2. It’s the VANT6 variant used by iiNet in AU. The TG789vac V2 is a slightly basic version of the TG799vac.

Basically, any component I use now I just have to make sure the modal URL being scrapped works with my modem and adjust if needed. Not a big deal for me.

hi guys was trying to set this up and getting some errors. I am also running tch-nginx-gui.the code is point to the right modal file but am getting the below error in my log for every bit value that is trying to be scrapped and obviously no data. Any ideas?

Traceback (most recent call last):
File “/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/entity_platform.py”, line 248, in _async_add_entity
await entity.async_device_update(warning=False)
File “/srv/homeassistant/lib/python3.6/site-packages/homeassistant/helpers/entity.py”, line 379, in async_device_update
await self.hass.async_add_executor_job(self.update)
File “/usr/lib/python3.6/concurrent/futures/thread.py”, line 56, in run
result = self.fn(*self.args, **self.kwargs)
File “/home/homeassistant/.homeassistant/custom_components/technicolormodem/sensor.py”, line 104, in update
self.modem.update()
File “/home/homeassistant/.homeassistant/custom_components/technicolormodem/sensor.py”, line 141, in update
self.data[‘up_speed’], self.data[‘down_speed’] = self.__fetch_pair(“Line Rate”, ‘Mbps’)
File “/home/homeassistant/.homeassistant/custom_components/technicolormodem/sensor.py”, line 181, in __fetch_pair
updown = lr[0].parent.parent.find_all(string=re.compile(unit))
IndexError: list index out of range