Comcast/Xfinity Data Usage Sensor

A couple of things to note here. First, I never run comcast.py. The comcast.py is the OLD way to do things (it doesn’t work anymore).

Second, I use the xfinity-usage python package. I call the xfinity-usage command in that tar file which is a little python wrapper that uses the xfinity-usage library you pip install or pgit clone from robert-alfaro/xfinity-usage. I do use the comcast.sh wrapper script which is called by hass, which then calls xfinity-usage. You have to tell xfinity-usage that you want to use firefox-headless. So you have to specify these options to xfinity-usage for things to work after you install that package: xfinity-usage -b firefox-headless -j

thank you Alan, unfortunately I was unable to replicate the setup.

It would be beneficial for vast US HA community who has 1TB cap with Comcast to perhaps write down a short guide for amatures like me.

does anybody have a working solution for monitoring xfinity usage? I couldn’t get @ralfaro’s component to work (HACS install success, no sensor, no error in logs) . Has xfinity just made it impossible to harvest that data now? Wouldn’t surprise me :roll_eyes:

If anyone would be so kind to perhaps provide a PR with a fix would be greatly appreciated!! I have absolutely no time to get to this.

I managed to get the my Xfinity data usage into Home Assistant through a roundabout way.

  1. I’m using this Docker image to pull my Xfinity usage into InfluxDB.
  2. I then setup an InfluxDB sensor in HomeAssistant to pull the values in.

I set this up today and it seems to be working well. It would be nicer if we had a direct integration into Home Assistant, but this will do for now.

If anyone goes this route, here is my InfluxDB sensor configuration:

- platform: influxdb
  host: !secret influxdb_host
  username: !secret influxdb_user
  password: !secret influxdb_pass
  queries:
    - name: xfinity_data_used
      database: xfinity
      field: used
      group_function: last
      where: "time > now() - 7d"
      measurement: '"comcast_data_usage"'
      unit_of_measurement: "GB"

    - name: xfinity_data_total
      database: xfinity
      field: total
      group_function: last
      where: "time > now() - 7d"
      measurement: '"comcast_data_usage"'
      unit_of_measurement: "GB"
1 Like

Also, here is the card I setup to monitor my Xfinity usage. You’ll need some custom cards and templates to use this.

Card Config:

type: custom:stack-in-card
cards:
  - type: custom:mini-graph-card
    icon: mdi:download-network
    entities:
      - entity: sensor.xfinity_data_used
        name: Xfinity Data
    hours_to_show: 672
    points_per_hour: 1
    aggregate_function: last
    font_size_header: 14
    font_size: 100
    line_width: 3
    group: false
    upper_bound: 1229
    lower_bound: 0
    animate: true
    # line_color: "#c8001d"
    color_thresholds:
      - value: 800
        color: "var(--accent-color)"
      - value: 1000
        color: "var(--label-badge-yellow)"
      - value: 1229
        color: "var(--label-badge-red)"
    show:
      points: false
      labels: false
      labels_secondary: false
      legend: false
    tap_action: none
    style: |
      .name > span { font-weight: 400 !important; text-transform: uppercase; }

  - type: entities
    show_header_toggle: false
    entities:
      - type: custom:bar-card
        name: Xfinity
        icon: mdi:download-network
        entity: sensor.xfinity_data_used
        unit_of_measurement: "GB"
        entity_row: true
        max: 1229
        severity:
          - from: 800
            to: 1000
            color: "var(--label-badge-yellow)"
          - from: 1000
            to: 1229
            color: var(--label-badge-red)
        positions:
          name: off
          value: off
          indicator: off
        #   minmax: inside
        height: 30px
        style: |-
          bar-card-background { margin-right: 13px !important; margin-top: 5px !important; }
          bar-card-value { margin-right: 0 !important; }
        tap_action: none
      - type: custom:bar-card
        name: Month Percent
        icon: mdi:calendar-arrow-right
        entity: sensor.month_percent
        unit_of_measurement: "%"
        entity_row: true
        max: 1229
        positions:
          name: off
          value: off
          indicator: off
        #   minmax: inside
        height: 30px
        style: |-
          bar-card-background { margin-right: 13px !important; margin-top: 5px !important; }
          bar-card-value { margin-right: 0 !important; }
        tap_action: none
    style: |
      .card-header { padding-top: 4px; padding-bottom: 12px; }
      .card-header .name { font-size: 16.8px; font-weight: 400; text-transform: uppercase; opacity: 0.65; }

Date Sensor Templates:


month_days_in_current:
      friendly_name: Days in Current Month
      icon_template: mdi:calendar
      value_template: >-
        {% if now().month in [1,3,5,7,8,10,12] %}
          31
        {% elif now().month in [4,6,9,11] %}
          30
        {% elif now().month == 2 and ((now().year-2000) % 4 > 0) %}
          28
        {% elif now().month == 2 and ((now().year-2000) % 4 == 0) %}
          29
        {% endif %}

    month_current_day:
      friendly_name: Curreny Day in Month
      icon_template: mdi:calendar
      value_template: >-
        {{ now().day }}

    month_percent:
      friendly_name: Percentage Through Month
      icon_template: mdi:calendar
      value_template: >-
        {{ ((states('sensor.month_current_day')|float / states('sensor.month_days_in_current')|float) * 100) | round(0) }}

1 Like

I agree on the direct integration being better but this is a clever workaround. I’ve tried to implement and I got the container setup with what I think is a correct configuration. However, the container doesn’t appear to be writing data to my InfluxDB database. Did you run into this issue at all? I’m using the influxDB addon for HA Supervised so I wonder if the container isn’t finding the database…

Did you configure the container to write to the same database as your Home Assistant install?

The default settings for the container have it writing to a separate database called comcast. If you go down that route, you’ll need to make sure your InfluxDB user has access to that database.

What do the logs of the container show?

docker logs <container_name>

I followed your lead and created a database in InfluxDB called xfinity. My config.yaml calls to that DB like yours does in your example code.

Here’s the relevant bits from the logs:

    return self.find_element(by=By.XPATH, value=xpath)
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element    'value': value})['value']  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute  self.error_handler.check_response(response)
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: {"errorMessage":"Unable to find element with xpath '//*[@ng-bind-html=\"usage.details.userMessage.monthlyUsageState\"]'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Content-Length":"150","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:44829","User-Agent":"selenium/3.14.1 (python linux)"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"xpath\", \"value\": \"//*[@ng-bind-html=\\\"usage.details.userMessage.monthlyUsageState\\\"]\", \"sessionId\": \"b2fc5180-4ca4-11eb-9605-1f55dec06333\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/b2fc5180-4ca4-11eb-9605-1f55dec06333/element"}}
Screenshot: available via screen
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/src/InfluxdbComcast.py", line 144, in <module>
    main()
  File "/src/InfluxdbComcast.py", line 140, in main
    collector.run()
  File "/src/InfluxdbComcast.py", line 93, in run
    res = xfinity.run()
  File "/usr/local/lib/python3.6/site-packages/xfinity_usage/xfinity_usage.py", line 138, in run
    res = self.get_usage()
  File "/usr/local/lib/python3.6/site-packages/xfinity_usage/xfinity_usage.py", line 302, in get_usage
    raise RuntimeError('Unable to find monthly usage div.')
RuntimeError: Unable to find monthly usage div.
Loading Configuration File config.ini
Configuration Successfully Loaded
/usr/local/lib/python3.6/site-packages/selenium/webdriver/phantomjs/webdriver.py:49: UserWarning: Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead
  warnings.warn('Selenium support for PhantomJS has been deprecated, please use headless '
[2021-01-02 02:47:42,083 ERROR] Error getting usage JSON; falling back to scraping page
[2021-01-02 02:48:07,679 CRITICAL] Unable to find monthly usage div on page
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/xfinity_usage/xfinity_usage.py", line 131, in run
    res.update(self.extract_current_from_json(deepcopy(res['raw'])))
  File "/usr/local/lib/python3.6/site-packages/xfinity_usage/xfinity_usage.py", line 267, in extract_current_from_json for item in raw['usageMonths']:KeyError: 'usageMonths'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/xfinity_usage/xfinity_usage.py", line 294, in get_usage
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 394, in find_element_by_xpath
    return self.find_element(by=By.XPATH, value=xpath)
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
    'value': value})['value']
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)

Hmm, it looks like it’s not finding your usage information once logged into the Xfinity website. It doesn’t look like an issue with InfluxDB.

The Docker image I linked to has a couple of fixes for finding elements on the page. The original image it’s forked has some issues

Here are a couple of things I would check:

  1. You’re using the Docker image from TomRoush. (If you follow the readme on the page, it actually has direction for the original image from billimek. Change it to tomroush and you’ll be good.)
  2. You can login to the Xfinity website and you don’t have an weird banners or messages when navigating to your data usage page.

Good catch. I was using the image in the readme. I fixed it to use the forked image and the log looks better but still getting this error:

  File "/usr/local/lib/python3.8/dist-packages/xfinity_usage/xfinity_usage.py", line 207, in do_login
    raise RuntimeError("Login button clicked but no redirect")

not sure what you mean by that.

not sure what you mean by that.

As Xfinity is rolling out the new data caps, on the account page they’re putting some prompts up to let customers know. I think when I first signed in I received a “here’s what’s new about your bill” prompt.

I would login manually and navigate to the data usage section, accepting and clearing any pages that might come up along the way prompting about the new data caps. After that, try again and see if you receive the same error.

1 Like

Gotcha. I checked that from an external connection (to verify nothing was cached) and nothing pops up. It looks like the script should have a clear path to scrape the data. I pulled the screenshot that gets written with that error:

It looks like it’s not even making it through the login page. Can you confirm that your config.ini looks like this:

[COMCAST]
Username = login #just the username without the "@comast.net"
Password = password

Assuming your real username and password are there in your config file (and the comment is removed), then yes, that’s where I inputed my credentials.

Something also to check is the your Docker mount of the config is correct. You could docker exec -it <container_name> /bin/bash and check the config.ini in the container is the one from your local file system.

That was it! I assumed because the logs were showing a configuration success that part was fine but turns out it never pulled in the updated .ini parameters. I’m still getting an influxDB write error but its scraping correctly now. I think the write error is just a permissions thing. Thanks for the help!

1 Like

Thanks Ron, for all the input on this problem. Using your instructions, I have my Xfinity data usage populating the Influxdb. However, when I use your custom template in the Template editor, I get this error: “ZeroDivisionError: float division by zero”. I’m a noob editing Home Assistant, so I’m sure I’m doing something wrong, but can’t figure it out with google. Any pointers would be appreciated.

Love the creative work around! I have the Docker image from TomRoush set up in Docker and running just fine with no errors in the logs. I can see it pointing to the config.ini file correctly, but it’s not sending any data to the InfluxDB. I used “comcast” as the name of the database when I spun up the Docker image and named the database “comcast” in InfluxDB, but nothing is coming through. I suspect that I’m not putting the correct IP address in the config.ini file. I’m running InfluxDB using the HA add-on and I’m not sure if I should be using the name of the docker image as the IP address, if I should be using the IP address of the InfluxDB docker image or the IP address of my host machine with the InfluxDB port. Anyone have any ideas?

Does the log show that it’s scraping the data correctly and the error is with writing to the InfluxDB database or is it not scraping the xfinity page?

If it’s scraping correctly you should have something like this in your container log:

Total: 1229
[{'measurement': 'comcast_data_usage', 'fields': {'used': 340, 'total': 1229, 'unit': 'GB'}}]

If anyone else is looking for another way to get the data usage into Home Assistant I came across this docker container that does will pull it and push it to MQTT. Been working nicely for me so far. Assuming you have MQTT already setup with Home Assistant it’s super simple to get up and running.

1 Like

I found the MQTT solution, and have it installed an communicating properly with my broker. What I’m not sure about is how to handle the message. It seems to be pulling in the entire API page text for the last 6 months of usage details. I’m not even sure that the full message is coming through MQTT, because it seems like it is getting cut off. That said, I’m pretty new to MQTT and have no experience parsing JSON data, so I may just not know what to do with it. If you’re willing to share what you message looks like, or how you have HASS configured to make sense of it, I think I could get the rest of the way there.