Monitor and Estimate ISP Internet Data Cap Usage

[robwolff3 (Rob Wolff)]

I recently created some sensors and statistics to monitor my ISP (Comcast) data cap after I went over last month.

I’m using VnStat on my router (VnStat supports many platforms), which you can query via REST API, ssh, etc. Using some math we can calculate how much data is being used per month, per day and based on how much being used per day we can estimate how much data you’re going to use by the end of the month.

I am converting GiB from VnStat to GB (opnsense_vnstat_total) as that is the unit of measurement my data cap is calculated in. Update the 1229 value in cap_used_estimated and cap_used_actual to use your data cap amount for calculation.

sensor:
  - platform: command_line
    name: opn_vnstat_total
    unique_id: opn_vnstat_total
    scan_interval: 300
    command: >-
      curl -s -k -u "XXXXXXXXXXXXXXXXXXXXXX":"XXXXXXXXXXXXXXXXXXXXXXXX" https://192.168.1.1/api/vnstat/service/monthly | rev | cut -d'|' -f 5 | rev | tr -d "[:blank:]","GiB"

template:
  - sensor:
      - name: days_in_this_month
        unique_id: days_in_this_month
        state: >-
          {% if now().month in [1,3,5,7,8,10,12] %}
            31
          {% elif now().month in [4,6,9,11] %}
            30
          {% elif now().month == 2 and ((now().year-2000) % 4 > 0) %}
            28
          {% elif now().month == 2 and ((now().year-2000) % 4 == 0) %}
            29
          {% endif %}
      - name: opnsense_vnstat_total
        # this is converting GiB from vnstat to GB which my ISP datacap is calulated in
        unique_id: opnsense_vnstat_total
        unit_of_measurement: GB
        state: >-
          {{ ( states('sensor.opn_vnstat_total') | float * 1.073741824 ) | round() }}
      - name: modem_daily_average
        unique_id: modem_daily_average
        unit_of_measurement: GB
        state: >-
          {{ ( states('sensor.opnsense_vnstat_total') | float / now().day ) | round(2) }}
      - name: modem_monthly_estimated
        unique_id: modem_monthly_estimated
        unit_of_measurement: GB
        state: >-
          {{ ( float(states('sensor.modem_daily_average')) * float(states('sensor.days_in_this_month')) ) | round() }}
      - name: cap_used_estimated
        unique_id: cap_used_estimated
        unit_of_measurement: "%"
        state: >-
          {{ ( states('sensor.modem_monthly_estimated') | float / 1229 * 100 ) | round() }}
      - name: cap_used_actual
        unique_id: cap_used_actual
        unit_of_measurement: "%"
        state: >-
          {{ ( states('sensor.opnsense_vnstat_total') | float / 1229 * 100 ) | round() }}

Now when I get to the end of the month and I have data left over I can resume the torrents I have queued up.

- Rob

---

As an added bonus, here is my current complete network monitoring page. Let me know if you’d like any additional detail on it.

[RobZed (Robert)]

Hey @robwolff3 great stuff, your dashboard looks amazing and I definitely like to know more about it (how you set it up, the different parts, etc…)

Also in regards to the vnstat network monitoring, i have been unable to replicate your setup (specifically the sensor), would you mind sharing how you setup your router and vnstat?

Thanks and much appreciated.

[HunZoo]

Hi @robwolff3.

@robwolff3

      curl -s -k -u "XXXXXXXXXXXXXXXXXXXXXX":"XXXXXXXXXXXXXXXXXXXXXXXX" https://192.168.1.1/api/vnstat/service/monthly | rev | cut -d'|' -f 5 | rev | tr -d "[:blank:]","GiB"

Could you kindly advise where do you get the URL for the curl command? where you have XXXX listed…
And if you could advise which files these need to be configured in. Sorry, im still new to home assistant and trying to get things done
Thanks

[bencorrado]

This looks great! I would encourage you @robwolff3 to add actual tracking from Comcast as well. Here is a docker container that will login and scrape your data from the Xfinity portal. I have found this has worked really well (been running it for about a year) and has Home Assistant over MQTT support.

[RobZed (Robert)]

@bencorrado would you mind sharing how you set it up (HA and Docker)? I’m pretty sure I tried this in the past but i just couldn’t get it to work.

Thanks, much appreciated.

[bencorrado]

Create an extra user in xfinity that you can dedicate to this, they do not need many permissions. I created a dedicated gmail account for this as the container will reset the password as required and use the gmail account to recover it. You will need the credentials for both below. Make sure the gmail account is one of the app specific passwords and the xfinity is not using two factor.

Add the mosquito addon to HAOS

Add the MQTT integration in Home Assistant.

I run the docker container on another machine. My a snip-it of docker compose looks like this (you don’t need any of the labels if you don’t use traefik and set your timezone appropriately):

version: "3"
services:
  xfinity:
    image: zachowj/xfinity-data-usage:latest
    container_name: xfinity
    restart: unless-stopped
    ports:
      - 7879:7878
    volumes:
      - ./xfinity:/config
    environment:
      - TZ=${TZ}
      - LOGGING_LEVEL=debug
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.xfinity.rule=Host(`xfinity.${DOMAIN}`)"
      - "traefik.http.routers.xfinity.service=xfinity"
      - "traefik.http.services.xfinity.loadbalancer.server.port=7879"
      - "traefik.http.routers.xfinity.entrypoints=websecure"
      - "traefik.http.routers.xfinity.tls.certresolver=myresolver"

in ./xfinity/config.yml

xfinity:
  username: "<XFINITY USERNAME>"
  password: "<XFINITY PASSWORD>"
  interval: 60

http:

mqtt:
  host: <HAOS IP ADDRESS (or other MQTT Broker)>
  port: 1883
  username: "user"
  password: "<MQTT-PASSWORD>"
  topic: "xfinity"
  homeassistant:
    prefix: "homeassistant"

imap:
  host: "imap.gmail.com"
  auth:
    user: "<DEDICATED-GMAIL-FOR-HA>@gmail.com"
    pass: "<PASSWORD>"

Now launch it with docker-compose and it should show up as sensor.xfinity_usage in Home Assistant.

[RobZed (Robert)]

Thank you @bencorrado and @robwolff3! I was able to get everything working and setup as desired.

In case anyone else comes by, i setup the xfinity usage monitor nearly exactly how @bencorrado did it (see post above). I then copied the template sensors from @robwolff3 (replacing sensor.opn_vnstat_total with sensor.xfinity_usage).

Dashboard Card:

Lovelace:

type: entities
entities:
  - entity: sensor.xfinity_usage
    name: Current Month
    secondary_info: last-updated
    icon: mdi:chart-line
  - entity: sensor.xfinity_remaining_usage
    name: Remaining Usage
    type: custom:secondaryinfo-entity-row
    secondary_info: "Remaining Days: {{ states('sensor.days_in_this_month') | int - now().day }}"
    icon: mdi:database
  - entity: sensor.cap_used_actual
    name: Current Used
    icon: mdi:chart-line
  - entity: sensor.modem_monthly_estimated
    name: Estimated Month
    type: custom:secondaryinfo-entity-row
    secondary_info: "Month Length: {{ states(''sensor.days_in_this_month'') }} days"
    icon: mdi:finance
  - entity: sensor.cap_used_estimated
    name: Estimated Used
    icon: mdi:finance
  - entity: sensor.modem_daily_average
    name: Daily Average
    icon: mdi:chart-bar
title: Internet Data Usage
state_color: false
show_header_toggle: false

Sensors Template:

template:
  - sensor:
      - name: days_in_this_month
        unique_id: days_in_this_month
        state: >-
          {% if now().month in [1,3,5,7,8,10,12] %}
            31
          {% elif now().month in [4,6,9,11] %}
            30
          {% elif now().month == 2 and ((now().year-2000) % 4 > 0) %}
            28
          {% elif now().month == 2 and ((now().year-2000) % 4 == 0) %}
            29
          {% endif %}
      - name: modem_daily_average
        unique_id: modem_daily_average
        unit_of_measurement: GB
        state: >-
            {% set x = states('sensor.time') %}  {# Needed because otherwise the sensor will not update automatically #}
            {{ ( states('sensor.xfinity_usage') | float / now().day ) | round(2) }}
      - name: modem_monthly_estimated
        unique_id: modem_monthly_estimated
        unit_of_measurement: GB
        state: "{{ ( float(states('sensor.modem_daily_average')) * float(states('sensor.days_in_this_month')) ) | round() }}"
      - name: cap_used_estimated
        unique_id: cap_used_estimated
        unit_of_measurement: "%"
        state: "{{ ( states('sensor.modem_monthly_estimated') | float / 1229 * 100 ) | round() }}"
      - name: cap_used_actual
        unique_id: cap_used_actual
        unit_of_measurement: "%"
        state: "{{ ( states('sensor.xfinity_usage') | float / 1229 * 100 ) | round() }}"
      - name: xfinity_remaining_usage
        unique_id: xfinity_remaining_usage
        unit_of_measurement: "GB"
        state: "{{ state_attr('sensor.xfinity_usage', 'remainingUsage') }}"

Much appreciated and thanks again.

[McCloud]

@RobZed Thanks for the setup on this. The only thing I needed to update was the sensor template under xfinity_remaining_usage. Pulls everything perfect now with the docker container.

For other folks not setup correctly, you also need to install GitHub - custom-cards/secondaryinfo-entity-row: Custom entity row for HomeAssistant, providing additional types of data to be displayed in the secondary info area of the Lovelace Entities card via HACS to get the cards to come in without editing.

template:
  - sensor:
      - name: days_in_this_month
        unique_id: days_in_this_month
        state: >-
          {% if now().month in [1,3,5,7,8,10,12] %}
            31
          {% elif now().month in [4,6,9,11] %}
            30
          {% elif now().month == 2 and ((now().year-2000) % 4 > 0) %}
            28
          {% elif now().month == 2 and ((now().year-2000) % 4 == 0) %}
            29
          {% endif %}
      - name: modem_daily_average
        unique_id: modem_daily_average
        unit_of_measurement: GB
        state: >-
            {% set x = states('sensor.time') %}  {# Needed because otherwise the sensor will not update automatically #}
            {{ ( states('sensor.xfinity_usage') | float / now().day ) | round(2) }}
      - name: modem_monthly_estimated
        unique_id: modem_monthly_estimated
        unit_of_measurement: GB
        state: "{{ ( float(states('sensor.modem_daily_average')) * float(states('sensor.days_in_this_month')) ) | round() }}"
      - name: cap_used_estimated
        unique_id: cap_used_estimated
        unit_of_measurement: "%"
        state: "{{ ( states('sensor.modem_monthly_estimated') | float / 1229 * 100 ) | round() }}"
      - name: cap_used_actual
        unique_id: cap_used_actual
        unit_of_measurement: "%"
        state: "{{ ( states('sensor.xfinity_usage') | float / 1229 * 100 ) | round() }}"
      - name: xfinity_remaining_usage
        unique_id: xfinity_remaining_usage
        unit_of_measurement: "GB"
        state: "{{ state_attr('sensor.xfinity_usage', 'allowable_usage') - int(states('sensor.xfinity_usage')) }}"

[McCloud]

Just for reference, if you want to use the pure VNStat implementation, you’ll have a sensor that looks like:

command_line:
  - sensor:
      name: opn_vnstat_total
      unique_id: opn_vnstat_total
      scan_interval: 300
      command: >-
        curl -s -k -u "XXX":"YYY" https://192.168.1.1:8443/api/vnstat/service/monthly | rev | cut -d'|' -f 5 | rev | tr -d "[:blank:]","GiB"

Where XXX and YYY are the API keys generated from your opnsense setup under user manager.

[ThisWayToo]

Has anyone come up with a tidy way to deal with how VnStat reports the values when they are still under 1 GiB? What I have run into can be seen here in this truncated table from VnStat:

      day        rx      |     tx      |    total    |   avg. rate
 ------------------------+-------------+-------------+---------------
  07/05/24      3.72 GiB |  127.46 MiB |    3.84 GiB |  382.24 kbit/s
  07/06/24      6.38 GiB |  211.49 MiB |    6.59 GiB |  655.23 kbit/s
  07/07/24    185.16 MiB |   26.21 MiB |  211.37 MiB |   53.25 kbit/s
 ------------------------+-------------+-------------+---------------

The above is from the “Daily Statistics” table in VnStat, but the same basic problem exists with the monthly as well. The Curl query is looking for the value in “GiB”, but VnStat presents it in MiB whenever that value is less than 1 GiB.

I have been able to see the values in MiB, but they show up as a “211.37M” string. But once the 1 GiB boundary has been crossed the values are then a number vs a string. So in essence it is a string value vs a numerical value issue. And I don’t see a way to force VnStat to always present the data as GiB regardless if it is a sub 1 GiB value.

My end goal is that I want to display actual near real-time daily usage based on VnStat daily values, but this (changing) MiB vs GiB has tripped me up.

Edit to add:

Opted to use utility_meter: to extract the usage info for a days use and graphing. So far seems to be doing what I want and expect it to.

# Daily Internet usage from VnStat
utility_meter:
  daily_internet:
    source: sensor.opnsense_vnstat_total
    cycle: daily

The basic issue I outlined for GiB values from VnStat when below 1GiB does still exist. But in my case it will only be present for a few hours of the first day of each month. I can live with that.

[exorcist]

Hi Pfsense only creates an api key not a secret ID . any idea how can i modify the above curl command to get data value from vnstat