Multiscrape with Amerigas propane supplier site

A little background, the company that supplies my LP, Amerigas, added a otodata tank reader to measure tank levels. I can use an app called “Nee-Vo” to pull information they are sending to Amerigas’s portal. Otodata has a website but it’s only open to the suppliers and customers who want the data can use the phone app or use the supplier’s portal.

I’m trying to get my propane tank reading into my Home Assistant setup so I figured I would try Multiscrape as it appears to do what I need. From what I’ve read it looks like there are other Amerigas customers trying to do this but no one has posted success. Reading other posts and the awesome documentation on multiscrape posted by @danieldotnl I thought I had it set up correctly but I must be missing something.

  • Core 2023.11.1
  • Supervisor 2023.11.0
  • Operating System 11.1
  • Frontend 20231030.1

This is what I’m trying in my configuration.yaml after loading the multiscrape custom component from HACS.

multiscrape:
  - resource: "https://www.myamerigas.com/Dashboard/Dashboard"
    name: Amerigas Scraper
    scan_interval: 300
    headers:
      User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
      referer: "https://www.myamerigas.com/"
      origin: "https://www.myamerigas.com"
      host: "www.myamerigas.com"
    form_submit:
      submit_once: True
      resubmit_on_error: True
      resource: "https://www.myamerigas.com/Login/Login?BrandId=002"
      select: ".body > div.container-fluid.p-0 > div > main > div > form.Login"
      input:
        EmailAddress: "myemaillogin"
        Password: "mypassword"
    sensor:
      - select: "#layoutDiv > div.container.pl-0.pr-0.pl-xl-3.pr-xl-3.pl-lg-3.pr-lg-3.pl-md-3.pr-md-3.pl-sm-0.pr-sm-0 > div:nth-child(2) > div.col-12.col-xl-6.col-lg-6.col-md-12.col-sm-12.pr-0.pl-0.pl-xl-3.pl-lg-3.pl-md-0.pl-sm-0 > div.col-12.bg-white.tankanddeliveries-padding.top-margin > div:nth-child(3) > div.col-12.col-xl-4.col-lg-4.col-md-12.col-sm-12.p-0.mt-3.EstimatedTankDiv > div > div.col-12.p-0.lblvalue-Estimatedtank"
        name: LP Percentage
        unique_id: tankpercentage
        unit_of_measurement: "%"
      - select: "#layoutDiv > div.container.pl-0.pr-0.pl-xl-3.pr-xl-3.pl-lg-3.pr-lg-3.pl-md-3.pr-md-3.pl-sm-0.pr-sm-0 > div.col-12.pt-5.row.mx-0.top-mobile > div.col-12.col-xl-6.col-lg-6.col-md-6.col-sm-12.pl-0.pr-0.pl-xl-0.pr-xl-3.pl-lg-0.pr-xl-3.pl-md-0.pr-md-3 > div > div.col-12.p-0.pb-4 > div:nth-child(2) > div.col-12.col-xl-6.col-lg-6.col-md-6.col-sm-12.p-0.mt-3 > div.col-12.p-0.lblprice.text-center.text-xl-left.text-lg-left.text-md-left"
        name: AmeriGas Payment Due
        unique_id: lpbilldue
        unit_of_measurement: "$"

This is the source I’m using for the resources and select:

This is the error, I can’t seem to get past the form-submit formatting.

This error originated from a custom integration.

Logger: custom_components.multiscrape.coordinator
Source: custom_components/multiscrape/coordinator.py:75
Integration: Multiscrape scraping component (documentation, issues)
First occurred: 7:28:39 PM (2 occurrences)
Last logged: 7:28:45 PM

Amerigas Scraper # Exception in form-submit feature. Will continue trying to scrape target page. Could not find form

Any help would be greatly appreciated.

Once I can figure out the form-submit issue, these are the two scrapes I want to use for sensors:

Tank reading:

Next Payment Due info:

I did notice when I change the select under form_submit to this:

select: "body > div.container-fluid.p-0 > div > main > div > form"

I dropped the period before body and remove the .Login at the end, the error message changes slightly to:

Amerigas Scraper # Exception in form-submit feature. Will continue trying to scrape target page.

It no longer says “Could not find form” but I still can’t get past the form-submit problem.

Did you get this working?

I just tried and it did not work.

I have no experience scraping, so I can’t be of much help.

Thanks!

Nope. Can’t get it to work and another person who posted success previously got back to me and said it stopped working for them months ago and they haven’t had a chance to figure out what is wrong. I have multiscrape commented out in my configuration.yaml and uncomment it out every once in a while to test it. Been working on a DIY Garage Door opener project so I haven’t had chance to revisit.

I also tried a few other utilities and get the same types of errors. Below is the latest multiscrape.yaml I have. You can see my attempts at variations of the select commented out.

- resource: "https://www.myamerigas.com/Dashboard/Dashboard"
  name: Amerigas Scraper
  scan_interval: 300
  #  headers:
  #    User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
  #    referer: "https://www.myamerigas.com/"
  #    origin: "https://www.myamerigas.com"
  #    host: "www.myamerigas.com"
  form_submit:
    submit_once: True
    resubmit_on_error: True
    resource: "https://www.myamerigas.com/Login/Login?BrandId=002"
    #select: "body > div.container-fluid.p-0 > div > main > div > form"
    select: "div > form"
    #select: "form"
    input:
      EmailAddress: !secret amerigas_username
      Password: !secret amerigas_password
  sensor:
    - select: "div.col-12.p-0.lblvalue-Estimatedtank"
      #    - select: "layoutDiv > div.container.pl-0.pr-0.pl-xl-3.pr-xl-3.pl-lg-3.pr-lg-3.pl-md-3.pr-md-3.pl-sm-0.pr-sm-0 > div:nth-child(2) > div.col-12.col-xl-6.col-lg-6.col-md-12.col-sm-12.pr-0.pl-0.pl-xl-3.pl-lg-3.pl-md-0.pl-sm-0 > div.col-12.bg-white.tankanddeliveries-padding.top-margin > div:nth-child(3) > div.col-12.col-xl-4.col-lg-4.col-md-12.col-sm-12.p-0.mt-3.EstimatedTankDiv > div > div.col-12.p-0.lblvalue-Estimatedtank"
      name: LP Percentage
      unique_id: tankpercentage
      unit_of_measurement: "%"
    #    - select: "layoutDiv > div.container.pl-0.pr-0.pl-xl-3.pr-xl-3.pl-lg-3.pr-lg-3.pl-md-3.pr-md-3.pl-sm-0.pr-sm-0 > div.col-12.pt-5.row.mx-0.top-mobile > div.col-12.col-xl-6.col-lg-6.col-md-6.col-sm-12.pl-0.pr-0.pl-xl-0.pr-xl-3.pl-lg-0.pr-xl-3.pl-md-0.pr-md-3 > div > div.col-12.p-0.pb-4 > div:nth-child(2) > div.col-12.col-xl-6.col-lg-6.col-md-6.col-sm-12.p-0.mt-3 > div.col-12.p-0.lblprice.text-center.text-xl-left.text-lg-left.text-md-left"
    - select: "div.col-12.p-0.lblprice.text-center.text-xl-left.text-lg-left.text-md-left"
      name: AmeriGas Payment Due
      unique_id: lpbilldue
      unit_of_measurement: "$"

- resource: "https://schedulepayment.com/aqua/Account/AccountSummary"
  name: AquaNC Scraper
  scan_interval: 300
  #  headers:
  #    User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
  #    referer: "https://www.schedulepayment.com/"
  #    origin: "https://www.scheduledpayment.com"
  #    host: "www.schedulepayment.com"
  form_submit:
    submit_once: True
    resubmit_on_error: True
    resource: "https://schedulepayment.com/aqua"
    #select: "body_id > div.main_container.clearfix > div > section > div > div > div.large_screen_left.col-sm-4.col-lg-4 > div > div.login_form > form"
    select: "div > form"
    #select: "form"
    input:
      UserName: !secret aquanc_username
      Password: !secret aquanc_password
  sensor:
    - select: "div.table_data > tbody > tr > td:nth-child(3)"
      name: AquaNC Balance
      unique_id: waterbilldue
      unit_of_measurement: "$"

- resource: "https://p-auth.duke-energy.com/my-account/dashboard"
  name: Duke Energy Scraper
  scan_interval: 300
  #  headers:
  #    User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
  #    referer: "https://www.duke-energy.com/"
  #    origin: "https://www.duke-energy.com"
  #    host: "www.duke-energy.com"
  form_submit:
    submit_once: True
    resubmit_on_error: True
    resource: "https://www.duke-energy.com/my-account/sign-in"
    #select: "main-content > section > div > form"
    #select: "main-content > section > div.form"
    select: "div > form"
    #select: "form"
    input:
      Split-Sign-In-signInUsername_tealeaf-unmask: !secret dukeenergy_username
      #usernameInput: !secret dukeenergy_username
      Split-Sign-In-signInPassword: !secret dukeenergy_password
      #passwordInput: !secret dukeenergy_password
  sensor:
    #    - select: "page-components > div.CCDashboard.ng-scope > div > div.CCDashboard__row.due-bnp-widget > div.CCDashboard__col.CCDashboard__col--auto > div.CCDashboard__col__container.ng-scope > div > div.CC-Widget__container.ng-scope > div.CC-Widget__body.CC-Widget-due"
    - select: "div.CC-Widget__body.CC-Widget-due"
      name: DukeEnergy Balance
      unique_id: electricbilldue
      unit_of_measurement: "$"

- resource: "https://www.spectrum.net/account-summary"
  name: Spectrum Scraper
  scan_interval: 300
  #  headers:
  #    User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
  #    referer: "https://www.spectrum.net/"
  #    origin: "https://www.spectrum.net"
  #    host: "www.spectrum.net"
  form_submit:
    submit_once: True
    resubmit_on_error: True
    resource: "https://id.spectrum.net/login?account_type=RESIDENTIAL&client_id=consumer_portal&code_challenge=..."
    select: "div > form"
    #main-content-card > ng-component > div > form
    input:
      cc-username: !secret spectrum_username
      #cc-username
      cc-user-password: !secret spectrum_password
      #cc-user-password
  sensor:
    - select: "amountDue"
      name: Spectrum Balance
      unique_id: spectrumbilldue
      unit_of_measurement: "$"

Yo folks, I really appreciate the timeliness of this post. I just got an SDR and integrated my power meter, so the gas was next on the list. I feel like I got really lucky finding this post since scraping the webpage wasn’t something I even knew was possible - this opens up a lot of cool doors.

Anyway, I think I’ve got it licked. Config is below, but here’s a summary of what I found:

Locating the login form

There’s really only one form on the page, so a selector of “form” should technically work. I wanted to future proof it a bit, so I used a selector that matches form elements with a name attribute of “Login”.

Scraping the Dashboard

Looking at the debug logs I was seeing success but there was no output at all for the Dashboard page fetch, just a selector failed message. This was really puzzling, so I started to think that maybe there was something weird going on. I watched the network tab in dev tools and noticed that before it makes a request to /Dashboard/Dashboard, it makes a quick GET request to another URL: https://www.myamerigas.com/eSign/eSign?refId=youmyguy%2540gmail.com where the refId is a double-urlencoded representation of the email address you logged in with. (%2540 is the @ symbol)

I initially thought that this was going to be complicated, and that this request was probably setting a cookie that was used in a subsequent request to the Dashboard. The first thing I did was block requests to that eSign URL. Going through the login flow again with that url blocked hangs it up. Looking at the headers what happens is the eSign request has a header that redirects to /Dashboard/Dashboard so if you block it you never get that redirect and there’s nowhere to go from the login form.

But what’s really interesting and really saved the day: the response body of the GET to /eSign/eSign includes the same payload as the dashboard!

Ensure you have multiscrape: !include multiscrape.yaml in configuration.yaml

# /homeassistant/multiscrape.yaml 
- resource: "https://www.myamerigas.com/eSign/eSign?refId=youmyguy%2540gmail.com"
  name: Amerigas Scraper
  scan_interval: 43200
  timeout: 60
  headers:
    User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
    referer: "https://www.myamerigas.com/"
    origin: "https://www.myamerigas.com"
    host: "www.myamerigas.com"
  form_submit:
    submit_once: True
    resubmit_on_error: False
    resource: "https://www.myamerigas.com/Login/Login"
    select: "form[name='Login']"
    input:
      EmailAddress: "[email protected]"
      Password: "nobutforreal"
  sensor:
    - select: ".EstimatedTankDiv .lblvalue-Estimatedtank"
      name: Amerigas Tank Percentage Raw
      unique_id: multiscrape_amerigas_tank_percentage_raw_3420398 
      # these unique ids aren't strictly necessary, but make sure you have them if 
      # you plan on creating the template sensors in the next section
    - select: ".lblprice"
      name: Amerigas Billing Amt Due Raw
      unique_id: multiscrape_amerigas_amt_due_raw_3420398

I set the scan interval at 12 hours because

  1. I have a 1000 gallon tank - that number isn’t going to change very often
  2. I don’t think Amerigas updates it very frequently. When I was setting this up the value was from 8+ hours ago
  3. I don’t want to make a ton of requests or else they might catch on and do the myQ thing and break our integration.

Optional: ditch the units and make numeric values

The scraped values are strings that include non-numeric values. This means they can’t be used in Gauges, and it’s always easier to add back units for most things, so I wanted to create new sensors based on a template that strips these.

# /homeassistant/configuration.yaml
# Template sensors to process the scraped data
template:
  - sensor:
      - name: "Amerigas Tank Percentage"
        state: >
          {% set value = states('sensor.multiscrape_amerigas_tank_percentage_raw_3420398') %}
          {{ value | replace('%', '') | int }}
      - name: "Amerigas Billing Amount Due"
        state: >
          {% set value = states('sensor.multiscrape_amerigas_amt_due_raw_3420398') %}
          {{ value | regex_replace('[^\\d.]', '') | float }}

Don’t forget to use the Developer Tools section in HA to reload and generate the template sensors

  • Check configuration
  • Restart > Quick Reload

This should cause the multiscrape to kick off, get your values, and once that’s done your template sensors should also have values.

Cards

type: gauge
entity: sensor.amerigas_tank_percentage
name: Percentage Remaining
unit: '%'
severity:
  green: 30
  yellow: 20
  red: 0
needle: true
max: 100

image

I didn’t bother making a card for the billing amount due - that’s not actually a stat I care about since it’s 100% of the time $0.00, but if I did, I think I’d want a statistics card, which isn’t compatible with the template sensor as defined. I think maybe this page is telling me how to do this, but I leave that as an exercise for the reader.

1 Like

I don’t have Amerigas but this is really great work nonetheless. Just wanted to comment that you can put the template directly into your multiscrape config rather than creating separate template sensors. For example:

sensor:
    - select: ".EstimatedTankDiv .lblvalue-Estimatedtank"
      name: Amerigas Tank Percentage
      unique_id: a01ebd10-0122-4ba8-94eb-cd62bac31a77 
      value_template: "{{ value | select('in', '.0123456789') | join | float }}"
      state_class: measurement
      unit_of_measurement: "%"

The template above will only pass through numbers and ignore everything else like units and spaces.

1 Like

Oh, yeah, that’s a lot better. I kinda gave up on trying to understand the usage of the fields noted in the documentation. Thanks Rick!

:beers:
Thanks a ton!

I have a few alternative cards to share.

This one changes color based on percentage:
image

entities:
  - entity: sensor.amerigas_multiscrape_raw
    name: Propane Tank Level
    icon: mdi:propane-tank
decimal: 1
max: 100
min: 10
severity:
  - color: '#0080FF'
    from: '81'
    to: '100'
  - color: '#40bf40'
    from: '31'
    to: '80'
  - color: '#bf9540'
    from: '21'
    to: '30'
  - color: '#c0392b'
    from: '0'
    to: '20'
direction: up
height: 150px
width: 200px
speed: '2'
positions:
  indicator: outside
style: |-
  .card-header {
    font-size: 18px;
  }
  bar-card-value, bar-card-name {
    font-size: 12px;
    transform-origin: 0 0;
    transform: rotate(270deg);
    text-shadow: 1px 1px #0008;
    white-space: nowrap;
  }
  bar-card-value {
    margin-right: auto;
    margin-left: 6px;
    margin-bottom: auto;
    margin-top: auto;
  }
  bar-card-name {
    margin-right: auto;
    margin-left: 0px;
    margin-bottom: -130px;
    margin-top: 130px;
    right: 20px;
  }
  bar-card-currentbar, bar-card-backgroundbar {
    border-radius: 12px;
    border: 1px solid #DDD9;
  }
stack: horizontal
type: custom:bar-card

This one displays the top of the shading flowing like fluid.

image

type: custom:fluid-level-background-card
card:
  show_name: true
  show_icon: true
  show_state: true
  type: glance
  state_color: true
  entities:
    - entity: sensor.amerigas_multiscrape_raw
      icon: mdi:propane-tank
  columns: 1
  title: Propane Tank Level
level_color:
  - 0
  - 191
  - 255
entity: sensor.amerigas_multiscrape_raw

1 Like

These are fun looking cards!

@GraciousLoser - Nice fix!

Based on what you found out here, it looks like you might be able to help me with a similar log-in struggle I’m having over here - Help with using Multiscrape with Form-submit to log into Lose It!

If you had a minute or two, that would be super nice of you.