Issue with Multiscrape/Scrape

farberm · December 18, 2021, 4:26pm

I am trying to extract the propane usage from Amerigas web site.

Loaded Dashboard page: SessionExpired

Using Chrome and inspect copy selector-…I have the following select value:…

#layoutDiv > main > div.container.pl-0.pr-0.pl-xl-3.pr-xl-3.pl-lg-3.pr-lg-3.pl-md-3.pr-md-3.pl-sm-0.pr-sm-0 > div:nth-child(2) > div.col-12.col-xl-6.col-lg-6.col-md-12.col-sm-12.pl-0.pr-0.pr-xl-3.pr-lg-3.pr-md-0.pr-sm-0 > div.col-12.bg-white.tankanddeliveries-padding.top-margin > div:nth-child(3) > div.col-12.col-xl-4.col-lg-4.col-md-12.col-sm-12.p-0.mt-3.EstimatedTankDiv > div > div.col-12.p-0.lblvalue-Estimatedtank

Here is my config file:

multiscrape:
  - resource: https://myamerigas.amerigas.com/Dashboard/Dashboard
    scan_interval: 3600
    headers:
      User-Agent: Mozilla/5.0
    form_submit:
      submit_once: True
      resource: https://myamerigas.amerigas.com/Login/Login?BrandId=002
      select: "#form-control-valid"
      input:
        email: myemail
        password: mypassword
    sensor:
        - select: "#layoutDiv > main > div.container.pl-0.pr-0.pl-xl-3.pr-xl-3.pl-lg-3.pr-lg-3.pl-md-3.pr-md-3.pl-sm-0.pr-sm-0 > div:nth-child(2) > div.col-12.col-xl-6.col-lg-6.col-md-12.col-sm-12.pl-0.pr-0.pr-xl-3.pr-lg-3.pr-md-0.pr-sm-0 > div.col-12.bg-white.tankanddeliveries-padding.top-margin > div:nth-child(3) > div.col-12.col-xl-4.col-lg-4.col-md-12.col-sm-12.p-0.mt-3.EstimatedTankDiv > div > div.col-12.p-0.lblvalue-Estimatedtank"
          unit_of_measurement: "%"
          name: Tankpercentage
        - unique_id: Tank_percentage

I am getting the following error however…

Error while setting up multiscrape platform for sensor
Traceback (most recent call last):
File “/usr/src/homeassistant/homeassistant/helpers/entity_platform.py”, line 249, in _async_setup_platform
await asyncio.shield(task)
File “/config/custom_components/multiscrape/sensor.py”, line 56, in async_setup_platform
sensor_selector = Selector(hass, conf)
File “/config/custom_components/multiscrape/selector.py”, line 31, in init
raise ValueError(
ValueError: Selector error: either select, select_list or a value template should be provided.

Any help would be greatly appreciated as to what I am doing wrong?

danieldotnl · January 3, 2022, 9:36pm

I think there is a spacing issue in your yaml below the sensor and you also shouldn’t have the - before unique_id.

danieldotnl · January 14, 2022, 11:13am

Please try the latest pre-release 6.0.0 and analyse the debug logging.

farberm · January 14, 2022, 7:53pm

I updated to 6.0.0. I am still having issues with maintaining the value. When I set the interval to 1800 (right side of the graph below) it appears it stops working after about 45 min or so. If I set the interval to something like 600 (left side of the graph below) then it seems to work, however there are gaps when it ready unavailable (grey area)?

I was hoping to query once a day and get the value and have it displayed for the day. Should I set the interval to 43,200 and then it will stay that way for the 24 hour period?

Not sure I have things coded correctly for that however.

Next step it to turn on the logging if there are no other suggestions

multiscrape:
  - resource: 'https://myamerigas.amerigas.com/Dashboard/Dashboard'
    scan_interval: 1800
    form_submit:
      submit_once: True
      resubmit_on_error: True
      resource: 'https://myamerigas.amerigas.com/Login/Login?BrandId=002'
      select: "form"
      input:
        EmailAddress: my email
        Password: "my password"
    sensor:
        - select: "div.lblvalue-Estimatedtank:nth-child(1)"
          value_template: '{{ value.split("%")[0]}}'
          name: tankpercentage
          unique_id: tank_percentage

danieldotnl · January 14, 2022, 7:58pm

I think you should turn on debug logging and find out why it’s becoming unavailable. Is the site unreliable?
If you set it to one a day, it will stay like that all day, but if you are unlucky, it will be unavailable all day…

farberm · January 14, 2022, 8:01pm

ok turned on debugging. Need to wait and see what error I may be getting. Other alternative it to set it every 5 minutes and then use value at the endow the day to calculate usage?

danieldotnl · January 14, 2022, 9:14pm

Thanks a lot. Seems like there was already something going wrong at 15:06:28.
I will dive into this when I’m in front of my computer.

farberm · January 14, 2022, 9:28pm

Thanks. Let me know how I can help more

danieldotnl · January 16, 2022, 4:24pm

I don’t understand why there is no exception text here in the logging message above…
Could you turn on log_response and check out the response to the POST request when this occurs?

AlohaFlyer · September 8, 2023, 6:51pm

Hi were you ever successful getting this data from the amerigas website? I am trying to do the same thing. THank you

farberm · September 11, 2023, 1:53pm

Yes I was… Had to do alot of coding…

  - resource: 'https://myamerigas.amerigas.com/Dashboard/Dashboard'
    scan_interval: 1800
    form_submit:
      submit_once: True
      resubmit_on_error: True
      resource: 'https://myamerigas.amerigas.com/Login/Login'
      select: "form"
      input:
        EmailAddress: !secret amerigas_username
        Password: !secret amerigas_password
    sensor:
        - select: "div.lblvalue-Estimatedtank:nth-child(1)"
          value_template: '{{ value.split("%")[0]}}'
          name: tankpercentage
          unique_id: tank_percentage

  - platform: template
    sensors:
      tank_gallons:
        value_template: "{{(((states('sensor.tank_percentage')) | float)*5.2) }}"
        unit_of_measurement: "gal"
        
      todays_propane_percent:
        value_template: "{{((states('sensor.tank_percentage')) | float) }} "
        unit_of_measurement: "%"
        
# calculate propane kWh - my tank... each percentage change is approx.5.2 gals, thus for each % of 5.2 gallons ~= 140 kWh  From websites   
      
      todays_propane_kwh:
        value_template: "{{ (((states('input_number.daily_propane_percent_used_summation') | float) * 134.35)) }}"
        unit_of_measurement: kWh


# Utility Meters
 
utility_meter:
  daily_energy_propane:
    source: sensor.todays_propane_kwh
    name: 'Propane Used Last 24h'
    cycle: daily
    net_consumption: true
    
  hourly_energy_propane:
    source: sensor.todays_propane_kwh
    name: 'Propane Used Last Hour'
    cycle: hourly
    net_consumption: true

I then run automtions for Percent Propane Yesterday input. set todays input propane percent and yesterdays propane summation

briadelour · November 10, 2023, 7:33pm

I’m using this exact configuration but with my username and password and I can’t get multiscrape to work. @farberm can you confirm this still works for you?

farberm · November 13, 2023, 11:58am

It is no longer working. I have not had the time to determine what they have changed on the site to scape the information but it as working for a long time prior to recently.

briadelour · November 14, 2023, 11:58am

Thanks for the reply. I’ll keep hacking away at it in my free time and post if I figure it out. I have a few other utilities that use the same format of a form inside a div. Some have the id to the form set while others don’t.

briadelour · November 20, 2023, 9:56pm

Just wanted to share that the Amerigas multiscraper is working, details here: Multiscrape with Amerigas propane supplier site - #4 by GraciousLoser