Scrape Sensor | How do I test secure connection?

Configuration
1

Hi,

I’m trying to see if I can get power rates from my power company but these are only available after login. The class I’ve deciphered is .rate which identifies the four values shown below (and yes, I know our power is very expensive!).

image

I tried using the following but I get an unknown value from the sensor. I don’t know how to test if I’m even logging in let alone whether any data is returned. Anyone got any tips? Ideally all I need to extract is one value <$0.20c and one value >$0.20c as the morning and evening rates are always the same. The damn energy platform has sent me down a deep dark rabbit hole! :blush:

- platform: scrape
  resource: https://secure.powershop.co.nz/customers/<mycustomernumber>/balance
  name: Scrape PowerShop Rates
  authentication: basic
  username: !secret powershop_user
  password: !secret powershop_pw
  # verify_ssl: false
  select: ".rate"
  headers:
    User-Agent: Mozilla/5.0

P.S I’ve tried using digest authentication too without success.

2

A login via a form is something else than HTTP basic/digest authenticating.
First find out if the values are present in the initial html and not being injected by Javascript (because then they cannot be scraped). Then try multiscrape form-submit.

3

Thanks @danieldotnl , but pretty much all of that went over my head. Might just have to program some averages. :pleading_face:

4

Did you manage to get you poweshop rates?
I tried using Scrape sensor improved - scraping multiple values , but didn’t manage to log in.

5

I gave up. Too hard for me which is a shame. Would be a really handy thing to know.

6

For those wanting to do this, I solved it here using the HACS version of the sensor which is a lot more capable.

2 Likes
7

For those that are interested, I added to the excellent work by @xbmcnut adding the Powershop Special and Weekend rates:

multiscrape:
  - name: Powershop Rates
    resource: 'https://secure.powershop.co.nz/rates'
    log_response: true
    scan_interval: 43200 #every 12hrs
    form_submit:
      submit_once: true
      resource: 'https://secure.powershop.co.nz'
      select: ".content > form"
      input:
        email: !secret powershop_user
        password: !secret powershop_pass
    sensor:
      - unique_id: powershop_offpeak
        name: Powershop Off Peak
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(1) > td.base-rates.current > span.rate.gst_inclusive"
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary     
      - unique_id: powershop_peak
        name: Powershop Peak
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(2) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary 
      - unique_id: powershop_weekend
        name: Powershop Weekend
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(3) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary 
      - unique_id: powershop_offpeak_special
        name: Powershop Off Peak Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(1) > td.base-rates.current > span.rate.gst_inclusive"
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary     
      - unique_id: powershop_peak_special
        name: Powershop Peak Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(2) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary
      - unique_id: powershop_weekend_special
        name: Powershop Weekend Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(3) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary
      - unique_id: powershop_daily_charge
        name: Powershop Daily Charge
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(4) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary      
        on_error:
          log: warning
          value: last
2 Likes