Scrape Sensor | How do I test secure connection?

xbmcnut · August 20, 2021, 5:10am

Hi,

I’m trying to see if I can get power rates from my power company but these are only available after login. The class I’ve deciphered is .rate which identifies the four values shown below (and yes, I know our power is very expensive!).

I tried using the following but I get an unknown value from the sensor. I don’t know how to test if I’m even logging in let alone whether any data is returned. Anyone got any tips? Ideally all I need to extract is one value <$0.20c and one value >$0.20c as the morning and evening rates are always the same. The damn energy platform has sent me down a deep dark rabbit hole!

- platform: scrape
  resource: https://secure.powershop.co.nz/customers/<mycustomernumber>/balance
  name: Scrape PowerShop Rates
  authentication: basic
  username: !secret powershop_user
  password: !secret powershop_pw
  # verify_ssl: false
  select: ".rate"
  headers:
    User-Agent: Mozilla/5.0

P.S I’ve tried using digest authentication too without success.

danieldotnl · August 20, 2021, 12:39pm

A login via a form is something else than HTTP basic/digest authenticating.
First find out if the values are present in the initial html and not being injected by Javascript (because then they cannot be scraped). Then try multiscrape form-submit.

xbmcnut · August 21, 2021, 11:17am

Thanks @danieldotnl , but pretty much all of that went over my head. Might just have to program some averages.

joem · February 21, 2022, 1:46am

Did you manage to get you poweshop rates?
I tried using Scrape sensor improved - scraping multiple values , but didn’t manage to log in.

xbmcnut · March 1, 2022, 3:10am

I gave up. Too hard for me which is a shame. Would be a really handy thing to know.

xbmcnut · September 28, 2022, 10:53am

For those wanting to do this, I solved it here using the HACS version of the sensor which is a lot more capable.

Greminn · July 31, 2023, 2:58am

For those that are interested, I added to the excellent work by @xbmcnut adding the Powershop Special and Weekend rates:

multiscrape:
  - name: Powershop Rates
    resource: 'https://secure.powershop.co.nz/rates'
    log_response: true
    scan_interval: 43200 #every 12hrs
    form_submit:
      submit_once: true
      resource: 'https://secure.powershop.co.nz'
      select: ".content > form"
      input:
        email: !secret powershop_user
        password: !secret powershop_pass
    sensor:
      - unique_id: powershop_offpeak
        name: Powershop Off Peak
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(1) > td.base-rates.current > span.rate.gst_inclusive"
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary     
      - unique_id: powershop_peak
        name: Powershop Peak
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(2) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary 
      - unique_id: powershop_weekend
        name: Powershop Weekend
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(3) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary 
      - unique_id: powershop_offpeak_special
        name: Powershop Off Peak Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(1) > td.base-rates.current > span.rate.gst_inclusive"
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary     
      - unique_id: powershop_peak_special
        name: Powershop Peak Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(2) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary
      - unique_id: powershop_weekend_special
        name: Powershop Weekend Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(3) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary
      - unique_id: powershop_daily_charge
        name: Powershop Daily Charge
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(4) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary      
        on_error:
          log: warning
          value: last

xbmcnut · October 1, 2024, 8:48am

Anyone got issues with Powershop NZ as of Monday evening? I’m getting the following error now with no values returned.

2024-10-01 11:20:09.418 ERROR (MainThread) [custom_components.multiscrape.coordinator] Powershop # Exception in form-submit feature. Will continue trying to scrape target page.

EDIT:
I found something about SSL in the issues and modified my config and it seems to have resolved the issue.

  form_submit:
    submit_once: true
    verify_ssl: false

Greminn · October 2, 2024, 8:32pm

@xbmcnut - Yep - seeing the same issues, I added your SSL config as follows, but still getting an error:

multiscrape:
  - name: Powershop Rates
    resource: 'https://secure.powershop.co.nz/rates'
    log_response: true
    scan_interval: 43200 #every 12hrs
    form_submit:
      submit_once: true
      verify_ssl: false
      resource: 'https://secure.powershop.co.nz'
      select: ".content > form"
      input:
        email: !secret powershop_user
        password: !secret powershop_pass

Error:

2024-10-03 09:31:06.757 ERROR (MainThread) [custom_components.multiscrape.coordinator] Powershop Rates # Exception in form-submit feature. Will continue trying to scrape target page.
cannot unpack non-iterable NoneType object

xbmcnut · October 2, 2024, 9:41pm

I’ve just added more SSL false code and rebooted and at least for now, we’re back online. Time will tell.

- resource: 'https://secure.powershop.co.nz/rates'
  name: Powershop
  log_response: true
  verify_ssl: false
  scan_interval: 43200
  form_submit:
    submit_once: true
    verify_ssl: false
    resource: 'https://secure.powershop.co.nz'

danieldotnl · October 4, 2024, 8:04am

Good one! The settings for the form/login page and the page to be scraped can differ, so you have to set them for both.

xbmcnut · October 4, 2024, 8:34am

Yep, I’ve been through two polling cycles now & still working so looks like the ticket. Add-on has been updated meantime too.

Greminn · October 8, 2024, 7:45pm

xbmcnut:

- resource: 'https://secure.powershop.co.nz/rates'
  name: Powershop
  log_response: true
  verify_ssl: false
  scan_interval: 43200
  form_submit:
    submit_once: true
    verify_ssl: false
    resource: 'https://secure.powershop.co.nz'

I have updated as per your config, but on restart there is quite the log. Mainly around the following error:

2024-10-09 08:41:31.760 ERROR (MainThread) [homeassistant.components.sensor] Error adding entity sensor.powershop_offpeak for domain sensor with platform multiscrape
ValueError: Sensor sensor.powershop_offpeak has device class 'monetary', state class 'None' unit 'NZD/kWh' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)
2024-10-09 08:41:31.934 ERROR (MainThread) [homeassistant.components.sensor] Error adding entity sensor.powershop_peak for domain sensor with platform multiscrape
ValueError: Sensor sensor.powershop_peak has device class 'monetary', state class 'None' unit 'NZD/kWh' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)
2024-10-09 08:41:31.936 ERROR (MainThread) [homeassistant.components.sensor] Error adding entity sensor.powershop_weekend for domain sensor with platform multiscrape
ValueError: Sensor sensor.powershop_weekend has device class 'monetary', state class 'None' unit 'NZD/kWh' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)
2024-10-09 08:41:31.938 ERROR (MainThread) [homeassistant.components.sensor] Error adding entity sensor.powershop_offpeak_special for domain sensor with platform multiscrape
ValueError: Sensor sensor.powershop_offpeak_special has device class 'monetary', state class 'None' unit 'NZD/kWh' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)
2024-10-09 08:41:31.939 ERROR (MainThread) [homeassistant.components.sensor] Error adding entity sensor.powershop_peak_special for domain sensor with platform multiscrape
ValueError: Sensor sensor.powershop_peak_special has device class 'monetary', state class 'None' unit 'NZD/kWh' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)
2024-10-09 08:41:31.941 ERROR (MainThread) [homeassistant.components.sensor] Error adding entity sensor.powershop_weekend_special for domain sensor with platform multiscrape
ValueError: Sensor sensor.powershop_weekend_special has device class 'monetary', state class 'None' unit 'NZD/kWh' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)

Full log: Hastebin

danieldotnl · October 9, 2024, 11:09am

Do you have this after every restart or only the first time?

Greminn · October 10, 2024, 7:54pm

Looks like just after restart.

danieldotnl · October 10, 2024, 8:36pm

So you still have this in your logs when you do a restart?

xbmcnut · October 11, 2024, 6:10am

I’m not seeing any issues in the log. Do you have this for your sensors?

      unit_of_measurement: "NZD/kWh"
      value_template: '{{ (value | int) / 100 | float | round(3)}}'
      device_class: monetary

Greminn · October 12, 2024, 2:10am

Yep - here is my config for multiscrape:

multiscrape:
  - name: Powershop Rates
    resource: 'https://secure.powershop.co.nz/rates'
    log_response: true
    scan_interval: 43200 #every 12hrs
    verify_ssl: false
    form_submit:
      submit_once: true
      verify_ssl: false
      resource: 'https://secure.powershop.co.nz'
      select: ".content > form"
      input:
        email: !secret powershop_user
        password: !secret powershop_pass
    sensor:
      - unique_id: powershop_offpeak
        name: Powershop Off Peak
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(1) > td.base-rates.current > span.rate.gst_inclusive"
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary     
      - unique_id: powershop_peak
        name: Powershop Peak
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(2) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary 
      - unique_id: powershop_weekend
        name: Powershop Weekend
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(3) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary 
      - unique_id: powershop_offpeak_special
        name: Powershop Off Peak Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(1) > td.base-rates.current > span.rate.gst_inclusive"
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary     
      - unique_id: powershop_peak_special
        name: Powershop Peak Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(2) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary
      - unique_id: powershop_weekend_special
        name: Powershop Weekend Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(3) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary
      - unique_id: powershop_daily_charge
        name: Powershop Daily Charge
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(4) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary      
        on_error:
          log: warning
          value: last

HasQT · January 5, 2025, 9:24pm

Nice one. Great little HACs job!

I stumbled on this and decided to try it out for Australian Powershop

I couldn’t get it to work using

resource: 'https://secure.powershop.com.au/rates'

I did manage however to get it working using

resource: 'https://secure.powershop.com.au/properties/12345/rates'

The 12345 is your var authenticated_consumer_id = from the page_soup.txt although it shows on your account if you go to the rates page via a web browser.
Perhaps there is a way to scrape that into the yaml, but I’m not that interested in finding that out.

I also added the solar feed in tariff as a sensor.

 multiscrape:  
    - resource: 'https://secure.powershop.com.au/properties/12345/rates'
    # resource: 'https://secure.powershop.com.au/rates'  
      name: Powershop
      log_response: true
      scan_interval: 43200 #every 12hrs
      verify_ssl: false
      form_submit:
        submit_once: true
        verify_ssl: false
        resource: 'https://secure.powershop.com.au'
        select: ".content > form"
        input:
          email: !secret powershop_user
          password: !secret powershop_pass
      sensor:
        - unique_id: powershop_offpeak
          name: Powershop Off Peak
          select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(1) > td.base-rates.current > span.rate.gst_inclusive"
          unit_of_measurement: "AUD/kWh"
          value_template: '{{ (value | int) / 100 | float | round(3)}}'
          device_class: monetary     
        - unique_id: powershop_peak
          name: Powershop Peak
          select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(2) > td.base-rates.current > span.rate.gst_inclusive"      
          unit_of_measurement: "AUD/kWh"
          value_template: '{{ (value | int) / 100 | float | round(3)}}'
          device_class: monetary   
        - unique_id: powershop_daily_charge
          name: Powershop Daily Charge
          select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(4) > td.base-rates.current > span.rate.gst_inclusive"      
          unit_of_measurement: "AUD/kWh"
          value_template: '{{ (value | int) / 100 | float | round(3)}}'
          device_class: monetary
        - unique_id: powershop_feed_in
          name: Powershop Feed In Tariff
          select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(3) > td.base-rates.current > span.rate.gst_inclusive"      
          unit_of_measurement: "AUD/kWh"
          value_template: '{{ (value | int) / 100 | float | round(3)}}'
          device_class: monetary        
          on_error:
            log: warning
            value: last

as aside

this link to the daily kWh total scrape by joem is kinda lost in the 500 odd messages in the other thread. so I shall leave it here for future reference.

Cheers.