Scrape Sensor | How do I test secure connection?

Configuration
1

Hi,

I’m trying to see if I can get power rates from my power company but these are only available after login. The class I’ve deciphered is .rate which identifies the four values shown below (and yes, I know our power is very expensive!).

image

I tried using the following but I get an unknown value from the sensor. I don’t know how to test if I’m even logging in let alone whether any data is returned. Anyone got any tips? Ideally all I need to extract is one value <$0.20c and one value >$0.20c as the morning and evening rates are always the same. The damn energy platform has sent me down a deep dark rabbit hole! :blush:

- platform: scrape
  resource: https://secure.powershop.co.nz/customers/<mycustomernumber>/balance
  name: Scrape PowerShop Rates
  authentication: basic
  username: !secret powershop_user
  password: !secret powershop_pw
  # verify_ssl: false
  select: ".rate"
  headers:
    User-Agent: Mozilla/5.0

P.S I’ve tried using digest authentication too without success.

2

A login via a form is something else than HTTP basic/digest authenticating.
First find out if the values are present in the initial html and not being injected by Javascript (because then they cannot be scraped). Then try multiscrape form-submit.

3

Thanks @danieldotnl , but pretty much all of that went over my head. Might just have to program some averages. :pleading_face:

4

Did you manage to get you poweshop rates?
I tried using Scrape sensor improved - scraping multiple values , but didn’t manage to log in.

5

I gave up. Too hard for me which is a shame. Would be a really handy thing to know.

6

For those wanting to do this, I solved it here using the HACS version of the sensor which is a lot more capable.

2 Likes
7

For those that are interested, I added to the excellent work by @xbmcnut adding the Powershop Special and Weekend rates:

multiscrape:
  - name: Powershop Rates
    resource: 'https://secure.powershop.co.nz/rates'
    log_response: true
    scan_interval: 43200 #every 12hrs
    form_submit:
      submit_once: true
      resource: 'https://secure.powershop.co.nz'
      select: ".content > form"
      input:
        email: !secret powershop_user
        password: !secret powershop_pass
    sensor:
      - unique_id: powershop_offpeak
        name: Powershop Off Peak
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(1) > td.base-rates.current > span.rate.gst_inclusive"
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary     
      - unique_id: powershop_peak
        name: Powershop Peak
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(2) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary 
      - unique_id: powershop_weekend
        name: Powershop Weekend
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(3) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary 
      - unique_id: powershop_offpeak_special
        name: Powershop Off Peak Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(1) > td.base-rates.current > span.rate.gst_inclusive"
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary     
      - unique_id: powershop_peak_special
        name: Powershop Peak Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(2) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary
      - unique_id: powershop_weekend_special
        name: Powershop Weekend Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(3) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary
      - unique_id: powershop_daily_charge
        name: Powershop Daily Charge
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(4) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3)}}'
        device_class: monetary      
        on_error:
          log: warning
          value: last
2 Likes
8

Anyone got issues with Powershop NZ as of Monday evening? I’m getting the following error now with no values returned.

2024-10-01 11:20:09.418 ERROR (MainThread) [custom_components.multiscrape.coordinator] Powershop # Exception in form-submit feature. Will continue trying to scrape target page.

EDIT:
I found something about SSL in the issues and modified my config and it seems to have resolved the issue.

  form_submit:
    submit_once: true
    verify_ssl: false
9

@xbmcnut - Yep - seeing the same issues, I added your SSL config as follows, but still getting an error:

multiscrape:
  - name: Powershop Rates
    resource: 'https://secure.powershop.co.nz/rates'
    log_response: true
    scan_interval: 43200 #every 12hrs
    form_submit:
      submit_once: true
      verify_ssl: false
      resource: 'https://secure.powershop.co.nz'
      select: ".content > form"
      input:
        email: !secret powershop_user
        password: !secret powershop_pass

Error:

2024-10-03 09:31:06.757 ERROR (MainThread) [custom_components.multiscrape.coordinator] Powershop Rates # Exception in form-submit feature. Will continue trying to scrape target page.
cannot unpack non-iterable NoneType object
10

I’ve just added more SSL false code and rebooted and at least for now, we’re back online. Time will tell.

- resource: 'https://secure.powershop.co.nz/rates'
  name: Powershop
  log_response: true
  verify_ssl: false
  scan_interval: 43200
  form_submit:
    submit_once: true
    verify_ssl: false
    resource: 'https://secure.powershop.co.nz'
11

Good one! The settings for the form/login page and the page to be scraped can differ, so you have to set them for both.

12

Yep, I’ve been through two polling cycles now & still working so looks like the ticket. Add-on has been updated meantime too.

13

I have updated as per your config, but on restart there is quite the log. Mainly around the following error:

2024-10-09 08:41:31.760 ERROR (MainThread) [homeassistant.components.sensor] Error adding entity sensor.powershop_offpeak for domain sensor with platform multiscrape
ValueError: Sensor sensor.powershop_offpeak has device class 'monetary', state class 'None' unit 'NZD/kWh' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)
2024-10-09 08:41:31.934 ERROR (MainThread) [homeassistant.components.sensor] Error adding entity sensor.powershop_peak for domain sensor with platform multiscrape
ValueError: Sensor sensor.powershop_peak has device class 'monetary', state class 'None' unit 'NZD/kWh' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)
2024-10-09 08:41:31.936 ERROR (MainThread) [homeassistant.components.sensor] Error adding entity sensor.powershop_weekend for domain sensor with platform multiscrape
ValueError: Sensor sensor.powershop_weekend has device class 'monetary', state class 'None' unit 'NZD/kWh' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)
2024-10-09 08:41:31.938 ERROR (MainThread) [homeassistant.components.sensor] Error adding entity sensor.powershop_offpeak_special for domain sensor with platform multiscrape
ValueError: Sensor sensor.powershop_offpeak_special has device class 'monetary', state class 'None' unit 'NZD/kWh' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)
2024-10-09 08:41:31.939 ERROR (MainThread) [homeassistant.components.sensor] Error adding entity sensor.powershop_peak_special for domain sensor with platform multiscrape
ValueError: Sensor sensor.powershop_peak_special has device class 'monetary', state class 'None' unit 'NZD/kWh' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)
2024-10-09 08:41:31.941 ERROR (MainThread) [homeassistant.components.sensor] Error adding entity sensor.powershop_weekend_special for domain sensor with platform multiscrape
ValueError: Sensor sensor.powershop_weekend_special has device class 'monetary', state class 'None' unit 'NZD/kWh' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)

Full log: Hastebin

14

Do you have this after every restart or only the first time?

15

Looks like just after restart.

16

So you still have this in your logs when you do a restart?

17

I’m not seeing any issues in the log. Do you have this for your sensors?

      unit_of_measurement: "NZD/kWh"
      value_template: '{{ (value | int) / 100 | float | round(3)}}'
      device_class: monetary
18

Yep - here is my config for multiscrape:

multiscrape:
  - name: Powershop Rates
    resource: 'https://secure.powershop.co.nz/rates'
    log_response: true
    scan_interval: 43200 #every 12hrs
    verify_ssl: false
    form_submit:
      submit_once: true
      verify_ssl: false
      resource: 'https://secure.powershop.co.nz'
      select: ".content > form"
      input:
        email: !secret powershop_user
        password: !secret powershop_pass
    sensor:
      - unique_id: powershop_offpeak
        name: Powershop Off Peak
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(1) > td.base-rates.current > span.rate.gst_inclusive"
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary     
      - unique_id: powershop_peak
        name: Powershop Peak
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(2) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary 
      - unique_id: powershop_weekend
        name: Powershop Weekend
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(3) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary 
      - unique_id: powershop_offpeak_special
        name: Powershop Off Peak Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(1) > td.base-rates.current > span.rate.gst_inclusive"
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary     
      - unique_id: powershop_peak_special
        name: Powershop Peak Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(2) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary
      - unique_id: powershop_weekend_special
        name: Powershop Weekend Special
        select: "#main_container > div > div.row > div.rates-table-container.special-rates-table > table > tbody > tr:nth-child(3) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD/kWh"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary
      - unique_id: powershop_daily_charge
        name: Powershop Daily Charge
        select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(4) > td.base-rates.current > span.rate.gst_inclusive"      
        unit_of_measurement: "NZD"
        value_template: '{{ (value | int) / 100 | float | round(3) }}'
        device_class: monetary      
        on_error:
          log: warning
          value: last