Scrape sensor improved - scraping multiple values

It was a response to Jonsson9, not to you :wink:

Aah sorry lol haha.
Anyway still thanks!

Yes, Managed to get it working how I wanted it. Again much appreciated was the last piece of the puzzle. Cheers!

Hi!
I am trying to scrape fuel prices from these two sites fuelGR | Πρατήρια και Τιμές Καυσίμων and http://oil.gge.gov.gr/
but I can’t get it to work.

- resource: https://fuelgr.gr/web/?lat=37.849917&lng=23.768234
    scan_interval: 3600
    sensor:
      - unique_id: gas_eko
        name: EKO
        select: "#map > div.leaflet-pane.leaflet-map-pane > div.leaflet-pane.leaflet-marker-pane > div:nth-child(2) > div > div.custom_marker_label.avg.ratebar.green"

this doesn’t work at all

and

- resource: http://oil.gge.gov.gr/
    scan_interval: 3600
    sensor:
      - unique_id: gas_unleaded
        name: Βενζίνη αμόλυβδη
        select: "#post-15266 > div.entry-content > table > tbody > tr:nth-child(6) > td:nth-child(2)"

provides the price of a fixed date.

can anyone help me and tell me where the error is?

  - resource: https://www.printables.com/de/social/259978-johination/about
    scan_interval: 14400 # alle 4 Stunden > Update #
    sensor:
      - unique_id: printables_downloads
        name: "Printables Downloads"
        icon: mdi:download-network-outline
        select: "#app-root > app-baselayout > div > div > ng-component > app-sticky-sidebar > aside > div > app-social-sidebar > div > div > div > div.stats > div:nth-child(1) > div:nth-child(1) > span"
        attributes:
          - name: "likes:"
            select: "#app-root > app-baselayout > div > div > ng-component > app-sticky-sidebar > aside > div > app-social-sidebar > div > div > div > div.stats > div:nth-child(1) > div:nth-child(2) > span"
          - name: "folgende:"
            select: "#app-root > app-baselayout > div > div > ng-component > app-sticky-sidebar > aside > div > app-social-sidebar > div > div > div > div.stats > div:nth-child(2) > a:nth-child(1) > span"
          - name: "folgend:"
            select: "#app-root > app-baselayout > div > div > ng-component > app-sticky-sidebar > aside > div > app-social-sidebar > div > div > div > div.stats > div:nth-child(2) > a:nth-child(2) > span"

get the following errors

Logger: custom_components.multiscrape.sensor
Source: custom_components/multiscrape/sensor.py:163
Integration: Multiscrape scraping component ([documentation](https://github.com/danieldotnl/ha-multiscrape), [issues](https://github.com/danieldotnl/ha-multiscrape/issues))
First occurred: 15:20:25 (1 occurrences)
Last logged: 15:20:25

Scraper_noname_3 # Printables Downloads # Unable to scrape data: Could not find a tag for given selector. Consider using debug logging and log_response for further investigation.

and

Logger: custom_components.multiscrape.entity
Source: custom_components/multiscrape/entity.py:80
Integration: Multiscrape scraping component (documentation, issues)
First occurred: 15:20:25 (3 occurrences)
Last logged: 15:20:25

Scraper_noname_3 # Printables Downloads # likes # Unable to extract data from HTML
Scraper_noname_3 # Printables Downloads # folgende # Unable to extract data from HTML
Scraper_noname_3 # Printables Downloads # folgend # Unable to extract data from HTML

regards
johann

hi, i need your help.
does not work. what am I doing wrong? thank you

multiscrape:
  - name: Offerta Enel E-light
    resource: https://www.enel.it/it/luce-e-gas/luce/offerte/e-light
    scan_interval: 36
    sensor:
      - unique_id: enel_f1
        name: Enel E-light F1
        select: "#splide-flex-card-slide03 li:nth-child(1) > span.Pcwil-tab > span > span"
        #value_template: '{{ value | replace (",", ".") }}'
        unit_of_measurement: "€/KWh"
      - unique_id: enel_f23
        name: Enel E-light F23
        select: "#splide-flex-card-slide03 li:nth-child(2) > span.Pcwil-tab > span > span"
        #value_template: '{{ value | replace (",", ".") }}'
        unit_of_measurement: "€/KWh"

Hi there. Did you ever get this working with Powershop?

Hey!
Yes, it is working.

  - resource: 'https://secure.powershop.co.nz/customers/123456/balance'
    name: Powershop
    log_response: true
    scan_interval: 3600
    form_submit:
      submit_once: true
      resource: 'https://secure.powershop.co.nz'
      select: ".content > form"
      input:
        email: !secret powershop_user
        password: !secret powershop_pass
    sensor:
      - unique_id: powershop_daily_consumption
        name: Powershop daily consumption
        select: ".usage-text > span"
        device_class: energy
        unit_of_measurement: "kWh"
        value_template: '{{ value.split(" ")[0] }}'
        on_error:
          log: warning
          value: last

Legend! Would you mind sharing your code? I’m pulling my hair out, what’s left of it. :thinking:

Just edited my previous comment.
Sorry @Crashbash2020 , somehow I missed your question, in August.

Wow, never would have figured that out. I was way off the mark. Trying now.

Arrr, I see you are pulling the usage. I want to pull the price of my peak and offpeak rates. I’ll use your code base to trying figure that out. Thanks for sharing. Very much appreciated.

You’re welcome.
I was struggling too, but @danieldotnl was kind to help me out.
Scraping the rates is a good idea too, but I’m mostly using/buying “special packs”, so the price fluctuates. I’m more interested in usage.
If you manage to scrape the rates, please share the code.

I’m getting an error when using your code and the sensor value is unknown. Bedtime now so I’ll investigate tomorrow.

2022-09-26 23:30:48.861 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Finished fetching multiscrape data in 4.305 seconds (success: True)
2022-09-26 23:30:48.888 DEBUG (MainThread) [custom_components.multiscrape.sensor] Powershop # Powershop daily consumption # Setting up sensor
2022-09-26 23:30:48.890 DEBUG (MainThread) [custom_components.multiscrape.sensor] Powershop # Powershop daily consumption # Start scraping to update sensor
2022-09-26 23:30:48.894 DEBUG (MainThread) [custom_components.multiscrape.scraper] Powershop # Powershop daily consumption # Tag selected: None
2022-09-26 23:30:48.894 DEBUG (MainThread) [custom_components.multiscrape.form] Powershop # Exception occurred while scraping, will try to resubmit the form next interval.
2022-09-26 23:30:48.894 WARNING (MainThread) [custom_components.multiscrape.sensor] Powershop # Powershop daily consumption # Unable to scrape data: Could not find a tag for given selector.
Consider using debug logging and log_response for further investigation.
2022-09-26 23:30:48.895 DEBUG (MainThread) [custom_components.multiscrape.sensor] Powershop # Powershop daily consumption # On-error, keep old value: None
2022-09-26 23:30:48.895 DEBUG (MainThread) [custom_components.multiscrape.entity] Powershop # Powershop daily consumption # Updated sensor and attributes, now adding to HA

Did you change the 123456 from the url?
I’m not at home for the next few days, so I have limited access to my config.

I sure did but I realised what why issue is. I’m on a peak/off-peak plan so there are four daily rates, not one. That means your select won’t work for me as my table is a little more complex. I’ll post a screenshot here and see if anyone can assist with the right select.

It would be more helpful to post logs and maybe the “JS path” - my two cents

Will do. Here are the JS Paths if that will help anyway define the correct select.

Off-peak Rate (both can be found in two locations)

document.querySelector("#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(1) > td.base-rates.current > span.rate.gst_inclusive > div")

document.querySelector("#unit-balance-container > div.tou-consumer-detail.white-box > section.bin-times > section > section > div.bin-detail-bar > span.bin-detail.bin-width-14 > span.expanded > span.rate")

Peak Rate

document.querySelector("#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(2) > td.base-rates.current > span.rate.gst_inclusive > div")

document.querySelector("#unit-balance-container > div.tou-consumer-detail.white-box > section.bin-times > section > section > div.bin-detail-bar > span:nth-child(2) > span.expanded > span.rate")

Try to perform those steps and post page_soup.txt if you are still stuck.

1 Like

Cracked it! For those wanting to scrape their pricing (standard rates) if the have a peak and off-peak plan with Powershop in NZ, here is the full working code.

[EDIT] 9th November 2022. Adjusted code so that the peak and off-peak dollar values can be used in the HA Energy dashboard.

- resource: 'https://secure.powershop.co.nz/rates'
  name: Powershop
  log_response: true
  scan_interval: 43200 #every 12hrs
  form_submit:
    submit_once: true
    resource: 'https://secure.powershop.co.nz'
    select: ".content > form"
    input:
      email: !secret powershop_user
      password: !secret powershop_pass
  sensor:
    - unique_id: powershop_offpeak
      name: Powershop Off Peak
      select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(1) > td.base-rates.current > span.rate.gst_inclusive"
      unit_of_measurement: "NZD/kWh"
      value_template: '{{ (value | int) / 100 | float | round(3)}}'
      device_class: monetary     
    - unique_id: powershop_peak
      name: Powershop Peak
      select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(2) > td.base-rates.current > span.rate.gst_inclusive"      
      unit_of_measurement: "NZD/kWh"
      value_template: '{{ (value | int) / 100 | float | round(3)}}'
      device_class: monetary   
    - unique_id: powershop_daily_charge
      name: Powershop Daily Charge
      select: "#main_container > div > div.row > div.rates-table-container.base-rates-table > table > tbody > tr:nth-child(4) > td.base-rates.current > span.rate.gst_inclusive"      
      unit_of_measurement: "NZD/kWh"
      value_template: '{{ (value | int) / 100 | float | round(3)}}'
      device_class: monetary      
      on_error:
        log: warning
        value: last

1 Like