Multiscrape get list of attributes

Configuration
1

Hi all. I’d like to scrape my power usage from my energy supplier’s site, but I’m running into a bit of an issue.
They display the half hourly power usage in a heatmap that’s basically an html table with blank cell values. But the values I need are the attributes of those cells.

image
image954×659 25.8 KB

I’ve tried multiple ways to get these values, but I either only get the first values or nothing at all.

  sensor:
    - unique_id: powershop_heatmap_time
      name: "Powershop Heatmap Time"
      select_list: "table#heatmap tr:nth-child(1) > td:nth-child(n+3):nth-child(-n+50)"
      attribute: data-time
    - unique_id: powershop_heatmap_usage
      name: "Powershop Heatmap Usage"
      select_list: "table#heatmap tr:nth-child(1) > td:nth-child(n+3):nth-child(-n+50)"
      attribute: data-usage

  sensor:
    - unique_id: powershop_heatmap_time
      name: "Powershop Heatmap Time"
      select_list: "table#heatmap tr:nth-child(1) > td"
      attribute: data-time
    - unique_id: powershop_heatmap_usage
      name: "Powershop Heatmap Usage"
      select_list: "table#heatmap tr:nth-child(1) > td"
      attribute: data-usage

sensor:
  - unique_id: powershop_heatmap_time
    name: "Powershop Heatmap Time"
    select_list: "table#heatmap tr:nth-child(1)  td"
    value_template: "{{ value | regex_findall_index('data-time=\"([^\"]+)\"') }}"
  - unique_id: powershop_heatmap_usage
    name: "Powershop Heatmap Usage"
    select_list: "table#heatmap tr:nth-child(1)  td"
    value_template: "{{ value | regex_findall_index('data-usage=\"([^\"]+)\"') }}"

I assumed I could just extract the raw html using multiscraper, but it doesn’t seem to work either.

Any help would be appreciated.

2

Are you certain they don’t also publish this data in something easier to read like XML or JSON?

3

Unfortunately they don’t. They used to have a public API, but they shut it down.

4

please share an html code of your current energy supplier data so we can try to scrape it on our side.

5

…and make sure it’s the source as initially received (View Source), excluding any dynamic updates.

6

Powershop has a “download CSV of usage”, which is a lot easier to deal with than trying to scrape and parse the “heatmap” (Honestly, that heatmap is cool to look at, but other than that doesn’t tell me anything useful).
To overcome the problem, I’ve hacked my way around it by having a Python script download the data directly instead of using a scraper.
You can read about what I did here: ⚡ Getting to know my power usage » Firesphere.dev
And, also, of course, directly my code, here: Firesphere/PowershopScraper - Codeberg.org