Scraping Alectra Utilities Power Website

Hello,

In Ontario, Canada we have three phases in the Time-of-Use Pricing. I would like to use the scraping platform to extract the current peak and the price. I have tried a few things, but I can’t get a satisfying result.

The easy option would be to select .updateprice and regex the price. but that seems to output a constant “mid-peak” even when it should be gving me the price!

.currentpeak scrapes: “Feb 23, 2021 – Apr 30, 2021 Current Rate: midpeak midpeak” which should yield the peak type and the price.

My code is here:

  - platform: scrape
    resource: https://alectrautilities.com/time-use-pricing
    name: Alectra Current Price
    select: '.time' 
    scan_interval: 3600

I am very new to scraping so I am sure I am doing something wrong. I have tried various combinations but nothing seems to work. I have read through Beautiful Soup Documentation — Beautiful Soup 4.9.0 documentation (crummy.com), but its difficult to know how those commands relate to what Im doing here.

Any help is mighty appreciated!
Thanks.

So I took a look and it seems Alectra is using javascript to set the values there after the page loads. If you view source and look at .updateprice you’ll see what I mean.

I found the Ontario Energy Board seems to set the rates for Alectra as well as most other utilities in Ontario. They were a bit easier to scrape.

Try the following:

- platform: scrape
    resource: https://www.oeb.ca/rates-and-your-bill/electricity-rates
    name: OEB Current Price
    select: '.off-peakactive .value,.mid-peakactive .value,.on-peakactive .value'
    value_template: '{{ value.split(" ")[0] }}'
    unit_of_measurement: "¢/kWh"
    scan_interval: 3600
1 Like

Ha!
Im happy to see someone reply to this, I ended up solving this a slightly different way and forgot to post my solution.
I also ended up finding the Ontario Energy Board and ended up scraping it also, but a different way.

I wanted three sensors that would tell me what the offpeak, onpeak and midpeak prices are at all times so I can use that information for my utility_meter. With the new Energy dashboard, I think I also need the three seperate price sensors, although Im not sure when it calculates the daily total.

Regardless, maybe you can take a look at my code and let me know if it makes sense to you.

In order to have all the prices always available to me, I scraped the prices from the legend on the alectra website, which is still visible, even without using javascript:

  #This finds the legend to the left of the clock and reports all three prices at once.
  - platform: scrape
    resource: https://alectrautilities.com/time-use-pricing
    name: Alectra Scrape Pricing
    select: ".peak-legend"
    scan_interval: 3600     # Request every hour

This gives me the following value:
$0.082 per kWh - Off-Peak $0.113 per kWh - Mid-Peak $0.170 per kWh - On-Peak

I then map each of these values to a seperate sensor. I was thinking of using one sensor with 3 attributes, but Im not sure the Energy dashboard supports attributes for pricing. Someone correct me if Im wrong.

- platform: template
     energy_price_offpeak:
        friendly_name: "Off-peak Price"
        value_template: "{{(states('sensor.alectra_scrape_pricing'))|regex_findall_index(find='[0]....', index=0, ignorecase=False)}}"
        icon_template: mdi:speedometer-slow
        unit_of_measurement: "$"

      energy_price_midpeak:
        friendly_name: "Mid-peak Price"
        value_template: "{{(states('sensor.alectra_scrape_pricing'))|regex_findall_index(find='[0]....', index=1, ignorecase=False)}}"
        icon_template: mdi:speedometer-medium
        unit_of_measurement: '$'

      energy_price_onpeak:
        friendly_name: "On-peak Price"
        value_template: "{{states('sensor.alectra_scrape_pricing')|regex_findall_index(find='[0]....', index=2, ignorecase=False)}}"
        icon_template: mdi:speedometer
        unit_of_measurement: '$' 

Now I want to know what the current peak is - this changes depending on the day of the week or for holidays. This could be hard coded into my utility meter, but it can also change on a whim, so I prefer having it scraped from the Ontario website:

  # Scrapes current peak, used to find the current peak, incase of holidays or seasonal changes.
  - platform: scrape
    resource: https://www.oeb.ca/rates-and-your-bill/electricity-rates
    name: Alectra Scrape Current Peak
    select: '.off-peakactive, .mid-peakactive, .on-peakactive'
    value_template: '{{ value| regex_findall_index(find="Mid-peak|On-peak|Off-peak", index=0, ignorecase=True)}}'
    scan_interval: 3600

This will return the currect peak name, as you also found out! So the value will be either Off-peak, On-peak or Mid-peak.

Once I have my three pricing sensors, and what the current peak is, I collect them all into one sensor:

      # Looks at scraped string and finds the three values that are "0.###" and picks the right one for the current peak. This block is under platform: template
      energy_price_current: 
        friendly_name: "Current peak Price"
        value_template: >-
          {% if is_state('sensor.alectra_scrape_current_peak', 'Off-peak') %}
            {{(states('sensor.energy_price_offpeak'))}}
          {% elif is_state('sensor.alectra_scrape_current_peak', 'Mid-peak') %}
            {{(states('sensor.energy_price_midpeak'))}}
          {% elif is_state('sensor.alectra_scrape_current_peak', 'On-peak') %}
            {{(states('sensor.energy_price_onpeak'))}}
          {% else %}
            0
          {% endif %}
        icon_template: >-
          {% if is_state('sensor.alectra_scrape_current_peak', 'Off-peak') %}
            mdi:speedometer-slow
          {% elif is_state('sensor.alectra_scrape_current_peak', 'Mid-peak') %}
            mdi:speedometer-medium
          {% elif is_state('sensor.alectra_scrape_current_peak', 'On-peak') %}
            mdi:speedometer
          {% else %}
            mdi:exclamation-thick
          {% endif %}
        unit_of_measurement: '$'

I think this last sensor is where I could make use of your proposed sensor - I didnt know I could extract the pricing like you did. Do you think there’s anything else I can do to reduce the amount of scraping I’m doing?

Here is the final result:
image
Thanks for the reply @kdubb !

If anyone else is looking at this, feel free to message me if you need help.

1 Like

This looks great but with the new energy management in HA, I would love to be able to pull my daily consumption from my Alectra’s account and feed it to HA for energy management

Any luck with pulling daily consumption?

I’m a brand new HA user… 1 day old to be precise but I’m LOVING the consolidation of all the smart apps into one. I was just browsing around looking what else can do and stumbled across energy monitoring - yesssss please!

I’m going to try to figure out how to do what Max has done above there, but daily would be great as well to narrow down any strange peaks.

Unfortunately not, it will require some work to sign in into the account and then navigate to the consumption page which uses graphs

Hi Max,

Could you point me in the direction of a good tutorial on how to do what you’ve done here for scraping data from the Ontario Energy Board?

Downloading daily and hourly use data can be done in around 50 lines of code.
Always wanted to build a library for it but unfortunately have no time to do it this year.
Here’s a simple and limited (only logins by account number and no 2FA are supported) proof of concept: alectra.py · GitHub

1 Like

This is fantastic! Im afraid I am not versed enough to figure out how to turn this into something I can import this into Home Assistant.

So I was able to very easily log in to alectra using this fantastic HACS integration:

https://github.com/danieldotnl/ha-multiscrape

With this code, I could log in, but not read anything interesting.

multiscrape:
  - resource: https://myaccount.alectrautilities.com/app/capricorn?para=selectAccount&userAction=search
    scan_interval: 3600
    form_submit:
      resubmit_on_error: true
      resource: 'https://myaccount.alectrautilities.com/app/login.jsp'
      select: "#accountLogin"
      input:
        accessCode: !secret alectra_user
        password2: !secret alectra_pass
        extra: field
    sensor:
      - unique_id: alectra_login_scrape
        name: Alectra Login Scrape Test
        select: "body > div.container-fluid.powerstream-page-container > div.container.powerstream-page-title"

The trouble is that most of the alectra page is built upon Javascript and neither scraper can extract any data from it. We’ll have to wait for @FelixKa :wink:

I built a custom component a while ago that extracts the going rates from the OEB opendata documents, and keeps track of the current peak.

This component was added to HACS and can be found at https://github.com/jrfernandes/ontario_energy_board, I hope it’s helpful to others.

1 Like

This is fantastic, thank you!

Now we just need a component to read our meter data from alectra :slight_smile:

2 Likes

Very interested in importing usage data from Alectra into Home Assistant!

Is this helpful? Someone made a python program to pull usage data from their HydroOne account and push that into a database. Can we modify this so that we can pull in usage data from Alectra? I have seen that same green button in my account.

1 Like

I looked into this a bit over the weekend and there isn’t really a way to do this cleanly without directly messing with the database from outside HA. That’d be something I’d try to avoid if possible.

1 Like

@maxshcherbina I managed to find a specific url that would allow to scrape daily consumption on a daily basis, but I somehow can’t make it past the login, I’ve tried your code but I only get to the “session expired” page, does that code still work for you?

Here is the direct link to daily consumption for a specific day (the day/month/year) elements would have to be dynamically updated in the url

https://myaccount.alectrautilities.com/app/capricorn?para=smartMeterConsumInqV2&downloadConsumption=&userAction=&dailyOrHourlyBreakdown=hourly&inquiryType=hydro&targetHourlyDate=2022-10-16&fromDate=2022-10-16&toDate=2022-10-16&print=N&fromDatePicker=10%2F16%2F2022&toDatePicker=10%2F16%2F2022&selectedRPP=TOU

with this selector for example, you should be able to get the off-peak day usage:

div.col-sm-7.col-md-8 > div.panel:nth-child(2) table > tbody > tr:nth-child(1) > td:nth-child(2)

Anyone been able to successfully get their daily use into HA? Maybe its its even downloading the usage now to google sheets?

The google sheets integration with Home Assistant doesnt help us in this case, as far as I can see - since that Google Sheets integration is mainly a way of appending new data, not reading old data.

Just would love to be able to get my data from Alectra into home assistant. Without buying something and getting it installed not sure of any way to get our power data in :frowning:

Has anyone had any luck with Alectra and the Green Button data export?