BCHydro component - where did it go?

Share your Projects! Custom Integrations
43

Progress update: @frenck pushed me in a good direction for managing API calls. After a major refactor and splitting the BCHydro API out to its own repo, this integration can now accept username and password via config flow, directly in the HASS UI!

Testing the branch over at https://github.com/emcniece/hass-bchydro/pull/6. The standalone BCHydro Python module can be found at https://github.com/emcniece/bchydro.

This refactor was challenging but now it just needs a bit of polish. It can be installed by hand right now, but I’m aiming for merging it into the official integration repo and the community store if possible.

I’d like to hear from the folks here: What would you like to do with this data? BCHydro makes a few fields public (can be seen on the hass-bchydro readme) but is there a particular automation or trigger that you want? We might be able to add some data processing to fit your use case :slight_smile:

1 Like
44

Personally, what I really want to do is track trends of hourly (or more? … probably not) usage against other sensors/states in HA.
ie: When the TV is on, the power consumption went up by X, so I can figure out the TV uses about X per hour… That sort of thing.
Maybe eventually even using the data to detect what else is going on in the house (dryer is running, AC is on, etc)

46

Hi @emcniece,

First of all, thanks for this awesome work!

I installed this component manually but the only sensor working is the sensor.latest_usage. All others generate this kind of error:


2020-10-21 17:08:44 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.cost_to_date fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 278, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 471, in async_device_update
    await self.hass.async_add_executor_job(self.update)  # type: ignore
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/bchydro/sensor.py", line 125, in update
    self._api.login()
  File "/config/custom_components/bchydro/sensor.py", line 155, in login
    request = self.call_api(
  File "/config/custom_components/bchydro/sensor.py", line 151, in call_api
    response.raise_for_status()
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://app.bchydro.com/sso/UI/Login

I don’t understand the Unauthorized error if one of the sensors work. Is that related to cookies?

Thanks and best regards,
José

47

hi again @emcniece,

The error I’ve reported was in the master branch. Now, I’ve followed instructions for the testing branch and it works.

Thanks again.

49

Hi,

Awesome to find this. I installed it and all sensors are showing unknown. How long does it take to get a reading or how often is it updated? Is there a way to trigger it?
I believe I am using the master branch if that matters.

Cheers!

50

Hi @bj24,

TL;DR : I was happy too soon

Long Version:

Mine came out unknown too but was credentials error. However, it updates the first time and later it print some HTML to the logs and bailout with error, so after two days I realized it was not being update, just throwing HTML stuff in the logs. If I reconfigure the integration it reads again the current values and never update again. So I removed it, will try in a dev box and see if I can help fix it

51

Thank you for the follow up, I will keep an eye on this. Cheers!

52

Hi

I’d love to get this working as well.

I tried and got the “401 Client Error: Unauthorized for url” error as well. Removed it and will wait.

Garry

53

Very excited to see some work on this! Wish I had the knowledge to help out.

54

I think the “401 Client Error” might be a rate limiting issue. The component sets up 4(5?) sensors and even though all the data needed is returned within a single request it still tries to log-in and fetch the data individually for every sensor. There might be rate limit setup on BC Hydro for the login request and the component might be making all the requests too quickly.

Looks like there is a major refactor going on in hass-bchydro and this might be fixed soon, but in the meanwhile I’ve prepared a temporary fix: https://github.com/arshsingh/hass-bchydro

It caches the data for 60 seconds (so other sensors don’t need to repeat the request). It’s not elegant but seems to be working for now.

1 Like
55

It works! Thank you!

56

Thanks for the fix @arsh! You’re right about the refactor. I’ll make sure to include rate limiting in the next release.

I was able to put some time into this over the holidays and the API + integration are close to being ready. Nebual was able to help get past an auth roadblock, now I’m polishing the integration. Just last night I finally got the config flow entry form working with visible labels, that was harder than expected.

The API itself needs some improved error and re-authentication handling, then we should be good for a full release.

I’ve been somewhat discouraged since learning that Home Assistant no longer accepts core integrations that use web scraping to obtain their data. I think that includes authentication, which is what this integration does since there is no real BCHydro API. There’s still hope for this plugin as a standalone integration (not sure about HACS) of course, but I agree with HA that we’re left at the whim of BCHydro… if they were to implement a login captcha, this integration would be toast.

I’ve been trying to reach BCHydro’s web/dev team. A support ticket was escalated and I learned that BCHydro did have a beta public API at one point, it’s closed now but it sounded like they might consider special cases. I have a phone number for their head office that I was going to try after the holidays, who knows, maybe they could provide a real auth endpoint.

Stay tuned for more…

57

Last night I merged an update to the bchydro API package that implemented caching, request retries, and rate limiting: https://github.com/emcniece/bchydro/pull/3

There’s a corresponding version bump over in the integration refactor branch: https://github.com/emcniece/hass-bchydro/pull/6

Most importantly, this has solved the re-authentication issues that cause the integration to spew errors after an hour of having it installed. The integration has been stable and silent for 12h now!

Screen Shot 2021-01-05 at 10.55.06 AM
Screen Shot 2021-01-05 at 10.55.06 AM1924×460 108 KB

We’re close to a full release. I’d like to go over the updatecoordinator code again and make sure that it is implemented well, and see if there is anything we can do to improve the sensors.

There’s a caveat with the hourly usage stats. I hoped that on first discovering the hourly data export we could get near-real-time usage, but this doesn’t seem to be the case; the export only shows data for the previous day, not for the current day. This means that as we use the export as the “latest” data, the sensor readings are actually showing yesterday.

There might be a way to plot data in the past, and if so we could get a more accurate representation in the sensors. If anyone knows how to do this, please speak up!

3 Likes
58

Hm, https://www.bchydro.com/powersmart/energy-management-trials/hydrohome-smart-home-trial.html describes an app that provides energy usage in real-time 30-second increments… neat.

59

I signed up for that and it requires you install their proprietary hub for most functionality and also has limited supported hardware (thermostats, light switches etc). In addition the reporting interval is the same as you already get on their website. In my case there is sometimes a delay of three or four days before their site updates and the same using the phone app.

60

Just wanted to thank you for your work on this. Looking forward to the release!

2 Likes
61

Hi @emcniece,

Thanks for putting together this addon.

I copied the master branch and managed to get it to login but something must have changed as it’s not working after the initial data retrieve. For example, when I restart the HA core, the add-on logs into BCHydro, retrieves the data, but then doesn’t update after that. I turned on debugging and after that initial data retrieval, it doesn’t seem to be able to re-authenticate. I’m not sure why - maybe the sessions is still good and the add-on isn’t detecting it?

There are no errors being displayed. I may add in some additional debugging outputs to see what might be going on but thought I’d see if you had any insight.

Thanks!

2021-05-19 16:13:52 DEBUG (SyncWorker_2) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/sso/UI/Login with payload={'realm': 'bch-ps', 'email': '<username>', 'password': '<password>', 'gotoUrl': 'https://app.bchydro.com:443/BCHCustomerPortal/web/login.html'}
2021-05-19 16:13:53 DEBUG (SyncWorker_2) [custom_components.bchydro.sensor] Redirect iterations: 2
2021-05-19 16:13:53 DEBUG (SyncWorker_2) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/evportlet/web/global-data.html with payload=None
2021-05-19 16:13:54 DEBUG (SyncWorker_2) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/evportlet/web/account-profile-data.html with payload=None
2021-05-19 16:13:54 DEBUG (SyncWorker_3) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/sso/UI/Login with payload={'realm': 'bch-ps', 'email': '<username>', 'password': '<password>', 'gotoUrl': 'https://app.bchydro.com:443/BCHCustomerPortal/web/login.html'}
2021-05-19 16:13:54 DEBUG (SyncWorker_0) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/sso/UI/Login with payload={'realm': 'bch-ps', 'email': '<username>', 'password': '<password>', 'gotoUrl': 'https://app.bchydro.com:443/BCHCustomerPortal/web/login.html'}
2021-05-19 16:13:54 DEBUG (SyncWorker_8) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/sso/UI/Login with payload={'realm': 'bch-ps', 'email': '<username>', 'password': '<password>', 'gotoUrl': 'https://app.bchydro.com:443/BCHCustomerPortal/web/login.html'}
2021-05-19 16:18:52 DEBUG (SyncWorker_4) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/sso/UI/Login with payload={'realm': 'bch-ps', 'email': '<username>', 'password': '<password>', 'gotoUrl': 'https://app.bchydro.com:443/BCHCustomerPortal/web/login.html'}
2021-05-19 16:18:52 DEBUG (SyncWorker_6) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/sso/UI/Login with payload={'realm': 'bch-ps', 'email': '<username>', 'password': '<password>', 'gotoUrl': 'https://app.bchydro.com:443/BCHCustomerPortal/web/login.html'}
2021-05-19 16:18:52 DEBUG (SyncWorker_8) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/sso/UI/Login with payload={'realm': 'bch-ps', 'email': '<username>', 'password': '<password>', 'gotoUrl': 'https://app.bchydro.com:443/BCHCustomerPortal/web/login.html'}
2021-05-19 16:18:52 DEBUG (SyncWorker_3) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/sso/UI/Login with payload={'realm': 'bch-ps', 'email': '<username>', 'password': '<password>', 'gotoUrl': 'https://app.bchydro.com:443/BCHCustomerPortal/web/login.html'}
2021-05-19 16:23:52 DEBUG (SyncWorker_7) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/sso/UI/Login with payload={'realm': 'bch-ps', 'email': '<username>', 'password': '<password>', 'gotoUrl': 'https://app.bchydro.com:443/BCHCustomerPortal/web/login.html'}
2021-05-19 16:23:52 DEBUG (SyncWorker_8) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/sso/UI/Login with payload={'realm': 'bch-ps', 'email': '<username>', 'password': '<password>', 'gotoUrl': 'https://app.bchydro.com:443/BCHCustomerPortal/web/login.html'}
2021-05-19 16:23:52 DEBUG (SyncWorker_4) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/sso/UI/Login with payload={'realm': 'bch-ps', 'email': '<username>', 'password': '<password>', 'gotoUrl': 'https://app.bchydro.com:443/BCHCustomerPortal/web/login.html'}
2021-05-19 16:23:52 DEBUG (SyncWorker_6) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/sso/UI/Login with payload={'realm': 'bch-ps', 'email': '<username>', 'password': '<password>', 'gotoUrl': 'https://app.bchydro.com:443/BCHCustomerPortal/web/login.html'}
2021-05-19 16:28:52 DEBUG (SyncWorker_0) [custom_components.bchydro.sensor] About to call https://app.bchydro.com/sso/UI/Login with payload={'realm': 'bch-ps', 'email': '<username>', 'password': '<password>', 'gotoUrl': 'https://app.bchydro.com:443/BCHCustomerPortal/web/login.html'}
62

Thanks @VendettaMike, looks like you’re crossing the event horizon and time is starting fold in on itself.

Jokes aside, I’ve really lost steam on this. The implementation just… sucks. We can’t get live data! The best data we can get is for the previous day. We can get hourly usage, but only for yesterday… and we can’t plot historical values in HA. So if we wanted hourly consumption, we would have to read the hours for yesterday and pretend that it’s happening today.

I’ve made several attempts at getting in touch with the platform teams at BCHydro and I get ghosted each time. I got some team lead phone numbers through corporate, left messages (no answers) and I’ve tried cold-messaging some people on LinkedIn that look like they might be close to the right teams. Not a single response.

Mike, your report looks like there’s 2 problems: first that auth is redirecting without succeeding (so the plugin needs inspection & better handling) and second that there are way too many sync workers firing authentication requests so close together. It’s been a couple months since I looked at the HA docs but I bet something has changed in the plugin setup flows.

I’ll see if I can muster up some courage and poke around in the next 2 weeks. My homelab is currently hosed as I’m trying to migrate from Rancher 1.6 to 2.x, Cattle to Kubernetes, not a small task. Sometimes I’d rather just go make furniture, you know? At least screwdrivers don’t break compatibility every month.

63

Eric,

Yeah, I know the feeling. I’m dealing with a similar situation at work where an API would makes things so much easier but instead we need to go through web scraping for our own data. Any little change and we’re back to the drawing board. It’s just a matter of time before it breaks again and I’m worried they’ll change it to the point where it’s near impossible to use this method.

I know a few people at Hydro but nobody in the platform group but I’ll see if I can get any inside info - I’m not holding out much hope though. I’m also not surprised with the lack of responses as they typically don’t deal with “outsiders”.

As for usage, unfortunately the data on the website is only updated daily. I do know the meters communicate throughout the day but we’re stuck with what’s available on the site. Rainforest Automation makes Zigbee devices which communicate directly to the meter - the only way to get real-time information. The downside is that the data is encrypted and hydro needs to authorize any MAC’s that communicate directly with their meters so you need to use a Rainforest device - nothing else will be allowed. I’m sure hydro gets a small cut of the action. You can’t monetize data if you’re giving it away for free. Likely why they pulled the public API.

So it looks like we’ll only be able to report yesterday’s usage in HA. Not ideal. Or, create a method to read physically form the meter itself. Not an easy path but doable. Another options is using current clamps on the main and feeding it back with an ESP8266 - may projects out there to make that work.

Regardless, it would be great to get better data from Hydro, but I’ll take what I can get for now. If I get a chance to dig in, I’ll see what I find. I’m also working on my homelab and doing some major changes. Never a dull moment!

1 Like
64

Honestly, I still think this has value with usage from yesterday. Take a break and come back to it, people like myself appreciate your efforts.

3 Likes