Energy consumption from provider - Genesis Energy NZ

Any smart cookies worked out how to pull energy information from NZ’s Genesis Energy (even better if you also have gas).

A while ago I did have a script that would read the power reading by hour from 48 hours ago and push it to influx. But genesis changed their api to oath and I couldn’t get it to authenticate.

If someone has something, or would want to work on something, I’ not sure if I had the code fully public, but can do so.

Was originally discussed here: Scraping Electricity usage (geekzone.co.nz)

Did you ever get a solution for this? I’m searching for the same thing.

Nope. Their authentication was too hard to reverse engineer.

And if you ask them for api access they tell your. You can manually log in and download usage. #headdesk

I put on a glow optical sensor on my power meter.

I forked the frank energy one and got it working with my genesis connection.

You can use HACS to load the custom repo

Do this work for anyone else? I just get the following error:

Logger: homeassistant.config_entries
Source: config_entries.py:749
First occurred: 13:50:47 (5 occurrences)
Last logged: 13:53:37

Error setting up entry Genesis Energy for genesisenergy
Traceback (most recent call last):
File “/usr/src/homeassistant/homeassistant/config_entries.py”, line 749, in __async_setup_with_context
result = await component.async_setup_entry(hass, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/config/custom_components/genesisenergy/init.py”, line 21, in async_setup_entry
await api.get_refresh_token()
File “/config/custom_components/genesisenergy/api.py”, line 97, in get_refresh_token
csrf_value = response.cookies.get(‘x-ms-cpim-csrf’).value
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: ‘NoneType’ object has no attribute ‘value’

I got the same error. Is it still working for you @JoelBrenstrum ? Maybe something has changed. Seems like the x-ms-cpim-csrf cookie isn’t there

I’m a developer but have extremely limited experience with Python. I’ll have a deeper dive into this when I get a chance.

I figured out the issue, it’s a breaking change in aiohttp
https://docs.aiohttp.org/en/stable/client_advanced.html#aiohttp-client-cookie-quoting-routine

jar = aiohttp.CookieJar(quote_cookie=False)
aiohttp.ClientSession(cookie_jar=jar)

@JoelBrenstrum I’m terrible with pull requests but the above should be enough info for you to fix this with more recent versions of home assistant

2 Likes

Hey I’m new around here, I hit this the other week too. Did you get HA configuring again? What was the change in the end? Happy to throw in merge request if you can point me to some more details, I’m familiar with python but very unfamiliar with web api’s.

I had a little bit of time to dig into this - it looks like ideally actually we should just get a genesis config setup for the opower integration that ships with HA - but I got this working with a temp fix…

Init a CookieJar()
Use it in the client sessions
Pull the csrf info from it

diff --git a/custom_components/genesisenergy/api.py b/custom_components/genesisenergy/api.py
index 474a031..7b51f64 100644
--- a/custom_components/genesisenergy/api.py
+++ b/custom_components/genesisenergy/api.py
@@ -36,6 +36,7 @@ class GenesisEnergyApi:
         self._refresh_token = None
         self._refresh_token_expires_in = 0
         self._access_token_expires_in = 0
+        self._cookie_jar = aiohttp.CookieJar(quote_cookie=False)
 
     def get_setting_json(self, page: str) -> Mapping[str, Any] | None:
         """Get the settings from json result."""
@@ -50,7 +51,7 @@ class GenesisEnergyApi:
         """Get the refresh token."""
         _LOGGER.debug("API get_refresh_token")
 
-        async with aiohttp.ClientSession() as session:
+        async with aiohttp.ClientSession(cookie_jar=self._cookie_jar) as session:
             url = f"{self._url_token_base}/oauth2/v2.0/authorize"
             scope = f'openid offline_access {self._client_id}'
             params = {
@@ -94,7 +95,8 @@ class GenesisEnergyApi:
             async with session.get(url, params=params) as response:
                 response_text = await response.text()
                 # Extract the new CSRF token from cookies because it changes here
-                csrf_value = response.cookies.get('x-ms-cpim-csrf').value
+                cookies = self._cookie_jar[('auth.genesisenergy.co.nz', '')]
+                csrf_value = cookies.get('x-ms-cpim-csrf').value
                 csrf = csrf_value
 
             payload = {
@@ -167,7 +169,7 @@ class GenesisEnergyApi:
             "refresh_token": self._refresh_token,
         }
 
-        async with aiohttp.ClientSession() as session:
+        async with aiohttp.ClientSession(cookie_jar=self._cookie_jar) as session:
             url = f"{self._url_token_base}/oauth2/v2.0/token"
             async with session.post(url, data=token_data) as response:
                 if response.status == 200:
@@ -208,7 +210,7 @@ class GenesisEnergyApi:
             'intervalType': "HOURLY"
         }
 
-        async with aiohttp.ClientSession() as session, \
+        async with aiohttp.ClientSession(cookie_jar=self._cookie_jar) as session, \
                 session.post(url, headers=headers, json=params) as response:
             if response.status == 200:
                 data = await response.json()
@@ -220,7 +222,7 @@ class GenesisEnergyApi:
                 message = await response.text()
                 _LOGGER.error("Could not fetch consumption, error: %s", message)
                 return None
-            
+
 
     async def get_gas_data(self):
         """Get data from the API."""
@@ -252,7 +254,7 @@ class GenesisEnergyApi:
             'intervalType': "HOURLY"
         }
 
-        async with aiohttp.ClientSession() as session, \
+        async with aiohttp.ClientSession(cookie_jar=self._cookie_jar) as session, \
                 session.get(url, headers=headers, params=params) as response:
             if response.status == 200:
                 data = await response.json()
@@ -264,4 +266,4 @@ class GenesisEnergyApi:
                 message = await response.text()
                 _LOGGER.error("Could not fetch consumption, error: %s", message)
                 return None
-            
+

Not entirely sure if it’s required to share the cookie jar with all the sessions, but will run this for a few days to make sure nothing crazy goes wrong and see if I can get it merged back into the git project.

Never contributed to a public git project before but I assume if I throw in a merge request that would be good @JoelBrenstrum ?

1 Like

What’s the recommended way to get this integration to automatically pull from genesis?
I noticed I have to restart HA (or reload the integration in the UI) to get it to trigger. I’m still finding my way around HA so might be missing something, so not sure if this is right, but have in the meantime setup an automation to reload the config entry at 3am daily which does appear to work:

Turns out the config reload doesn’t work - testing a few more options and will report back after a few days of testing.

Calling “update entity” on them after removing the time-based token validity check in the api.py file seems to have done the trick - it’s been working reliably in ways I haven’t had to touch for a while now.


Hoping to look into updating the sensor.py next, to more cleanly handle solar/grid return (currently just adds negative grid draw values).

I have taken JoelBrenstrum genesisenergy fork and made significant changes to it. Its been working on my system without any issue for the past few months.

Here is a link to my repo, its a manual install if you want to try it out
https://github.com/ddahya/ha-genesisenergy

Here are some of the features that i have added

  • Energy Dashboard Integration:

    • Creates long-term statistics for Electricity Consumption (kWh) and Gas Consumption (kWh).
    • Also creates statistics for daily Electricity Cost (NZD) and Gas Cost (NZD) for detailed tracking.
  • Electricity Forecast Sensors:

    • Integrates Genesis’s daily and weekly electricity forecast.
    • Today's Forecast Usage (kWh) and Today's Forecast Cost ($) sensors.
    • Attributes include the predicted high/low range and the full 7-day forecast data, perfect for advanced automations.
  • Usage Breakdown Sensors:

    • Shows how Genesis categorizes your electricity use from your last completed billing period.
    • Creates sensors for Appliances, Electronics, Lighting, and Other usage (in kWh).
  • EV Plan Sensors:

    • If you’re on an EV plan, it automatically creates sensors for your daily Day (Peak) Usage/Cost and Night (Off-Peak) Usage/Cost.
    • A “Savings” sensor shows how much you saved on your last full day of usage compared to the standard rate.
  • Power Shout Sensors:

    • Sensors for Power Shout Eligibility and current Balance (in hours).
    • Attributes include details on upcoming bookings, active offers, and expiring hours.
  • Billing Cycle Sensors:

    • Provides sensors for costs within your current billing cycle: Electricity Used ($), Gas Used ($), Total Used ($), Estimated Total Bill ($), and Estimated Future Use ($).
  • Detailed Account Sensor:

    • A single sensor.genesis_energy_account_details entity with a wealth of information in its attributes, including your billing plans, account IDs, and raw data from various dashboard widgets.
  • HA Services:

    • genesisenergy.add_powershout_booking: Book your Power Shouts from automations or scripts.
    • genesisenergy.backfill_statistics: A powerful tool to import historical usage data into Home Assistant.
    • genesisenergy.force_update: Trigger an immediate data refresh.
1 Like

very nice @dd77 the integration looks really good.

I can’t seem to get the backfill of data to work though. I get an unknown error. Did you want me to post in your github or here some logs? Snippet below in case it is useful.

Thanks again for the effort on this.

  File "/config/custom_components/genesisenergy/__init__.py", line 152, in async_backfill_statistics_service
    has_electricity, has_gas = get_available_services(coordinator)
                               ^^^^^^^^^^^^^^^^^^^^^^
NameError: name 'get_available_services' is not defined

This is great! Thanks very much.
I forked it so that I could add a version tag so that it can be installed with HACS, could be worth doing that to your repo so others can use that.

I also couldn’t get the backfill to work - unknown error.

Should be fixed now, just download the init.py from the repo, reboot and try the backfill service.

I tried the update which now doesn’t fail with error, but seems to complete immediately with no data pulled. Should I add some debug to give more info?

I’ve tested the backfill service on my test machine, and it’s working as expected. I was able to import the past 100 days of Gas and Electricity data into blank genesisenergy:electricity_consumption_daily and genesisenergy:gas_consumption_daily sensors.

Please check whether your genesisenergy:electricity_consumption_daily and/or genesisenergy:gas_consumption_daily sensors are completely empty. If they already contain any data, the backfill service cannot be used to “fix” or re-import data from before the earliest recorded date. Backfill can only add missing data starting from the point where your current data ends.

Thanks for helping troubleshooting. I have checked the entities (via the Energy Dashboard) and can’t see any data in either of them.

I will delete all data and try again, maybe I have data in there somewhere. I will remove the integration and set it up again.

OK after scrapping with the database and statistics a bit, I managed to clear it all out and get 100 days in. Just don’t have any data from yesterday for electricity, but do for Gas. Not sure why that is yet.

Nice work! Thank-you heaps. sorry I must have had some data in there somewhere.