First-Utility.com (UK) Smart Meter Data

Third party integrations
1

I have just had a smart meter installed by my utility company, First Utility. The monitor (IHD) that they provide is a very simple affair and evidently runs on the Zigbee protocol. To my knowledge, however, it is not possible to read the data going to this device.

Instead, I have written a little python script that logs on to their website and downloads that day’s data in csv format.

#!/usr/bin/env python

import requests, datetime

now = datetime.datetime.now()
date = now.strftime("%Y-%m-%d")

s = requests.session()

account = '12345678'
uid = 'username'
pwd = 'password'

form = {'LogonForm[uid]':uid,'LogonForm[pwd]':pwd,'LogonForm[remember]':'0'}
response = s.post('https://www.first-utility.com/login', data=form)

ele = s.get('https://www.first-utility.com/myaccount/down/usage?accountId=' + account + '&interval=halfhourly&type=electricity&enddate=' + date + '&startdate=' + date)

file = open("/home/homeassistant/.homeassistant/data/ele.csv", "w")
file.write(ele.content)
file.close()

gas = s.get('https://www.first-utility.com/myaccount/down/usage?accountId=' + account + '&interval=halfhourly&type=gas&&enddate=' + date + '&startdate=' + date)
file = open("/home/homeassistant/.homeassistant/data/gas.csv", "w")
file.write(gas.content)
file.close()

This creates two csv files - one for gas and one for electricity. Obviously, the code can be tweaked for different dates. I’m not even sure that it’s possible to download the current day’s data. I think it is updated overnight with the previous day.

Now. My question is what to do with this data. What I would like is a history graph in the lovelace interface. I have no idea how I would go about doing that. Anyone have any ideas?

2

I see there is a restful API I can use to post the data from the CSV into HA. For example…

# Take the value and using the Home Assistant RESTful API use curl to POST to information where you can then add it to any way you would a normal sensor
curl -s -X POST -H "Content-Type: application/json" -H "x-ha-access: $hass_pass" -d '{"state":"'$value'"}' "$hass_url/api/states/sensor.gas_usage_sensor" > /dev/null

My python knowledge is very limited. Does anyone know the code I could use to take data in this format

"Please note that the following time data is based in the UTC format"
"day","calculated usage (kwh)","type"
"2019-02-08 00:30","25.3","ACTUAL"
"2019-02-08 01:00","13.1","ACTUAL"
"2019-02-08 01:30","","NODATA"
"2019-02-08 02:00","","NODATA"
...
...
...
lots of data
...
...
...
"2019-02-11 00:00","","NODATA"
"","","",""
"units","maxUsage","minUsage"
"kWh","","0.00"

And loop around it in python so there is an API call for each line of “ACTUAL” data? All other lines can be ignored.

3

Maybe here:

Related I just write this:

4

I have managed to get this working by limiting the data to daily, rather than half-hourly.
I now have a python script that returns a single value. It takes yesterday’s date and queries the energy provider for that energy use value. I have created a command line sensor that calls this script.

The only caveat is that the data is yesterday’s data, but Home Assistant records it in the DB as today’s data (since that’s when the script is run).

5

@bigstoo Please could you share your updated script?

6

Hi. Here’s my updated script.

#!/usr/bin/env python

import requests, sys
from datetime import date, timedelta

yesterday = date.today() - timedelta(days=1)
date = yesterday.strftime("%Y-%m-%d")

s = requests.session()                                                       

account = '[account number]'
uid = '[username]'
pwd = '[password]'

form = {'LogonForm[uid]':uid,'LogonForm[pwd]':pwd,'LogonForm[remember]':'0'}

response = s.post('https://www.shellenergy.co.uk/login', data=form)

data = s.get('[https://www.shellenergy.co.uk/myaccount/down/usage?accountId=' + account + '&interval=daily&type=' + sys.argv[1] + '&enddate=' + date + '&startdate=' + date
kwh = (str(data.content)[str(data.content).find("type")+21:str(data.content).find("ACTUAL")3])

if sys.argv[1] == 'electricity':
  cost = ( float(kwh) * [put cost per kWh here] ) + [put daily standing charge here]
elif sys.argv[1] == 'gas':
  cost = ( float(kwh) * [put cost per unit here] ) + [put daily gas standing charge here]
else:
  cost = 0

sys.stdout.write(str(cost))
sys.exit(0)

You then create a command_line sensor in Home Assistant that calls the command line
python3 /path-to-script/script.py electricity
or
python3 /path-to-script/script.py gas

7

@bigstoo Legend, I’ve been looking for something to query my Shell Energy usage. This is ideal :slight_smile:

8

@bigstoo Have you tried running this in AppDaemon?

9

I haven’t. In fact I had to Google AppDaemon to see what it was about.

I may well have a look, as it seems I could integrate it more tightly with Home Assistant.

10

Well it looks like first-utility.com (now Shell Energy) have updated their website and the above script no longer works.

The login form parameters have been changed. However, there now appears to be a javascript running to check that the input is entered by a user (antiClickJack).

Unfortunately, with my limited knowledge of Python, I’m unable to get it to work. If anyone has any suggestions I’d be very grateful.

11

Has anyone revisited this recently or is it still the case that it has Javascript validation or equivalent?

12

This is as close as I have got. I have tried to copy browser activity closely but just can’t get this request to response with a 200. There must be some header combination that will make this work but I can’t figure it out.

const authRootUrl = "https://accounts.shellenergy.co.uk";
const dataParams = new URLSearchParams({
  formType: "curity",
  userName,
  userPassword,
}).toString();

const res = await client.post(`${authRootUrl}/authn/authenticate/one-login`, dataParams, {
  headers: {
    "accept": "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
    'content-type': 'application/x-www-form-urlencoded'
  }
});