Integrating Anglia water usage for smart meters (UK)

Thanks @Zeunas. I’ve updated the script and restarted. It seems to be working - the input_boolean turns off and can see today’s current usage correctly reflected in sensor.water_usage_x .

image

The only way I have managed to get this into the HA Energy dashboard is to edit the customize.yaml file adding a state_class

sensor.water_usage:
    device_class: water
    unit_of_measurement: L
    state_class: total_increasing

In the energy dashboard configuration water usage now appears.

However, the data does not flow through to the dashboard…

Your help is greatly appreciated.

From what I found the template sensor name you create has to be the same as the name you set in the python script.

Otherwise a new sensor is created by appdaemon which is not connected to the water meter.



You shouldn’t require to customise the state_class of the utility meter, if you have set it up as such. You need to give the utility meter some time to cycle (it can take a couple of hours), it will eventually pull the information from the sensor.water_usage_x. If you really want to force test it, you can always use the below service and input the value you want, however bear in mind if in one day you use the force pull options more than once (input_boolean switch and calling the below service) it will eventually skew your usage trend, as your utility meter will stack up the values your force into it. You can always amend it under Statistics tab on the Developer tools, which will fix your Energy Dashboard but won’t fix it at Frontend.

What @rodgers86 is saying above is correct.

Tried both now.

Option 1
I’ve previously only used option1. Which eventually worked, but the documentation is terrible. Took ages to do something which should be simple.

Option 2. Much easier to set up as you can pretty much write what you need in SQL query format however it puts the time stamp as the state?!

It does put the correct data in the attributes though so I’ve amended the pyscript so that the state is the Value.

As you can see in picture my usage was
2023L until I amended the script.

Yes! I remember that now…not the greatest but it was the easiest to use. Well done finding a solution thou! My work around back then was to create another sensor to read from the attributes but yours is so much cleaner!

Ok @Zeunas - so I unpicked this from the customize.yaml file and the energy dashboard now has this as an error;

image

I guess I just don’t understand how this stuff should hang together but what is more confusing and probably unrelated is that the script keeps returning a usage value of 0. Checking on the Anglian website the figure should be 138L for yesterday. I’ve even tried restarting AppDaemon. Here is the log;

Any thoughts?

That’s strange, it should set the state_class automatically by setting the device class as water, in that case add it back to the customize.yaml state_class: total_increasing

I can see that you’re script is running at 8.38, it should be running after 11am, make sure that the first part of the script under “Starting APP” has the ‘datetime.time’ set to 11, as per below:

# Starting APP #

  def initialize(self):
    self.log("Starting Water Usage App")
    runtime  = datetime.time(11, 00, 0)
    self.run_daily(self.StartProcess, runtime)
    self.listen_state(self.StartProcessForced,reading, state="on")

Script is fine. It ran at 8:36 because I forced it via the input boolean to show you the log entries… :grinning:

Let’s wait for 11am or push the switch again? And let me know what you get. I think that at 8.36 the website might not have data for the script pull information from and it won’t pull from the previous day (2nd August for this example).

image

The system clock is an hour ahead in the logs so I assume it must be on UTC. You can see the script ran at 11:00:00 returning a Usage: 0, and again when I forced via the switch at 16:55 with the same result.

Interestingly this is what I see on the Anglian Website for the daily view;

Note nothing for yesterday… And this for the hourly view;

Which does have some data!!?

:thinking:

That’s indeed very strange! I believe only when you get the full 24h usage data it will then show on your Daily Usage but it seems for you that it takes longer for that to be available, try checking later to see if you have it and change the script to run shortly after. Alternatively you can also adapt it to take the info from the hourly section like rodgers86 did.

Did you @Zeunas opt for more detailed readings when the meter was installed or do you only get the daily usage stats?

@rodgers86 - would you be so kind as to share the script you developed to pull the hourly data?

No, the ones you have but as I said perhaps at this time you already have the readings from the 3rd on your daily usage, worth to check and change the script to run at 10pm.

Happy to share! However the one that @Zeunas has for Daily works for the Energy dashboard, my amendment definitely does not as it stands!

It’s a bit of a botch job at the moment, so could probably be tidied up a bit.

First

You’ll need to add influxdb into the python packages in the configuration of appdaemon and then restart the addon.

You’ll also need to download the influxdb add on, start it,

then open the Web UI and create a database called “openhab_db” (that’s the name that was in the original script!)

Create a username and password and give it read and write permissions for the openhab_db

Next

Add the following to /config/appdaemon/apps/apps.yaml


WaterUsageHourly:
  module: GetWaterUsageHourly
  class: GetWaterUsageHourly

Then

Save the following script as GetWaterUsageHourly.py in config/appdaemon/apps

Adding your username and passwords for anglian water and influxdb in the process.

from influxdb import InfluxDBClient
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
import appdaemon.plugins.hass.hassapi as hass
import datetime
import time
import json
import ctypes

#####################
# App configuration #
#####################
# Complete with your settings
anglia_username = "USERNAME"
anglia_password = "PASSWORD"

influxdb_username = "USERNAME"
influxdb_password = "PASSWORD"
influxdb_url = "192.168.5.121"
influxdb_port = 8086

##################
# WATER USAGE APP #
##################

webpage = r"https://my.anglianwater.co.uk/"

class GetWaterUsageHourly(hass.Hass):

# Starting APP #

  def initialize(self):
      self.log ("intialising")
      self.run_every(self.StartProcess, "now", 8*60*60)

  #Callback Function to Start the process 
  def StartProcess(self, kwargs):
    self.log("starting process")
    usage = self.GetHourlyUsage()

# Get Water Usage from Anglia Water website #

  def GetHourlyUsage(self):
    try:
      self.log("starting request")
      service = Service(executable_path=r'/usr/bin/chromedriver')
      chrome_options = webdriver.ChromeOptions()
      chrome_options.add_argument('--headless')
      chrome_options.add_argument('--no-sandbox')
      chrome_options.add_argument('--disable-gpu')
      chrome_options.add_argument('--disable-dev-shm-usage')
      
      browser = webdriver.Chrome(service=service, options=chrome_options)
      browser.set_window_size(1600, 1200)
    
      self.log("logging in")
      browser.get(webpage)
      sbox = browser.find_element(By.ID,'existUser')
      sbox.send_keys(anglia_username)
      sbox = browser.find_element(By.ID,'existPass')
      sbox.send_keys(anglia_password)
      button = browser.find_element(By.ID,'existingLogIn')
      button.click()

      self.log("selecting usage")
      button = browser.find_element(By.ID,'btnViewUsage')
      button.click()
      time.sleep(2)

      self.log("selecting hourly")
      button = browser.find_element("xpath", '//*[@id="dbserialnumber"]/div/div[1]/div[1]/label/span')
      button.click()
      time.sleep(2)

      self.log("saving usage")
      html = browser.page_source
      browser.quit()
      startdata = html.find("var myUsageDetails_days = ") + 26
      enddata = html.find(";;", startdata)
      result = html[startdata: enddata]
      data = json.loads(result.replace("'", '"'))
      json_body = []

      for x in data:
          date_object = datetime.datetime.strptime(x['groupDate'] + " " + x['time'], "%d-%b-%Y %H:%M")
          item = {"measurement": "anglia_water_usage", "time": date_object, "fields": {"value": x["usage"] * 1000}}
          json_body.append(item)
      try:
          client = InfluxDBClient(influxdb_url, influxdb_port, influxdb_username, influxdb_password, 'openhab_db')
          client.write_points(json_body)
      except:
          self.log("Influxdb Error")
    except:
      self.log

If you get all this working, then next step is linking grafana, building dashboards and also potentially getting data into HA as sensors. I’m still working on this bit! But I’ve got sensors working and a chart in grafana, so I’m half way there!

I’m happy to share the work I’ve done getting the sensors into HA in both the methods as discussed in previous posts. None are really useful as yet for the energy dashboard, but I’ve got sensor for last 48hours, 7dayd and 30days working.

The limitation at the moment that I’ve not managed to get a way of pulling specifically yesterdays data in as a influxql query yet. I think it can be done in flux. But last 24hours or 48hours plus from “now()” so it doesn’t work the way I’d want it yet.

That’s why I may just ignore the energy dashboard and create my own water dashboard in grafana, as it’s much easier to use.

1 Like

Thanks both!

I want to try and keep my set-up as simple as possible without too much customisation. I’m going to have to set aside a good block of time to look/work through this.

Have you checked this? It does seem that your daily readings are only reading later in the day, as opposed to mine that are read before 11am.

It is bizarre… My daily readings are >24 hours behind. It is currently Saturday 5th August @ 16:20 and despite the the AW site indicating there is a reading for yesterday there is not;

However, when I look at the hourly readings, yesterdays data is available…

Looking at the script I think it is trying to pull the data from the daily view? And if it is presumably it is correctly pulling zero for the last day because it is zero for me?

        self.log("saving usage")
        html = browser.page_source
        browser.quit()
        startdata = html.find("var myUsageDetails_days = ") + 26
        enddata = html.find(";;", startdata)
        result = html[startdata: enddata]
        data = json.loads(result.replace("'", '"'))
        json_body = []

        for x in data:
            item = x["usage"] * 1000
            json_body.append(item)
        
        usage = str(json_body[-1])

Yes, it’s weird but that’s how it works for me too, before 11am I won’t have any readings from the previous day in the Daily tab but can see some readings from the previous day on the Hourly tab. At 11am is when I can see readings from the previous day on the Daily tab and the scrip will access that part of the site to get the daily usage. Hence why I’ve been saying that you should check when exactly can you see the previous day data and adjust the runtime accordingly.

Thanks @Zeunas. I was being a bit thick not understanding the significance of the timing. Needless to say I’ve just checked 21:30 and yesterdays readings are in now in the daily view on the Anglican Water website. Forcing a run with the input_boolean the script returns the expected value.

I just need to work out now how to get sensor.water_usage_x into the energy dashboard!

Have you figured it out? You basically need to follow the below Helper set up for a “utility meter” (I have highlighted the options you need to make sure are completed properly):

  • Make sure the input sensor you select is the ´sensor.water_usage_x’
  • Make sure the “utility meter” you create is called “sensor.water_usage”