Templates in command line sensor + Octopus Energy API

Hello all,

Struggling at the moment to get a sensor working to read out my electricity and gas meter readings.
My provider uses an API which is documented here: https://developer.octopus.energy/docs/api/ - see the section about List Consumption for a meter.
A REST GET command will return a JSON with all of the half-hourly meter usage readings for the time interval stated in the parameters. The JSON looks like this:

{
    "count": 48,
    "next": null,
    "previous": null,
    "results": [
        {
            "consumption": 0.063,
            "interval_start": "2018-05-19T23:00:00Z",
            "interval_end": "2018-05-19T23:30:00Z"
        },
        {
            "consumption": 0.071,
            "interval_start": "2018-05-19T22:30:00Z",
            "interval_end": "2018-05-19T23:00:00Z"
        }
    ]
}

I can confirm the service works using both REST and the cURL formatting using a static URL, for example:

platform: command_line
name: electricity
value_template: '{{ value_json.count }}'
command: 'curl -H "Authorization: Basic XXXXXXXXXXXXXXXXX" "https://api.octopus.energy/v1/electricity-meter-points/YYYYYYYYYYYYYYYYY/meters/18P5016779/consumption/?period_from=2019-01-31T00:00:00"'

Here’s the problem: if you don’t state the period_from parameter the GET returns thousands of values. I want just the previous day’s values, because from what I can see, the meter readings only update once a day around midnight - so I can’t get live readings, and the most recent readings I can get are for the 24 hours leading up to midnight the previous night.

It therefore makes sense that I use a GET request with a period_from parameter that dynamically changes to whatever yesterday’s date was.

Just to keep things simple, I made

value_template: '{{ value_json.count }}'

…because that should just return a value of 48 each time (48 half-hourly readings over a 24-hour period).

This worked absolutely fine with sensor.command_line and a URL without templating - I got the value 48 returned without issues.

From what I understand - the REST sensor doesn’t support templates in the URL, but the command line sensor does (or at least, is meant to).

At first, I tried this:

command: 'curl -H "Authorization: Basic XXXXXXXXXXXXXXXXX" "https://api.octopus.energy/v1/electricity-meter-points/YYYYYYYYYYYYYYYYY/meters/18P5016779/consumption/?period_from={{ (as_timestamp(now()) - (24*3600)) | timestamp_custom("%Y-%m-%d", True) }}T00:00:00"'

…which didn’t work, presumably because I ran out of " and ’ to play with. So I made a separate sensor:

sensors:
  date_yesterday:
    friendly_name: Date Yesterday
    value_template: '{{ (as_timestamp(now()) - (24*3600)) | timestamp_custom("%Y-%m-%d", True) }}'

and then tried:

command: 'curl -H "Authorization: Basic XXXXXXXXXXXXXXXXX" "https://api.octopus.energy/v1/electricity-meter-points/YYYYYYYYYYYYYYYYY/meters/18P5016779/consumption/?period_from={{ states.sensor.date_yesterday.state }}T00:00:00"'

This didn’t work either. I even tried making a separate sensor purely to generate the URL string:

  electricity_url:
    friendly_name: Electricity URL
    value_template: 'curl -H "Authorization: Basic c2tfbGl2ZV9QbjhvYVBOZFNEMUxJV2I4dU8xS2NEZWc6" "https://api.octopus.energy/v1/electricity-meter-points/2200042780238/meters/18P5016779/consumption/?period_from={{ states.sensor.date_yesterday.state }}T00:00:00"'

…followed by:

platform: command_line
name: electricity
value_template: '{{ value_json.count }}'
command: '{{ states.sensor.electricity_url.state }}'

This hasn’t worked either! I have checked the states of both sensor.electricity_url and sensor.date_yesterday and they both give the correct values.

The moment I revert back to using a URL without templating in the command line sensor - it works again.

I’m stumped as to what to try next - it really does just seem like the problem is the inclusion of templating in the command line for the sensor. Does anyone have any insights into this or suggestions I could try?

Many thanks

Al

I don’t really have a solution for you, and this might already be really obvious to you, but are you aware of the dev-template page in HA?

If you go to your URL with “/dev-template” at the end, or click on the icon of a a sheet of paper (4th from the left) at the bottom of the left-hand-side toolbar under “Developer tools”, it’s a template playground to test out templating.

For example I can see that:

value_template: '{{ (as_timestamp(now()) - (24*3600)) | timestamp_custom("%Y-%m-%d", True) }}'

Evaluates to:

value_template: '2019-01-31'

You can try and use this page to test out and refine those templates, if you weren’t already aware it exists.

BTW:
It looks like

command: 'curl -H "Authorization: Basic XXXXXXXXXXXXXXXXX" "https://api.octopus.energy/v1/electricity-meter-points/YYYYYYYYYYYYYYYYY/meters/18P5016779/consumption/?period_from={{ (as_timestamp(now()) - (24*3600)) | timestamp_custom("%Y-%m-%d", True) }}T00:00:00"'

Evalues just fine to:

command: 'curl -H "Authorization: Basic XXXXXXXXXXXXXXXXX" "https://api.octopus.energy/v1/electricity-meter-points/YYYYYYYYYYYYYYYYY/meters/18P5016779/consumption/?period_from=2019-01-31T00:00:00"'

Try this (which replaces the double-quote characters inside the template with escaped single-quote characters):

command: 'curl -H "Authorization: Basic XXXXXXXXXXXXXXXXX" "https://api.octopus.energy/v1/electricity-meter-points/YYYYYYYYYYYYYYYYY/meters/18P5016779/consumption/?period_from={{ (as_timestamp(now()) - (24*3600)) | timestamp_custom(''%Y-%m-%d'', True) }}T00:00:00"'

FYI, in YAML, a string that is quoted with single-quote characters can contain a single-quote character by entering the single-quote character twice.

Just tried this but unfortunately doesn’t seem to work either…

Can you expand on “doesn’t work”? Are there any useful (INFO, WARNING or ERROR) messages in home-assistant.log, such as Running command: ..., or Command failed: ..., etc.? If so, what are they?

Here’s another way, perform the date calculation in a shell script and pass the result to curl as a shell variable.

Create a shell script (get_data.sh) containing these two lines:

STARTDATE=`date -d yesterday +"%Y-%m-%dT00:00:00"`
curl -H "Authorization: Basic XXXXXX" "https://api.octopus.energy/v1/electricity-meter-points/YYYYYY/meters/18P5016779/consumption/?period_from=$STARTDATE"

Then call the shell script from the command_line sensor:

command: "get_data.sh"

NOTE:
In command, you might have to specify the path to the shell script.

It may be difficult to see but the date command is delimited by backquotes ( ` )

Oooh, I like that. Will try that now.

I’ve been struggling to debug it @pnbruckner - the sensor shows up in my entities list but state is just ‘unknown’. There’s nothing relating to it in the logs on the Info screen and nothing in Hassio --> System Log. I’ve set up Logger with:

logger:
  default: critical
  logs:
    homeassistant.components.sensor.electricity: debug

…but the only error I’m getting in home-assistant.log is:

2019-02-01 20:14:55 WARNING (MainThread) [homeassistant.config] Incomplete core configuration. Auto detected elevation: 0

Try changing:

    homeassistant.components.sensor.electricity: debug

to:

    homeassistant.components.sensor.template: debug

or even:

    homeassistant.components.sensor: debug

Ok, changed to:

logger:
  default: critical
  logs:
    homeassistant.components.sensor.template: debug
    homeassistant.components.sensor.command_line: debug

So I put @123’s script into the config folder and changed the sensor to:

  - platform: command_line
    name: electricity
    value_template: '{{ value_json.count }}'
    command: "/config/get_data.sh"

…but get this in the log:

2019-02-01 20:48:51 WARNING (MainThread) [homeassistant.config] Incomplete core configuration. Auto detected elevation: 0
2019-02-01 20:49:25 INFO (SyncWorker_8) [homeassistant.components.sensor.command_line] Running command: /config/get_data.sh
2019-02-01 20:49:25 ERROR (SyncWorker_8) [homeassistant.components.sensor.command_line] Command failed: /config/get_data.sh
2019-02-01 20:50:27 INFO (SyncWorker_6) [homeassistant.components.sensor.command_line] Running command: /config/get_data.sh
2019-02-01 20:50:27 ERROR (SyncWorker_6) [homeassistant.components.sensor.command_line] Command failed: /config/get_data.sh

Reverting back to @123’s syntax gives this in the log:

2019-02-01 20:55:59 WARNING (MainThread) [homeassistant.config] Incomplete core configuration. Auto detected elevation: 0
2019-02-01 20:56:35 INFO (SyncWorker_3) [homeassistant.components.sensor.command_line] Running command: curl -H Authorization: Basic xxxxxxxxxxx https://api.octopus.energy/v1/electricity-meter-points/yyyyyyyy/meters/18P5016779/consumption/?period_from=2019-01-31T00:00:00
2019-02-01 20:57:38 INFO (SyncWorker_11) [homeassistant.components.sensor.command_line] Running command: curl -H Authorization: Basic xxxxxxxxxxx https://api.octopus.energy/v1/electricity-meter-points/yyyyyyyy/meters/18P5016779/consumption/?period_from=2019-01-31T00:00:00

This is just weird! The log above suggests that everything is parsing correctly, yet it doesn’t yield the same result… I’ll try again with the non-template version and see what the log spits out and compare it.

Ok, progress. When it works with the non-template URL, the log shows:

2019-02-01 21:06:37 INFO (SyncWorker_16) [homeassistant.components.sensor.command_line] Running command: curl -H "Authorization: Basic xxxxxxxxxxxx" "https://api.octopus.energy/v1/electricity-meter-points/yyyyyyyyyyyy/meters/18P5016779/consumption/?period_from=2019-01-31T00:00:00"
2019-02-01 21:07:40 INFO (SyncWorker_2) [homeassistant.components.sensor.command_line] Running command: curl -H "Authorization: Basic xxxxxxxxxxxx" "https://api.octopus.energy/v1/electricity-meter-points/yyyyyyyyyyyy/meters/18P5016779/consumption/?period_from=2019-01-31T00:00:00"

So it looks like when there’s a template in there, it excludes the " from around the header and url, and therefore fails, but without the template, the " are preserved and it works

FIXED IT!

For anyone who has the same issue in future:

  - platform: command_line
    name: electricity
    value_template: '{{ value_json.count }}'
    command: >-
      curl -H '"'Authorization: Basic xxxxxxxxxxxx'"' '"'https://api.octopus.energy/v1/electricity-meter-points/yyyyyyyyyyyy/meters/18P5016779/consumption/?period_from={{ (as_timestamp(now()) - (24*3600)) | timestamp_custom("%Y-%m-%d", True) }}T00:00:00'"'

The solution was a) to put it in a new line and b) to surround double quotes with single quotes to protect them being stripped out by Jinja2 parsing (like this - ’ " ’ )

Thanks for steering me towards the solution everyone!

Glad to hear you resolved the problem!

Not that it matters anymore but for the solution I had proposed, this is what I needed to do to make it work:

  • Specify the full path to the script file. In my case it was:
command: "/opt/homeassistant/config/test.sh"
  • Ensure the user that owns the Home Assistant process (in my case the user’s name is homeassistant) has execution permission for the script file.

Error messages “Command failed: test.sh” followed by “test.sh: not found” mean the specified path was wrong.

Error messages "“Command failed: test.sh” followed by “Permission denied” means the process did not have permission to access the script file.

Ah, I thought I might have screwed up the path - whoops! Thanks anyway!

Any chance you could post the working setup? I’m a relatively new Octopus customer and would like to see the energy usage in HA.

Sure:

  - platform: command_line
    name: [gas or electricity]
    value_template: '{{ value_json.count }}'
    json_attributes: 
      - results
    command: >-
      curl -H '"'Authorization: Basic [encoded API key here]'"' '"'https://api.octopus.energy/v1/[insert 'gas' or 'electricity' here]-meter-points/[mpan or mprn]/meters/[meter serial number here]/consumption/?period_from={{ (as_timestamp(now()) - (24*3600)) | timestamp_custom("%Y-%m-%d", True) }}T00:00:00'"'

Note that for authorization you will need the encoded API key - easiest way to do this is to get your API key, go to https://client.restlet.com, click ‘set an authorization’, copy the API key into the username box, and click ‘set’ - you should then see the header formed with 'Basic + [encoded API key].

This should return a state of ‘48’ which is the number of 30minute intervals returned by the request, then all the attributes are the measurements of the intervals.

I’m rubbish at coding so the way I got a useable value is with this:

  gas_yesterday:
    friendly_name: Gas Use Yesterday
    icon_template: mdi:fire
    unit_of_measurement: 'kWh'
    value_template: "{{(state_attr('sensor.gas', 'results')[0]['consumption']+state_attr('sensor.gas', 'results')[1]['consumption']+....state_attr('sensor.gas', 'results')[47]['consumption'])|round(3)}}"

…yes, it’s a long line of code adding together all 48 values, then rounding it to 3 decimal places, but it works, and gives the total gas/electricity used in the previous day.

From what I can tell the meter sends the data for the previous day off around 0700h each day because that’s when the sensor refreshes with the next lot of data - so from 0000h to 0700h this won’t have a value because the API hasn’t caught up with the command line sensor.

I show both gas and electricity meter readings on the front end in a nice little graph showing my weekly energy usage:

A much more elegant solution would be a nice little bash script that loops around and adds the dictionary array of consumption values together - if you manage this please let me have a copy of your code!

Let me know if you run into any issues.

Oh - by the way - don’t be alarmed by your gas consumption numbers, the API reports it in kWh, not metres cubed!!!

2 Likes
  gas_yesterday:
    friendly_name: Gas Use Yesterday
    icon_template: mdi:fire
    unit_of_measurement: 'kWh'
    value_template: "{{(state_attr('sensor.gas', 'results')[0]['consumption']+state_attr('sensor.gas', 'results')[1]['consumption']+state_attr('sensor.gas', 'results')[2]['consumption']+state_attr('sensor.gas', 'results')[3]['consumption']+state_attr('sensor.gas', 'results')[4]['consumption']+state_attr('sensor.gas', 'results')[5]['consumption']+state_attr('sensor.gas', 'results')[6]['consumption']+state_attr('sensor.gas', 'results')[7]['consumption']+state_attr('sensor.gas', 'results')[8]['consumption']+state_attr('sensor.gas', 'results')[9]['consumption']+state_attr('sensor.gas', 'results')[10]['consumption']+state_attr('sensor.gas', 'results')[11]['consumption']+state_attr('sensor.gas', 'results')[12]['consumption']+state_attr('sensor.gas', 'results')[13]['consumption']+state_attr('sensor.gas', 'results')[14]['consumption']+state_attr('sensor.gas', 'results')[15]['consumption']+state_attr('sensor.gas', 'results')[16]['consumption']+state_attr('sensor.gas', 'results')[17]['consumption']+state_attr('sensor.gas', 'results')[18]['consumption']+state_attr('sensor.gas', 'results')[19]['consumption']+state_attr('sensor.gas', 'results')[20]['consumption']+state_attr('sensor.gas', 'results')[21]['consumption']+state_attr('sensor.gas', 'results')[22]['consumption']+state_attr('sensor.gas', 'results')[23]['consumption']+state_attr('sensor.gas', 'results')[24]['consumption']+state_attr('sensor.gas', 'results')[25]['consumption']+state_attr('sensor.gas', 'results')[26]['consumption']+state_attr('sensor.gas', 'results')[27]['consumption']+state_attr('sensor.gas', 'results')[28]['consumption']+state_attr('sensor.gas', 'results')[29]['consumption']+state_attr('sensor.gas', 'results')[30]['consumption']+state_attr('sensor.gas', 'results')[31]['consumption']+state_attr('sensor.gas', 'results')[32]['consumption']+state_attr('sensor.gas', 'results')[33]['consumption']+state_attr('sensor.gas', 'results')[34]['consumption']+state_attr('sensor.gas', 'results')[35]['consumption']+state_attr('sensor.gas', 'results')[36]['consumption']+state_attr('sensor.gas', 'results')[37]['consumption']+state_attr('sensor.gas', 'results')[38]['consumption']+state_attr('sensor.gas', 'results')[39]['consumption']+state_attr('sensor.gas', 'results')[40]['consumption']+state_attr('sensor.gas', 'results')[41]['consumption']+state_attr('sensor.gas', 'results')[42]['consumption']+state_attr('sensor.gas', 'results')[43]['consumption']+state_attr('sensor.gas', 'results')[44]['consumption']+state_attr('sensor.gas', 'results')[45]['consumption']+state_attr('sensor.gas', 'results')[46]['consumption']+state_attr('sensor.gas', 'results')[47]['consumption'])|round(3)}}"

  electricity_yesterday:
    friendly_name: Electricity Use Yesterday
    icon_template: mdi:flash
    unit_of_measurement: 'kWh'
    value_template: "{{(state_attr('sensor.electricity', 'results')[0]['consumption']+state_attr('sensor.electricity', 'results')[1]['consumption']+state_attr('sensor.electricity', 'results')[2]['consumption']+state_attr('sensor.electricity', 'results')[3]['consumption']+state_attr('sensor.electricity', 'results')[4]['consumption']+state_attr('sensor.electricity', 'results')[5]['consumption']+state_attr('sensor.electricity', 'results')[6]['consumption']+state_attr('sensor.electricity', 'results')[7]['consumption']+state_attr('sensor.electricity', 'results')[8]['consumption']+state_attr('sensor.electricity', 'results')[9]['consumption']+state_attr('sensor.electricity', 'results')[10]['consumption']+state_attr('sensor.electricity', 'results')[11]['consumption']+state_attr('sensor.electricity', 'results')[12]['consumption']+state_attr('sensor.electricity', 'results')[13]['consumption']+state_attr('sensor.electricity', 'results')[14]['consumption']+state_attr('sensor.electricity', 'results')[15]['consumption']+state_attr('sensor.electricity', 'results')[16]['consumption']+state_attr('sensor.electricity', 'results')[17]['consumption']+state_attr('sensor.electricity', 'results')[18]['consumption']+state_attr('sensor.electricity', 'results')[19]['consumption']+state_attr('sensor.electricity', 'results')[20]['consumption']+state_attr('sensor.electricity', 'results')[21]['consumption']+state_attr('sensor.electricity', 'results')[22]['consumption']+state_attr('sensor.electricity', 'results')[23]['consumption']+state_attr('sensor.electricity', 'results')[24]['consumption']+state_attr('sensor.electricity', 'results')[25]['consumption']+state_attr('sensor.electricity', 'results')[26]['consumption']+state_attr('sensor.electricity', 'results')[27]['consumption']+state_attr('sensor.electricity', 'results')[28]['consumption']+state_attr('sensor.electricity', 'results')[29]['consumption']+state_attr('sensor.electricity', 'results')[30]['consumption']+state_attr('sensor.electricity', 'results')[31]['consumption']+state_attr('sensor.electricity', 'results')[32]['consumption']+state_attr('sensor.electricity', 'results')[33]['consumption']+state_attr('sensor.electricity', 'results')[34]['consumption']+state_attr('sensor.electricity', 'results')[35]['consumption']+state_attr('sensor.electricity', 'results')[36]['consumption']+state_attr('sensor.electricity', 'results')[37]['consumption']+state_attr('sensor.electricity', 'results')[38]['consumption']+state_attr('sensor.electricity', 'results')[39]['consumption']+state_attr('sensor.electricity', 'results')[40]['consumption']+state_attr('sensor.electricity', 'results')[41]['consumption']+state_attr('sensor.electricity', 'results')[42]['consumption']+state_attr('sensor.electricity', 'results')[43]['consumption']+state_attr('sensor.electricity', 'results')[44]['consumption']+state_attr('sensor.electricity', 'results')[45]['consumption']+state_attr('sensor.electricity', 'results')[46]['consumption']+state_attr('sensor.electricity', 'results')[47]['consumption'])|round(3)}}"

Here’s the full code for the sensors that calculate the consumption total for you to copy and paste - should save you a bit of time!

1 Like

Thank you very much, I’ll give this a shot over the weekend! :slight_smile:

How did it go?

If you want to convert kWh to m3 for gas consumption you just need to divide by 11.36 (changes at very end of line 5):

  gas_yesterday:
    friendly_name: Gas Use Yesterday
    icon_template: mdi:fire
    unit_of_measurement: 'kWh'
    value_template: "{{(state_attr('sensor.gas', 'results')[0]['consumption']+state_attr('sensor.gas', 'results')[1]['consumption']+state_attr('sensor.gas', 'results')[2]['consumption']+state_attr('sensor.gas', 'results')[3]['consumption']+state_attr('sensor.gas', 'results')[4]['consumption']+state_attr('sensor.gas', 'results')[5]['consumption']+state_attr('sensor.gas', 'results')[6]['consumption']+state_attr('sensor.gas', 'results')[7]['consumption']+state_attr('sensor.gas', 'results')[8]['consumption']+state_attr('sensor.gas', 'results')[9]['consumption']+state_attr('sensor.gas', 'results')[10]['consumption']+state_attr('sensor.gas', 'results')[11]['consumption']+state_attr('sensor.gas', 'results')[12]['consumption']+state_attr('sensor.gas', 'results')[13]['consumption']+state_attr('sensor.gas', 'results')[14]['consumption']+state_attr('sensor.gas', 'results')[15]['consumption']+state_attr('sensor.gas', 'results')[16]['consumption']+state_attr('sensor.gas', 'results')[17]['consumption']+state_attr('sensor.gas', 'results')[18]['consumption']+state_attr('sensor.gas', 'results')[19]['consumption']+state_attr('sensor.gas', 'results')[20]['consumption']+state_attr('sensor.gas', 'results')[21]['consumption']+state_attr('sensor.gas', 'results')[22]['consumption']+state_attr('sensor.gas', 'results')[23]['consumption']+state_attr('sensor.gas', 'results')[24]['consumption']+state_attr('sensor.gas', 'results')[25]['consumption']+state_attr('sensor.gas', 'results')[26]['consumption']+state_attr('sensor.gas', 'results')[27]['consumption']+state_attr('sensor.gas', 'results')[28]['consumption']+state_attr('sensor.gas', 'results')[29]['consumption']+state_attr('sensor.gas', 'results')[30]['consumption']+state_attr('sensor.gas', 'results')[31]['consumption']+state_attr('sensor.gas', 'results')[32]['consumption']+state_attr('sensor.gas', 'results')[33]['consumption']+state_attr('sensor.gas', 'results')[34]['consumption']+state_attr('sensor.gas', 'results')[35]['consumption']+state_attr('sensor.gas', 'results')[36]['consumption']+state_attr('sensor.gas', 'results')[37]['consumption']+state_attr('sensor.gas', 'results')[38]['consumption']+state_attr('sensor.gas', 'results')[39]['consumption']+state_attr('sensor.gas', 'results')[40]['consumption']+state_attr('sensor.gas', 'results')[41]['consumption']+state_attr('sensor.gas', 'results')[42]['consumption']+state_attr('sensor.gas', 'results')[43]['consumption']+state_attr('sensor.gas', 'results')[44]['consumption']+state_attr('sensor.gas', 'results')[45]['consumption']+state_attr('sensor.gas', 'results')[46]['consumption']+state_attr('sensor.gas', 'results')[47]['consumption']) /11.36 | round(3)}}"

After seeing the value_template you created, it made me wonder if there was a way in Jinja2 to make it more compact.

At first I jumped on the idea of using a for-loop. Just iterate through all the consumption values and add them up. Unfortunately, the latest version of Jinja2 (2.10) has tightened up its scoping rules and, long story short, you can’t calculate the sum using a for-loop.

Today I saw this post and thought it was not only clever but a potential solution for making your value_template more compact.

It performs its magic using map and sum. The map creates a list containing the desired attribute values and then sum adds up all entries in the list. Very neat!

Using the Template Editor, I created some data and then tested the template.

{% set x = {"results":[{"consumption": 10}, {"consumption": 20}, {"consumption": 30}, {"consumption": 40}, {"consumption": 50} ]} %}

{{ x.results | map(attribute="consumption") | sum }}

The result is 150 as would be expected from 10+20+30+40+50.

I don’t have access to the actual data you are using so I can’t confirm that what I have to offer will instantly work for you. Here it is:

    value_template: '{{ (state_attr("sensor.gas", "results") | map(attribute="consumption") | sum / 11.36) | round(3)}}'

If it doesn’t work, please give me the data from your state_attr('sensor.gas', 'results') and I’ll make it work.


EDIT
As expected, the suggested template did contain an error and has now been corrected as per ajoyce’s suggestion.

2 Likes

Oh @123.

I like you a lot.

That works absolutely perfectly - being of a non-coding background I spent ages looking at python script of variables and for/sum loops and couldn’t figure out any way of making it work with a Jinja2 template.

What a beautifully simple bit of code! Many, many thanks for fixing the biggest headache in my configuration.yaml!

1 Like