Octopus Agile - display tariff in graphs & tables, best import/export periods - all done using Node-RED and JSONata

Yes Friday night - I really should get a life.

It is not unknown for people to post ‘I can’t see anything in my entities’ and then ‘oh yes they are here’. I do think that sometimes it takes a while, and then there is ‘caching’ in web browsers, and all that stuff.

This is, really, rather complex and a lot is going on, so frankly it is amazing that a) it works and b) someone else can download my Node-RED and flow and make it work for themselves.

Yes, the _2 sensors are a pain. This happens because all entities are registered with HA (created) and HA keeps the registrations for a while, and if you remove the Node-RED entity then it vanishes from HA, but when you add it back, HA already remembers it, so adds a second. (Then you patch everything in HA to _2, and then after the next reboot HA has forgotten your first attempt, then re-creates without the _2 and you have to update everything again. Deep Joy)

I sincerely hope it adds value for you.
There are other integrations and bits and pieces using Octopus Agile, but the advantage here is that you can do anything with the Node-RED flow (code).

I have moved on to a new update but it is too big to post here, so I have at some point to get around to posting it to the Node-RED forum. There have been a few issues, but mostly this all works, day after day, and pulls down the Agile Prices and does a bit of simple processing.

Good luck with your project work - I started with Solar PV two years ago and this has turned into quite a time consuming project, but I have learnt so much!

Best wishes.

1 Like

This is really a great piece of work!

I have been playing with Home Assistant on and off for a year or so, and experimented with Node Red too.

I am on Octopus Agile. I don’t have a battery, but am taking delivery of an EV in the next couple of weeks. The use case I am looking at is setting a target state of charge and ready-to-go time for the EV, then based on the amount of energy required, picking the cheapest ‘n’ 30-minute timeslots starting now to charge the car, reaching the desired state of charge at or before the ready-to-go time.

My NodeRed experience is pretty much limited to turning stuff on when Saving Sessions occur, activating appliances when prices go negative, switching christmas tree lights on and off according to a schedule and so on…

Looks like you’ve done most of the work for me with your code, need to see whether I can get the rest of what I need over the line!

I’m up and running, with two-and-a-half automated use cases

  1. immersion on/off in the low price periods; working nicely!
  2. heat pump auto off at the start of the high export price periods. Still trying to decide whether to do the inverse and turn heat pump on again at the end of the high export period
  3. car chargepoint automatically on at the start of the low price period. Weirdly, it seems my Zappi unit isn’t easy to turn off! So I either need to work out how to programmatically set end time for the (smart) boost, or rely on the car’s charge limit, or change the mode of the chargepoint to “stopped” and then back to what it was. Not ideal.

One little question - I haven’t done any definitive testing, but on the Apex chart, the previous day’s optimal periods disappear after the new rates are imported. It is likely that a high export period starts before 4pm and ends after the rates have been refreshed - in the example below (imagining I were using export array rather than import array which is the one likely to cross over the refresh time) do you expect the “off” template will still be triggered?

I think the automation code provided by @Shaggy91 sets offtime at the point in time of the template running and therefore only a failure of Home Assistant, not a refresh of the sequence table, would prevent off trigger running?

Also, is there any way to keep the previous day’s periods displayed on the Apex Chart? I’m guessing the array gets refreshed, so likely no.

alias: Immersion Agile
description: Immersion Agile price integration
trigger:
  - platform: template
    id: "on"
    value_template: >
      {% set ns=namespace() %}

      {% set ns.flag=false %}

      {% for event in state_attr('sensor.octopus_agile_sequence_table_2',
      'import_array')%}
        {% set ontime=(as_timestamp(event.from)|int-as_timestamp(now())|int) %}
        {% set ns.flag = ns.flag or ( 0 < ontime < 120) %}
      {% endfor %}

      {{ns.flag}}
  - platform: template
    id: "off"
    value_template: >
      {% set ns=namespace(flag = false) %}

      {% for event in state_attr('sensor.octopus_agile_sequence_table_2',
      'import_array')%}
        {% set offtime=(as_timestamp(event.upto)|int-as_timestamp(now())|int) %}
        {% set ns.flag = ns.flag or (0 < offtime < 120) %}
      {% endfor %}

      {{ns.flag}}
condition: []
action:
  - service: switch.turn_{{ trigger.id }}
    target:
      entity_id: switch.fstwifitu_immersion_ext2_switch_1
mode: single

Glad to hear it is working for you.

I confess that I don’t actually use Agile. I have it all running and do check the display from time to time, but I don’t use it ‘in anger’ so I have never really check the end use-cases like this.

The tariff array holds 48 hours worth so it spans both up to 23:00 today, and after 16:00 also tomorrow. Yes you are correct - once the update runs the best groups and the best consecutive periods are re-calculated on the new (tomorrow) tariff part, and these then overwrite the stored context variables.
Yes, if you pre-load the on / off times as required before the 16:00 update then switching should continue over the transition period. However, I am sure this is something that could be improved.

In the short term, you can change the update trigger time to later. I originally set this to just after 16:00, which is the earliest point Octopus update the tariffs, and mostly reliable, but Octopus had a period back in the year when it was happening nearer 19:00. I have changed the code so that update runs when there are only 10 off 30 minute periods left, so as to run every half hour after say 18:00 and keep on checking until a successful update happens.

I will have a look at the code and see if I can merge the new calculation with the existing values, then trim to just today-tomorrow, and save that. This should be relatively easy, and will fix both display and availability for control - I probably need to add an extra field for “yesterday” and “today” so it is possible to tell the two combined sets of data apart!

After your post I have been mindful that I have not yet published my new version which has been tested for DST and year end (just fixed a bug in that). This update is overdue, but I have to push the code out via Node-RED forum as it is too big for this site. Let me see what I can do…

Thanks Geoff!

I’m not on Agile -yet- but I’m planning my automations as if I were. In the meantime, or if I stay on Intelligent Go, I figure that these automations will still reduce my household carbon profile because Agile prices reflect CO2e very closely.

I’m using the Octopus Compare app which looks at usage data for my account via the Octopus API. In the month since I had the heat pump installed, Agile would have been nearly 40% cheaper (the app doesn’t know about the intelligent cheap rate periods, only the 6 hour cheap rate periods, so it overestimates the Agile saving slightly). Incidentally Agile would have been 70% cheaper than Flexible for me this month. Flexible is not meant for high off peak users!

It’s been a windy month so this may be unusual but if I get the automations right I will be able to ditch Octopus’ version they call intelligent for the EV, and use your version for everything, thus getting a true indication of my Agile savings. Yours does seem to match up very closely with the schedules they create for the EV (unsurprisingly but reassuringly.)

FYI, I’ve changed the trigger time to 20:05 as the last “high export” period commonly finishes around 19:30 and I’d like to be able to see that it’s still active for debugging (which is commonly an evening activity for me.) Looking forward to seeing your up to date flow when you find a way to publish it. In the mean time this is great and I didn’t perceive any end of year bug!

Hi,

I ended up splitting the on/off routine to:-

alias: Octopus Agile On when cheap
description: Switches the charger on as required due to price
trigger:
  - platform: template
    value_template: >
      {% set ns=namespace() %}

      {% set ns.flag=false %}

      {% for event in state_attr('sensor.octopus_agile_sequence_table',
      'import_array')%}
        {% set ontime=(as_timestamp(event.from)|int-as_timestamp(now())|int) %}
        {% set ns.flag = ns.flag or ( 0 < ontime < 120) %}
      {% endfor %}

      {{ns.flag}}
condition: []
action:
  - type: turn_off
    device_id: c6a94c2771b1017d237a2296bdcedfd2
    entity_id: e5459e5851f10461467b65b25e3ff78d
    domain: switch
  - type: turn_off
    device_id: 7edec7d57d25aec2087890f5a5f04a67
    entity_id: 8ebeb8b43536bd1b90b49f4c15c9aeb7
    domain: switch
  - delay:
      hours: 0
      minutes: 1
      seconds: 0
      milliseconds: 0
  - type: turn_on
    device_id: 9b47a99753f04f2492261578d394f039
    entity_id: 1091735bf66ed005ae641aa6e72eb8b2
    domain: switch
  - if:
      - type: is_battery_level
        condition: device
        device_id: 1b2ce1663e3ac6a5d2f0e62b8d31d15a
        entity_id: ba4c9140c65fea26f759bd9ef9709af5
        domain: sensor
        below: 92
      - condition: numeric_state
        entity_id: >-
          sensor.octopus_energy_electricity_22j0694042_1100019822666_current_rate
        below: sensor.current_import_data
        enabled: false
    then:
      - type: turn_on
        device_id: e5cb3e8425f3ee9e4e6fb40b9e6fdf09
        entity_id: 97f67dea5e5ec0468f40f42e13439060
        domain: switch
mode: single

and

alias: Octopus Agile Off when not cheap
description: Switches the charger off as required due to price
trigger:
  - platform: template
    value_template: >
      {% set ns=namespace(flag = false) %}

      {% for event in state_attr('sensor.octopus_agile_sequence_table',
      'import_array')%}
        {% set offtime=(as_timestamp(event.upto)|int-as_timestamp(now())|int) %}
        {% set ns.flag = ns.flag or (0 < offtime < 120) %}
      {% endfor %}

      {{ns.flag}}
condition: []
action:
  - if:
      - condition: device
        type: is_on
        device_id: e5cb3e8425f3ee9e4e6fb40b9e6fdf09
        entity_id: 97f67dea5e5ec0468f40f42e13439060
        domain: switch
    then:
      - type: turn_off
        device_id: e5cb3e8425f3ee9e4e6fb40b9e6fdf09
        entity_id: 97f67dea5e5ec0468f40f42e13439060
        domain: switch
  - type: turn_off
    device_id: 9b47a99753f04f2492261578d394f039
    entity_id: 1091735bf66ed005ae641aa6e72eb8b2
    domain: switch
  - delay:
      hours: 0
      minutes: 0
      seconds: 30
      milliseconds: 0
  - type: turn_on
    device_id: c6a94c2771b1017d237a2296bdcedfd2
    entity_id: e5459e5851f10461467b65b25e3ff78d
    domain: switch
    enabled: false
  - if:
      - condition: numeric_state
        entity_id: >-
          sensor.octopus_energy_electricity_22j0694042_1100019822666_current_rate
        below: 0.99
        above: input_number.locked_in_price
    then:
      - type: turn_on
        device_id: c6a94c2771b1017d237a2296bdcedfd2
        entity_id: e5459e5851f10461467b65b25e3ff78d
        domain: switch
      - type: turn_on
        device_id: 7edec7d57d25aec2087890f5a5f04a67
        entity_id: 8ebeb8b43536bd1b90b49f4c15c9aeb7
        domain: switch

Which controls multiple switches with caveats to charge batteries and protect elements. The offtime on the table works well. The only failure is if the switch drops off the network (on wifi) or if home assistant has a bad day. To try and put these niggles to rest i have 3 asus routers around the house in AIMesh mode and they reboot every 2 or 3 days at 0310 in the morning. This helps keep the network fresh and reduces hiccups with devices dropping off.

1 Like

Nice coding :clap:

I’m finding that the octopus intelligent integration is going offline at random times. I have set up an automation to reload the integration when it goes “offline” or in to an “unknown” state which can work done off the times, but the other times the error log says:

Logger: homeassistant.components.automation.intelligent_reload
Source: components/automation/__init__.py:690
Integration: Automation (documentation, issues)
First occurred: 09:16:14 (3 occurrences)
Last logged: 11:13:32

Error while executing automation automation.intelligent_reload: The config entry (octopus_intelligent) with entry_id 7ecb0d4804d0d6fa151bb98d91e79aca cannot be unloaded because it is not in a recoverable state (ConfigEntryState.FAILED_UNLOAD)

Has anyone else experienced this and has a better work Around? I also checked another friend HA instalation when my intelligent integration was offline in this state, his was still working OK…

1 Like

I’m trying a different way (I’m not on Agile yet so I can’t use the Octopus sensors that you have). However I’ve run out of coding skill. The data I want does exist and below is what ChatGPT suggests to only run if the average is above 15p/kWh (the minimum average price I want to export) but I’m not convinced the arrays will match in the enumeration - any thoughts please?

EDIT: removed nonworking code

Yeah the arrays don’t match so that didn’t work. Two ways I was overcomplicating

  1. Every day I’ve observed, the higher price peak is in the afternoon, so I can just add a condition to ignore the morning. Only thing is, I’m not sure if this will always be true on sunny days in the summer. Time will tell I guess.
condition:
  - condition: time
    after: "15:00:00"
    before: "19:30:00"
    weekday:
      - mon
      - tue
      - wed
      - thu
      - fri
      - sat
      - sun
  1. We’re already in the export array, so it’s simpler to use the export prices as a proxy for the import prices, and just adjust the trigger event value accordingly, then find the average of the next few events. I think I have an appropriate template with conditions and I’m testing it.

The first code below works, using export array as a proxy for high import price/high carbon to turn off power-intensive device(s), in my case all AC zones.

What mistake have I made with the template below it, which attempts to use the import prices sensor directly? I realise that would trigger for any half-hour period, not for a contiguous block but I would like to experiment with both (rather than deviate from Geoff’s node-RED flow and add a high-import prices block to the array, which would be yet another alternative I may yet pursue. I read that there are limits on array size though…)

The message from Developer Tools: Template for the second set of code is
TypeError: ‘NoneType’ object is not iterable

alias: Agile Export peak above slider value
description: >-
  Agile high price/carbon turn AC off and create notification (test for slider,
  works with value)
trigger:
  - platform: template
    id: "off"
    value_template: >
      {% set ns=namespace() %}

      {% set ns.flag=false %}


      {% for event in state_attr('sensor.octopus_agile_sequence_table_2',
      'export_array') %}
        {% set ontime=(as_timestamp(event.from)|int-as_timestamp(now())|int) %}
        {% if 0 < ontime < 120 and event.average | float >= states('input_number.agile_export_target') | float %}
          {% set ns.flag = true %}
        {% endif %}
      {% endfor %}

      {{ ns.flag }}
condition: []
action:
  - service: climate.set_hvac_mode
    data:
      hvac_mode: "off"
    target:
      entity_id:
        - climate.bedroom_2_c
        - climate.bedroom_3_d
        - climate.downstairs_open_plan
        - climate.master_bedroom_a
        - climate.study_b
  - service: persistent_notification.create
    metadata: {}
    data:
      message: >-
        AC turned off because average for upcoming period exceeded export target
        slider value. Adjust the export target slider if action desired.        
      title: High carbon:AC off!
  - service: notify.whatsapp
    metadata: {}
    data:
      message: >-
        Price/carbon optimisation active: AC off! Average for upcoming period
        exceeded export target slider value. 
mode: single

Note: number helper slider defines the export target in the code above; for testing simplicity it’s hard-coded in the one below.

{% set ns=namespace() %}
{% set ns.flag=false %}

{# We use the calculated high export period to turn *off* some high-current devices, saving money on Agile and/or reducing carbon load by proxy #}
{% for event in state_attr('sensor.octopus_agile_prices_2', 'array') %}
  {% set ontime=(as_timestamp(event.from)|int-as_timestamp(now())|int) %}
  {# Templates run each minute, so allow 60 secs each way, then check if the average is above the target set by a number helper (inconvenient if low impact) #}
  {% if 0 < ontime < 120 and event.import >= 18 %}
    {% set ns.flag = true %}
  {% endif %}
{% endfor %}
{{ ns.flag }}

Edit: corrected the capitalisation of array as per Geoff’s advice

It should probably be ‘array’ and not ‘Array’

Doh! Thanks… :slight_smile: It appears capitalised in the entities view!

I’ve been trying to get this working and im getting an error in the logs and i’m a bit stuck.

When i check the state in the developer tools, it says unavailable, however if i put a debug node on the end of the pricing flow i do see an entity with 96 attributes.

I’m quite new to this - this is my first delve into Node-Red so its quite possible im doing/have done something wrong! So any advice is much appreciated

After a quite a lot of googling, tinkering and trying stuff out, i think i have found the solution. It came down to the sensor configuration. The state class of measurement was the issue, if you clear that down and restart it the problem goes away.

Heres a pic of the config that worked for me;

Alas, my original over-exuberant use of State class ‘measurement’ hit problems last year when the rules were tightened.

Although this was picked up at the time, it was remiss of me not to try to edit my original post (I don’t think I was aware that this was actually possible at the time).

I have now re-loaded the original Node-RED flow (in first post) with slight modifications to clear the offending State Class entries.

1 Like

Hard to believe that it is one year since I first posted this project…

Although I still don’t use Octopus Agile tariff myself, I continue to look at the graphical display of agile prices, and have been tinkering with the code ever since. Last summer I attempted to add DST and local time. I waited until the time-change in October to test it, and I did for the first time in my life get up in the night just to watch the clocks change. The code did, for the most part, actually work.

From discussions on this thread, I have added a binary sensor to act as a switch. Updates to the display (graph, tables) as well as general improvements to the JSONata code itself.

The basic JSON file for the Node-RED flow is now too large to post here. I have finally got round to learning Github and all that entails, so I am now officially an old git. Yet more learning achieved.

So, the latest update is now on Github. I have collated the documentation into the read.me file, and tested the release, so time to commit.

https://github.com/oatybiscuit/Node-RED-HA-Octopus-Agile-JSONata?tab=readme-ov-file#node-red-ha-octopus-agile-jsonata

1 Like

Hi Geoff - I don’t know if you’d prefer this feedback/ask on github or here… but as I’ve already posted about a different issue there, I’m starting here with this one until you say different :slight_smile:
First of all thank you for such an amazing job in terms of intellectual effort AND such helpful writeups and support to people who’ve come unstuck, and all whilst turning down ‘buy me a coffee’ - what an amazing bloke.I am just struggling to get the entity sensor.octopus_agile_sequence_table to display anywhere* and have just discovered the following warnings repeating in nodered:

“Deprecated API warning: Calls to RED.util.evaluateJSONataExpression must include a callback. This will not be optional in Node-RED 4.0. Please identify the node from the following stack and check for an update on npm. If none is available, please notify the node author.”

18/02/2024, 18:08:41msg : string[1378]
</>
“Error: ↵ at Object.evaluateJSONataExpression (/opt/node_modules/@node-red/util/lib/util.js:775:18)↵ at JSONataService.evaluate (/config/node_modules/node-red-contrib-home-assistant-websocket/dist/common/services/JSONataService.js:58:39)↵ at TypedInputService.getValue (/config/node_modules/node-red-contrib-home-assistant-websocket/dist/common/services/TypedInputService.js:58:85)↵ at SensorController.onInput (/config/node_modules/node-red-contrib-home-assistant-websocket/dist/common/controllers/SensorBaseController.js:76:40)↵ at SensorController._InputOutputController_preOnInput (/config/node_modules/node-red-contrib-home-assistant-websocket/dist/common/controllers/InputOutputController.js:62:76)↵ at /opt/node_modules/@node-red/runtime/lib/nodes/Node.js:214:26↵ at Object.trigger (/opt/node_modules/@node-red/util/lib/hooks.js:166:13)↵ at Sensor.Node._emitInput (/opt/node_modules/@node-red/runtime/lib/nodes/Node.js:206:11)↵ at Sensor.Node.emit (/opt/node_modules/…”
</>
18/02/2024, 18:08:42msg : string[270]

Have I broken something or is there a direction of investigation you could recommend please?

*example symptom:

apexcharts-card custom card attempts produce:
Entity not available: sensor.octopus_agile_sequence_table
Entity not available: sensor.octopus_agile_sequence_table

Hi.

I have seen your messages on Github, I’m just not able to respond easily at the moment.

The issue you quote above is most likely due to out of date Websocket nodes. With Node-RED v3+ all JSONata calls now expect a callback function, which was indeed added in the HA Websocket nodes in a recent update. If you are on the latest HA core, the latest NR addon, the latest NR companion integration and the latest Websocket nodes, you should not see this error and it should all work.

I suggest checking your versions. BUT DO BE CAREFUL. There have been several breaking changes across these components, and incompatible updates may break your system.

As a start, I suggest:
Take an inventory of your versions, and ideally save a backup copy and remove your old (this) Node-RED flow completely (delete, restart HA and NR), before installing again afresh. So much has changed since my first flow that it is probably better to start again rather than try and copy over the old flow - but you will need the latest versions.

Backup first!

Thank you Geoff. I backed up this morning, and have inventoried as you kindly suggested:
HA Core 2024.1.6 vs 2024.2.2
NR Addon 17.0.7 - current
NR companion 3.1.2 vs 3.1.3
Websocket nodes 0.55.1 vs 0.63

so you were right on the money with your diagnosis. I disabled config nodes, deleted the whole flow, restarted, re installed, re configured. Whenever I try this I do run into nodered alerting me to pre-existing nodes, and I’ve not yet taught myself how to delete these from homeassistant & nodered successfully.

Having updated websockets I am no longer seeing either of those nodered warnings - so thanks very much.

I can get to green lights on all areas of your flow, working price array and table, but no joy on the sensor.octopus_agile_sequence_table which appears still to not exist. I need to go and re-read your notes in case it needs to wait for cronplus and will magically appear in a few hours.

Anyway - a huge leg up on my journey to assessing solar forecast, battery soc and current load trend and making sensible charging decisions. Really brilliant!