Octopus Agile - display tariff in graphs & tables, best import/export periods - all done using Node-RED and JSONata

Biscuit · August 2, 2023, 8:05pm

Not an issue I have seen, but at a guess the companion integration is missing, or needs updating.

https://github.com/zachowj/hass-node-red#installation

To make all this work, as well as Node-RED (the HA addon) which comes with the pallet containing the HA nodes (node-red-contrib-home-assistant-websocket), the ‘custom integration’ is the bit that does the work to make the HA server and websocket/API operate.

Suggest…
Check for the Node Red Companion in the HA integration page.
Check your config/custom_components/nodered folder, and if necessary remove everything and reinstall the config (link above).

When the Node-RED addon is updated the pallet updates, including the bundled HA websocket nodes. The version of the custom integration that these require also updates. The latest is 2.1.1. I did have issues with updating a version some time ago, the problem being that I kept old copies of the files renamed ‘old*’ in the same folder but the python engine was finding and reading the old file.

Hope this helps

Shaggy91 · August 2, 2023, 8:52pm

Cheers Geoff, I got the integration added back in and the error has gone away, no errors whatsoever but the entity agile_prices remains empty even though i have updated with my API http code ?page_size=96. The sequence connectd and works with no issue, just no prices. The Agile_sequence has worked giving me times when the best rates are…

I guess i need to wait til update tomorrow and see if it all loads up…

Regards

Shaggy

Shaggy91 · August 3, 2023, 7:23pm

Hi,

Basically i have jumped in on this after looking at it - HA now switches my boiler off for hot water if the agile prices will be at least 1p kwh cheaper than gas (which i am on the gas tracker), and then will switch on the immersion when the electric price gets to at least 1 p/kw cheaper than gas. As for price comparisons i have Octopus compare on my phone that tracks my usage daily/weekly/monthly and shows me stats on usage compared to another tariff i have chosen, not sure if this is any help to you…
When prices go negative thats when i go nuts…I have battery storage that gets charged up but i will hold the charging back until i can take advantage of max -tive prices. the immersion goes on as well, air conditioners, oven, grill, 3 old bitcoin miners that had been retired and a dehumidifier…so far the best i have done is they paid me £1.85…

Regards

Shaggy

PS there is also an historical website for agile prices along with one for when the prices go negative, and a gas/electric price tracker for the tracker tariffs going back to december 2022…

PedroKTFC · August 4, 2023, 6:42am

That’s very interesting. So you’re using the agile tariff! When did you go onto it and what from? I presume it’s saved you money? I wonder what would have happened to you last year when the prices sky rocketed? Would you have been “toasted”?! :o)

Shaggy91 · August 5, 2023, 1:51pm

I was originally on econ 7 so without the battery i was getting stung at £300 a month with electricity. Moving on with the battery reduced that to £130 to £170 and with agile currently that is down to £76. What would it have been with agile pricing last year? Alot more than econ 7! Prices were high all the time according to https://agileprices.co.uk/ so yes i would have been toasted…but that being said i am assured that i can jump off the agile tariff, without notice or penalty, onto another tariff but cannot rejoin agile for 30 days - so if prices rocket again i shall be looking at other tariffs but at this time agile fits the bill.
Gas Tracker allows me to cut my gas price in half. Looking at the history for gas and electricity, last december the electric price peaked at 60p kw but averaged 30p kw through December then dropped to 20p/kw January onwards. I am on the gas tracker so that did peak above standard prices but has been below standard for months now. If that goes nuts i can drop off onto a standard tariff but it will take two weeks to do so and i cannot go back on it for 9 months, so really that is the one i need to monitor and calculate using past prices…

Best regards

Shaggy

Shaggy91 · August 6, 2023, 1:40pm

For those interested the code someone wrote for me using Geoffs code, to turn stuff on and off according to the cheapest points that Geoffs code creates, is this:

trigger:
  - platform: template
    id: 'on'
    value_template: |
      {% set ns=namespace() %}
      {% set ns.flag=false %}
      {% for event in state_attr('sensor.octopus_agile_sequence_table', 'import_array')%}
        {% set ontime=(as_timestamp(event.from)|int-as_timestamp(now())|int) %}
        {% set ns.flag = ns.flag or ( 0 < ontime < 120) %}
      {% endfor %}
      {{ns.flag}}
  - platform: template
    id: 'off'
    value_template: |
      {% set ns=namespace(flag = false) %}
      {% for event in state_attr('sensor.octopus_agile_sequence_table', 'import_array')%}
        {% set offtime=(as_timestamp(event.upto)|int-as_timestamp(now())|int) %}
        {% set ns.flag = ns.flag or (0 < offtime < 120) %}
      {% endfor %}
      {{ns.flag}}
condition: []
action:
  - service: switch.turn_{{ trigger.id }}
    target:
      entity_id: switch.testing

Substitute switch.testing with whatever switch you wish to have turn on and off…I can vouch it works and works well with one switch and charges my batteries up accordingly.

I claim no kudos for the code, that lies with those who originally wrote it…

Regards

Shaggy

GavinAllison · August 12, 2023, 11:22am

Firstly, thanks everyone for all the work everyone has done on this.

I think I’ve managed to set most of this up correctly. NodeRed seems to be running but I’m getting some strange behaviours.

NodeRed is running with no errors in the logs. (Can’t attach many images as only new user)

The sequence table entity is working and showing data if I select ‘more’

But the prices entity is not:

I can see the best times table but the prices graph isn’t working. It shows the axis but shows no data and says “loading”.

I would really appreciate any help anyone can offer. Thanks.

GavinAllison · August 12, 2023, 12:23pm

I solved this by re-reading the thread and finding the solution: Octopus Agile - display tariff in graphs & tables, best import/export periods - all done using Node-RED and JSONata - #31 by PedroKTFC

Biscuit · August 12, 2023, 1:31pm

I put the Node-RED flow together, and it was (and still is) all working fine for me. A challenge in exporting NR flows that contain Home Assistant websocket nodes is the HA server and the entity configuration nodes that are required to make the sensor nodes work.

Best practice is to ‘scrub’ the flow to remove the configuration nodes, as importing these when you may already have them results in extra unwanted configuration nodes at best, and potential conflict at worst. I do now try to both remove the HA config node and disable entity nodes for export.

This prevents conflict issues at import, however it does mean that each sensor node requires the associated HA entity config node to be reconfigured and reattached to a working HA server node. I do not know quite what the issue is here, but I am assuming that the configuration node for the ‘prices’ sensor node is not setup and working correctly, hence the sensor will not be updated.

As fixing issues with HA entity nodes can be tricky at the best of times, whilst deleting and creating new sensor and their configuration nodes is perhaps a blunt-instrument approach, it will almost always address the issue.

I hope to improve the ‘import instructions’ at some point to cover this!

PedroKTFC · September 26, 2023, 8:30am

Octopus have recently introduced “free periods” for their customers. I don’t know if it’s for every tariff or limited (I’m on Flexible Octopus). It works like this:

You get an email the day before asking if you want to opt in (why wouldn’t you?!)
You click on the link in the email and you go to a page where you opt in
To date it’s always been the same time: 2pm to 4pm
You will get credited on your bill for any electricity used in this free period

The last point is crucial. It’s not that your tariff is zero in the free period. Octopus’s billing system applies the credit retrospectively so you can’t write an automation to exploit this by looking at the tariff.

At the moment I do the following to make the most of this:

I have a toggle helper I manually set the day before to flag a free period is coming
I have a node red flow that checks at 2pm if this helper is set. If so, it switches on my immersion heater.
At 4pm it switches the immersion heater off and resets the helper.

I wonder if there’s any way of automating this fully? Reading the email and automatically opting in and setting my helper?

Biscuit · September 30, 2023, 5:36pm

Congratulations on living in the right area! Octopus free energy is only available to people near an offshore windfarm.

Personally I have not been invited (I live just in ‘the west’) so I do not have sight of the email, however with access to an IMAP server emails can be read. Octopus usually use html format, so parse the email body to a JSON object, then use JSONata to find the relevant info.

I am currently using the contrib cron plus node which allows for dated schedules, so it would be possible to set both on and off events for a given date/time, and use these to switch a binary sensor or automations.

The more challenging bit is probably the auto opt in, but I guess once the correct ‘button’ has been found in the HTML structure, it is just a case of proving that your automation is not a robot.

Should be possible…

PedroKTFC · October 2, 2023, 10:55am

Blimey, I hadn’t appreciated how “lucky” I was. I have put together a node red flow using IMAP (and just looking for specific text in the subject) and it works with test emails. However, as luck would have it, I haven’t had a real email to prove it yet. When I do, I’ll post how I did it.

PedroKTFC · October 11, 2023, 11:25am

I’ve now put together a substack article on how I’m automating the Octupus Power up free periods as much as possible. It’s here for anyone that’s interested.

aryankids · October 22, 2023, 7:09pm

@PedroKTFC
Many thanks for sharing your experience. Will follow your footsteps.

Have noticed that you are using a ZigBee device to control the immersion heater. I am sure you’ve taken into consideration that immersion heaters are high-load devices and could melt down any smart device that controls them. This could be catastrophic as we tend to use the heater in the middle of the night etc.

To prevent a disaster, I would suggest controlling the heater with a high-load switch like this and controlling the switch with a smart device like Sonoff basic relay in turn. Putting all of these in an electrical fire-rated box is also highly recommended. The whole setup should not cost more than £25 but gives a huge peace of mind. Happy to expand for anyone who is interested.

PedroKTFC · October 22, 2023, 11:28pm

Thanks for your comments. I did check the load rating of my ZigBee switch and it’s more than enough for my immersion heater (which, as it happens, is a slightly lower load than our kettle!).

bazcurtis · October 26, 2023, 10:15am

I am sorry if this is something very simple, but I am a little confused. Where do I put the API key?

I am not sure if this is also related to the not having entered the API key

It says I need to install This node requires Node-RED custom integration version 1.1.0+ and I have done that. What have I missed?

I’ll add the below as well just in case it is relevant.

Thanks for the help

Biscuit · October 26, 2023, 10:58am

Your Home Assistant server node (in the config nodes) is not connected and working.

The dotted line around the config node suggests that it is not set up and connected to HA.
The ‘0’ in the used box suggests that it is not actively connected to any of the HA websocket nodes that use this server to connect back to HA.

When I distribute Node-RED flows I use the scrubber

https://zachowj.github.io/node-red-contrib-home-assistant-websocket/scrubber/

to remove all the HA service configuration settings. This is done to prevent my copy of my HA server config node from being exported and imported into your flow. In most cases you will already have an active and working HA server config node, so you don’t want a second (third / fourth …) one.

To get the flow working requires re-connected the HA websocket nodes back to your working Home Assistant config node. AND if you don’t already have a working HA server config node, then that needs creating and setting first.

There is no API key required. The Agile prices can be pulled from Octopus without any API key.
Your Home Assistant server node should not require any authorisation either.

Assuming that you are running Node-RED as an addon, all the necessary bits are included, however do make sure that you:

are running the latest version of the included Node-RED addon (see addons)
have the latest (matching) version of the home assistant websocket nodes (see NR pallet manager)
have the latest (matching) version of the Node RED companion (see integrations)

It is important that each of these three components match. The latest version is not essential, however old version of the NR Companion do not work with the later websocket nodes.

First, check your nodes are correctly set up
Take each websocket sensor node (with the error triangles) and edit them

You will see the ‘entity config’ which is the configuration node behind the flow node. Edit this.

You will see in the ‘ha entity config node’ the ‘server’ entry, which when you import the flow will be blank (see the scrubber). You need to connect this node to the Home Assistant server. If you already have a Home Assistant server config node then it will auto-select this. You just have to accept and save the edits. This will update the config node, then the entity (sensor) node will work.

If you do not have a Home Assistant server node already set up and working (perhaps this is your first Node-RED flow) then you will need to ‘add new server’

NOTE that for Node-RED addons the HA server node manages the internal connection when you select ‘using the Home Assistant Add-on’. If you are using NR outside of HA then you will have to set up your own connection back to HA.

Once you have the Home Assistant server (config) node working the dotted line should go.
Once you have the entity config nodes corrected connected to the HA server node, the red triangles should go.
Then it should all work.

bazcurtis · October 26, 2023, 11:04am

Thanks for that very detailed reply. I appreciate and will try it asap. Sorry, one more very silly question. How do I get back to the Node Red setup page? I have it in the side bar and when I click of it and back on again, I go straight to the Web UI.

I just made a very simple, lights on, lights off flow with notification and that worked a treat.

Biscuit · October 26, 2023, 11:11am

Try settings>addons>node red> open web UI

You should have ‘Node Red’ in your HA left menu list too.

I really should have posted a big WARNING at the top of this project since this is quite an advanced subject and a complicated flow, however I hope you manage to get it all working and can use it successfully.

bazcurtis · October 26, 2023, 12:33pm

Considering the difficulty level, I think you documentation is very good. Obviously people haven’t had the same issues and I have. I can see why you love it. That light on/off flow we really easy to do.

Thanks for taking the time to write this of Octopus Agile