Well hello again. Yes the thread is very dated, and I thought that you had cracked all of this with some neat coding last year
.
Home Assistant is an entity-state machine and spends most of its energy dealing with ‘now’. History is limited, so I think that there are three ways to generate an historical daily-consumption graph [you don’t say over what period, but I assume calendar day total for, say the past 90 days].
- pull historical meter consumption readings by API call from Octopus Account
- record daily consumption yourself and store in a separate database
- pull long-term records of the daily consumption from HA statistics
I have worked on all three approaches, but now like using the long-term stats. API calls to Octopus can return the half-hour consumption/cost values for any period, but getting one year of readings returns over 17,500 records. Dealing with this quantity of data can be interesting.
The LT stats are there. BottleCap Dave’s integration generates sensor(s) for the previous consumption, and this goes into LT stats so it can be retrieved for any period beyond the default 10 days. Yes, Apex-charts will by default plot any entity sensor from history, however this is ‘history’ and not ‘long term statistics’ hence just plotting the previous_accumulative_consumption only works for the past 10 days, and there is also the problem of the ‘date’.
Exhibit A: - Octopus Energy Integration, electric (import) previous_accumulative_consumption, in kWh.
Since I have a Solar PV system, which is currently doing its job very well, my import has been just the system balancing jitter at around 0.5 kWh per day, otherwise this is what I believe you are after. Note that, for the 7th June, the figure changed at 13:19 in the afternoon. This is quite normal, since the value in this sensor is the accumulated (total) consumption (not cost) value for the day, and it comes from the smart meter to Octopus during the night, arriving in my account sometime between 09:00 and 20:00 on the day after. On a good day, I get yesterdays readings by 09:00, but on a bad day not until early evening. Gas is another matter, and not getting a reading at all for two or three days in a row can happen.
Therefore, the figure 0.593 (0.6 kWh) needs to be captured early evening for the 7th June, and plotted on a graph for the 6th June.
Home Assistant can do this, if you are happy to work with SQL calls to the database and process an array of data. For myself, I use Node-RED. So much easier.
I modified something else I already had so as to generate this:-
This is correct, in that the value captured during the evening of the 7th is actually plotted against the 6th. Thank you for asking the question, now I have another nice graph I wanted to add to my collection. It has been a very sunny spring, almost entirely ‘off-grid’ since the start of April…
If you want the code to do this, you need Node-RED with the additional node-red-node-sqlite node set. The following code will
- find the relevant octopus energy accumulative previous consumption meta-id in the statistics-meta table, which you can then add as the setting id in the main flow (you only need to do this once to find the correct id)
- pull 400 records from long-term stats with time and date, at late evening, and group by date
- analyse the returned data array, adding in a timestamp for 12:00 the day before (for back-plotting on the correct date)
- export to an HA sensor as yesterday (value in state) and the entire day-table array in an attribute
There is other stuff in the code to build a year table by calendar month, and to build a billing table by Octopus billing date.
[{"id":"c5569d9c9b38ada0","type":"group","z":"2dd6b0b4a5f86125","name":"Consumption day-graph (for testing only)","style":{"label":true},"nodes":["c523a00720fbfd24","2b1273152bdb234e","f2a7abb264333158","c8ca6fb16faa1ead","03bdc0c644b5d3b2","92607112b89f4b45","343ae2b4d83cb0f5","1bec2fbf0ac54bc1","05a1494f02f9f815","36094bab72c774ae"],"x":34,"y":5819,"w":1072,"h":202},{"id":"c523a00720fbfd24","type":"sqlite","z":"2dd6b0b4a5f86125","g":"c5569d9c9b38ada0","mydb":"56ce7a19f8e148e7","sqlquery":"prepared","sql":"SELECT statistic_id, id FROM \"statistics_meta\"","name":"Long Term stats (name & id)","x":420,"y":5860,"wires":[["2b1273152bdb234e"]]},{"id":"2b1273152bdb234e","type":"change","z":"2dd6b0b4a5f86125","g":"c5569d9c9b38ada0","name":"Filter electricity previous cost sensors","rules":[{"t":"set","p":"test_a","pt":"msg","to":"sensor","tot":"str"},{"t":"set","p":"test_b","pt":"msg","to":"electricity","tot":"str"},{"t":"set","p":"test_c","pt":"msg","to":"previous_accumulative_consumption","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"payload[$contains(statistic_id, $$.test_a) and $contains(statistic_id, $$.test_b) and $contains(statistic_id, $$.test_c)]","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":730,"y":5860,"wires":[["f2a7abb264333158"]]},{"id":"f2a7abb264333158","type":"debug","z":"2dd6b0b4a5f86125","g":"c5569d9c9b38ada0","name":"debug 6","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":990,"y":5860,"wires":[]},{"id":"c8ca6fb16faa1ead","type":"inject","z":"2dd6b0b4a5f86125","g":"c5569d9c9b38ada0","name":"Get long-term stats","props":[{"p":"params","v":"{\"$id1\":117,\"$days\":400}","vt":"json"},{"p":"billday","v":"24","vt":"num"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":170,"y":5940,"wires":[["03bdc0c644b5d3b2"]]},{"id":"03bdc0c644b5d3b2","type":"sqlite","z":"2dd6b0b4a5f86125","g":"c5569d9c9b38ada0","mydb":"56ce7a19f8e148e7","sqlquery":"prepared","sql":"SELECT id, state, udate FROM\n (SELECT metadata_id AS id, state, date(created_ts, \"unixepoch\") AS udate, time(created_ts, \"unixepoch\") AS utime\n FROM \"statistics\"\n WHERE metadata_id IN ($id1) AND unixepoch()-created_ts < 86400 * $days AND utime > \"20\" AND utime < \"23\"\n ORDER BY udate, id, utime DESC)\nGROUP BY id, udate","name":"Long term stats array","x":400,"y":5940,"wires":[["92607112b89f4b45","1bec2fbf0ac54bc1"]]},{"id":"92607112b89f4b45","type":"change","z":"2dd6b0b4a5f86125","g":"c5569d9c9b38ada0","name":"Analyse","rules":[{"t":"set","p":"payload","pt":"msg","to":"(\t/* June 25 - mods to include a day table, backed to yesterday, for day-consumption reporting */\t\t/* count entities recorded - expect one import or two as import and export */ \t $ids:=$distinct(payload.id);\t $imp:=$ids[0]; /* import id */\t $exp:=$ids[1]; /* export id */\t $bill:=billday;\t\t/* add month and year for easy parsing, also midday timestamp for yesterday */\t $base:=payload~>|$|(\t $x:=$split(udate,\"-\");\t {\"month\": $number($x[1]),\t \"day\": $number($x[2]),\t \"year\": $number($x[0]),\t \"timestamp\": (udate & \"T12:00:00\"~>$toMillis())-86400000 })|;\t\t/* get data end date, should be yesterday, and trim base array to one year */\t $last:=($base[id=$ids[0]].udate)[-1];\t $from:=$number($substringBefore($last,\"-\"))-1 & $substring($last,4,6);\t $year:=$base[udate>$from];\t $days:=$count($year)/$count($ids);\t \t/* compose analysis for all entities */\t $results:=$ids.(\t $id:=$;\t $array:=$year[id=$id];\t $year_total:=$sum($array.state)~>$round(2);\t $day_count:=$count($array);\t $start:=$array[day=1][0].udate;\t $month_array:=$array[udate>=$start];\t $table:=[1..12].(\t $m:=$;\t $month:=$month_array[month=$m];\t {\"month\": $m, \"year\": $month[0].year,\t \"timestamp\": $month[0].udate & \"T12:00:00\"~>$toMillis(),\t \"total\": $month.state~>$sum()~>$round(2)};\t );\t {\"id\": $id,\t \"from\": $from,\t \"base\": $month_array[0].udate,\t \"last\": $last,\t \"days\": $day_count,\t \"total\": $year_total,\t \"dtable\": $base,\t \"months\": $table^(timestamp)}\t );\t\t/* block by billing day, summing prior bill day inclusive to next bill day exclusive */\t/* array of billing dates over data period */\t $bdates:=$base[id=$imp and day=$bill].udate;\t $bills:=$bdates[[1..20]]#$mon.(\t $dnext:=$;\t $dfrom:=$bdates[$mon]; /* mon is already from 0 when starting map at 1 */\t $dupto:=$fromMillis($toMillis($dnext & \"T12:00:00.000\")-86400000)~>$substringBefore(\"T\");\t $period:=$base[udate>=$dfrom and udate<$dnext];\t $debit:=$period[id=$imp].state~>$sum()~>$round(2);\t $credit:=$period[id=$exp].state~>$sum()~>$round(2);\t {\"from\": $dfrom,\t \"upto\": $dupto,\t \"kWh\": $debit,\t \"credit\": $credit,\t \"bill\": $round($debit-$credit,2)};\t \t );\t\t {\"import\": $results[id=$imp],\t \"export\": $results[id=$exp],\t \"bills\" : $bills\t };\t\t\t\t)","tot":"jsonata"},{"t":"set","p":"payload","pt":"msg","to":"(\t $l:=$count(payload.import.dtable);\t payload.import.dtable[[$l-90..$l]];\t)","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":640,"y":5940,"wires":[["343ae2b4d83cb0f5","36094bab72c774ae"]]},{"id":"343ae2b4d83cb0f5","type":"debug","z":"2dd6b0b4a5f86125","g":"c5569d9c9b38ada0","name":"debug 7","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":720,"y":5980,"wires":[]},{"id":"1bec2fbf0ac54bc1","type":"debug","z":"2dd6b0b4a5f86125","g":"c5569d9c9b38ada0","name":"debug 8","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":480,"y":5980,"wires":[]},{"id":"05a1494f02f9f815","type":"inject","z":"2dd6b0b4a5f86125","g":"c5569d9c9b38ada0","name":"Find id we want","props":[{"p":"params","v":"{}","vt":"json"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":180,"y":5860,"wires":[["c523a00720fbfd24"]]},{"id":"36094bab72c774ae","type":"ha-sensor","z":"2dd6b0b4a5f86125","g":"c5569d9c9b38ada0","name":"Oct Import Elec Daily Use Table","entityConfig":"1feb8331337d9514","version":0,"state":"payload[-1].state","stateType":"jsonata","attributes":[{"property":"table","value":"payload","valueType":"msg"}],"inputOverride":"allow","outputProperties":[],"x":950,"y":5940,"wires":[[]],"server":""},{"id":"56ce7a19f8e148e7","type":"sqlitedb","db":"/homeassistant/home-assistant_v2.db","mode":"RO"},{"id":"1feb8331337d9514","type":"ha-entity-config","server":"","deviceConfig":"","name":"SC Oct ImE Daily","version":6,"entityType":"sensor","haConfig":[{"property":"name","value":"Octopus Import Electricity Daily Consumption"},{"property":"icon","value":""},{"property":"entity_picture","value":""},{"property":"entity_category","value":""},{"property":"device_class","value":""},{"property":"unit_of_measurement","value":""},{"property":"state_class","value":""}],"resend":false,"debugEnabled":false}]
For this exercise I have curtailed the 400 output array to just the last 90 days, and set a usual Apex chart with data generator to pull from the table and plot the last three months.
type: custom:apexcharts-card
header:
show: true
title: Octopus Electricity Import (kWh) - Daily 3 months
show:
last_updated: true
graph_span: 3 months
span:
start: day
offset: "-3month"
series:
- entity: sensor.octopus_import_electricity_daily_consumption
data_generator: |
return entity.attributes.table.map((reading) => {
return [new Date(reading.timestamp).getTime(), reading.state];
});
type: column
color: Red
unit: kWh
I will leave you to experiment and provide a better version of my code in due course.
NOTE: I suggest leaving the sqlite node setting as ‘prepared statement’ and the configuration server to Read Only to prevent any accidental writes to the HA database!