Energy metering and load balancing

Hi there, I have now reached a point of “almost finished” with my energy meter and energy load balancing project and it’s time to share my findings and configuration. Despite this being apparently a simple configuration, there were a number of challenges along the way and this article shows how I solved those. For example my home-assistant became unresponsive at one stage just by adding a new sensor.

In this project thread (there will be more than one post over a few days) I will tell:

  • How to reduce the impact of real time sensors on HA performance
  • How to record adequate history data for plotting with low impact on performance
  • How I plotted the energy readings with the versatile mini-graph-card
  • How I implemented some rudimentary load balancing rules to reduce my electricity usage and reduce energy peaks.
  • Some miscellaneous tips relating to smart control and IoT.

My energy meter can show energy consumption in bar graphs. In the example case below at 3 different scales in addition to the real time readings.

I addition I have some figures showing which devices are using energy in order to help me understand better how to load-balance automatically.

image

The Sensor

I am using the “pulse” sensor for our smart meter in Norway. The Pulse sensor is available from Tibber and there is a home assistant integration available for this. https://www.home-assistant.io/integrations/tibber

The device gives me access to real time current and daily usage statistics. It’s not necessary to purchase your electricity from them just to use the device, and they provide enough access to the API without a subscription to make this work.

I have 2 locations with smart meters that I am monitoring, so I have 2 Tibber pulse meters integrated:

image

The Tibber integration required me to also contact my electricity network provider so that the HAN port could be enabled on my smart meter. That was a nightmare in itself but was solved eventually!

The Tibber integration now provides me with 2 new entities sensor.real_time_consumption_xx1 and sensor.real_time_consumption_xx2 (names here have been anonymized). Inside each sensor there are a bunch of attributes that look like this:

Note the flat line there on the history graph! This is because I have turned off history for these sensors. The sensor and all of it’s attributes are updating every few seconds. This fills up the history database very quickly and leads to a number of problems. Some of these are:

  1. Large .db file in the config folder after a few days. Mine got up to over 4GB before I took action.
  2. Unresponsive Raspberry PI
  3. Very long response or no response at all when retrieving history data
  4. Unuseable user interface (for these plots) when using Nabu Casa Home Assistant Cloud

One of the first things that you can do to prevent future problems is to disable storing the real-time data in the recorder. Also I only need history data for 7 days, and in this configuration we change the recorder to update every 2 minutes to avoid too much traffic. These settings should be placed in configuration.xml:


recorder:
  commit_interval: 120
  purge_keep_days: 7
  auto_purge: true
  exclude:
    entities:
     - sensor.real_time_consumption_xx1
     - sensor.real_time_consumption_xx2

If you’re reading this and you already have an unresponsive or slow home assistant due to a large database full of real time data then changing these settings will not immediately help. You will either have to wait a week or so for the data to thin out, or if you want to just start from the beginning then delete your .db file in config and restart home assistant.

Before I made the above changes I monitored the user interface with the chrome web tools and my home assistant web interface was having to retrieve 1.8MB of data in order to show the simple bar chart with energy consumption for the last 7 days, whereas now its only fetching a small number of kB. 1.8MB is not huge but the server processing and subsequent data transmission is quite slow. I haven’t looked at the code but I have some ideas how that could be optimised using my knowledge of HPC, but that will have to be for the future.

I could barely show the graph using my Home Assistant Cloud address, and it would take several minutes if I used a direct IP address to my installation.

The problem now of course is that now we have no history data to plot. We could just create new template sensors from the real_time_consumption sensors that are not excluded from the recorder, but we would still be stuck with sensors updating every 1 or 2 seconds filling up our database.

Links to documentation for configurations used in this post:

I will tell how I solved that in the next post.

8 Likes

Template Sensors

In the previous section Energy metering and load balancing I explained how I have set up my Home Assistant to receive real time updates from my smart energy meters.

The tibber sensor gives me access to a number of attributes. The ones that are of most interest are the real time current (I use this to plot the energy used for the previous hour, and no more), and the accumulated consumption. The tibber integration gives us the accumulated consumption on a daily basis, it resets to zero at midnight every day. From this attribute we can get daily and hourly consumption figures.

The utility_meter integration gives us the possibility to extract daily energy and hourly energy usage from the accumulated consumption. We want to avoid though getting continuous updates every second or 2 that we would automatically get if we just do this in our configuration.xml:

Don’t do this

sensor:
  - platform: template
    sensors:
      accumulated_consumption_xx1:
        friendly_name: Daily Accumulated Consumption
        value_template: "{{ (state_attr('sensor.real_time_consumption_xx1','accumulatedConsumption') |float * 1000)|round(2)}}"
        unit_of_measurement: 'Wh'
 
utility_meter:
  hourly_energy:
    source: sensor.accumulated_consumption_xx1
    cycle: hourly
  daily_energy:
    source: sensor.accumulated_consumption_xx1
    cycle: daily

Retrieving this daily_energy value is challenging since it is updated every few seconds with the latest value, recorded in the database, and results in a large amount of data that we just don’t need. Recorder doesn’t have any settings to restrict what is stored for individual sensors, it’s just off or on.

A better solution is to filter the sensor using the filter platform to throttle the sensor data so that we drastically reduce the number of updates:

Do this

sensor:
  - platform: template
    sensors:
      accumulated_consumption_xx1:
        friendly_name: Daily Accumulated Consumption
        value_template: "{{ (state_attr('sensor.real_time_consumption_xx1','accumulatedConsumption') |float * 1000)|round(2)}}"
        unit_of_measurement: 'Wh'
        availability_template: >
                {% if is_state('sensor.real_time_consumption_xx1', 'unavailable') %}
                  false
                {% else %}
                  true
                {%- endif %}

 

  - platform: filter
    name: "filtered accumulated consumption xx1"
    entity_id: sensor.accumulated_consumption_xx1
    filters:
      - filter: time_throttle
        window_size: "00:01"

utility_meter:
  hourly_energy:
    source: sensor.filtered_accumulated_consumption_xx1
    cycle: hourly
  daily_energy:
    source: sensor.filtered_accumulated_consumption_xx1
    cycle: daily

Now our utility_meter integration will only be updated every minute, instead of every second or 2. In addition we will only be recording the attribute that we need and ignore the ones we don’t. We could obviously change the window_size and only update the utility_meter every 5 minutes or some other value, but we would then have greater inaccuracies at the end of the day (we would miss the last 5 minutes of data).

By using the time_throttle filter, in my case the database was reduced in size to a third of what I was getting. In addition the amount of data that needs to be retrieved to show daily energy use is reduced to less than a tenth.

Links to documentation for the integrations used on this page:

Edit: I recently realized that the filter template sensors can get “out of whack” during a restart. The solution is to add an availability_template as described here: https://github.com/home-assistant/core/issues/38710 so that the utility meter does not try to read the template values before they are available. My example above has been changed to reflect this.

In the next post I will tell how to use a template sensor to provide the utility_meter data that we will need for plotting the bar graphs.

5 Likes

So far in this project description I have presented how to set up the tibber integration with templates and filters so that the important attributes are available for plotting. Now I will look at one way to visualise this data for hourly, daily and weekly energy usage.

Lovelace Mini Graph Card

The mini-graph-card (https://github.com/kalkih/mini-graph-card) is the solution that I have chosen for plotting the energy use data. I chose this because it has the possibility to display data as a bar chart, which looks better than using a history line graph. The mini-graph-card is not currently available by default with Home Assistant so if you want to use this then follow the installation instructions given on the page linked to in this post.

I have chosen to display the following figures (you can see an example in the first post of this thread):

  • consumption every minute for the last hour with real time consumption
  • Energy use every hour for the last 24 hours
  • Energy use every day for the last 7 days

Its also possible to add some colour to show high or low levels of energy use. I have chosen to display the mini-graph-card in a vertical stack so the code looks like this:

  - title: Power all
    path: power-all
    badges: []
    cards:
      - type: vertical-stack
        cards:
          - type: 'custom:mini-graph-card'
            entities:
              - entity: sensor.real_time_consumption_xx1
            name: Energy Consumption last hour
            hours_to_show: 1
            points_per_hour: 60
            bar_spacing: 1
            aggregate_func: max
            group_by: interval
            update_interval: 60
            lower_bound: 0
            hour24: true
            color_thresholds:
              - value: 0
                color: '#00FF00'
              - value: 4000
                color: '#FF8700'
              - value: 6000
                color: '#FF0000'
            show:
              graph: bar
          - type: 'custom:mini-graph-card'
            entities:
              - entity: sensor.energy_hour_xx1
            name: Hourly energy consumption last 24h
            hours_to_show: 24
            aggregate_func: last
            group_by: hour
            show:
              graph: bar
            lower_bound: 0
            hour24: true
            color_thresholds:
              - value: 0
                color: '#00FF00'
              - value: 4000
                color: '#FF8700'
              - value: 6000a
                color: '#FF0000'
          - type: 'custom:mini-graph-card'
            entities:
              - entity: sensor.energy_day_xx1
            name: Daily energy consumption last 7 days
            hours_to_show: 168
            aggregate_func: last
            group_by: date
            points_per_hour: 1
            show:
              graph: bar
            lower_bound: 0
            hour24: true
            color_thresholds:
              - value: 0
                color: '#00FF00'
              - value: 40000
                color: '#FF8700'
              - value: 80000
                color: '#FF0000'

You will need to edit your lovelace dashboard and paste in this configuration for a tab in the dashboard. Alternatively you can use the interactive dashboard editor to add the vertical stack and the mini-graph-cards. If you put it all together you should get something that looks similar to this:

Links to documentation for the integrations used on this page:

4 Likes

Load balancing

I have had a simple monitoring of energy use for some time now (several years infact) using the smappee integration but I was garnered in to action to come up with a better solution due to an unusually cold winter. January saw many of us with very high bills for electricity, usually a cheap commodity here where I live in Norway. Mine was at least double what it normally is and friends had double that again! In addition, due to a fault with our main fuse combined with low temperatures (ours was down to -10C) and high usage, our main fuse was tripping regularly and at its worst was tripping 3 times a day - quite hard to live with in lockdown and home office!

To mitigate the problems with the fuse tripping all the time while we waited for an electrician to come and replace the faulty part, I implemented a simple load balancing using the real time current measurements - first from smappee and later replacing this with the tibber integration.

The first thing I did was to go around turning everything electrical off and then using smappee to map out which appliances were using the most current. Smappee tries to do this using some artificial intelligence solution but I have never got that to work very well. I did a survey of most of the big energy users (eg heating, cooking, car charging, and hot water) and wrote down in a spread sheet which circuit they were on and which 2 phases (like most Norwegian houses we have a 3 phase supply) were being loaded. This helped to prioritize what to turn off if the power use crept up too high.

Our heating is all electricity, with the addition of a wood burning stove, so when the electricity use is high and the fuse is in danger of tripping, the only solution is to start turning things off.

To control the heating in our house I have a combination of:

  • 433.92Mhz receivers, either wired in or plug in, for electric panel heaters
  • zwave thermostats for floor heating
  • Additional temperature measurement from Netatmo (Netatmo integration for HA)

I have a number of the Heatit thermostats such as this one (and some of their older models):

https://products.z-wavealliance.org/products/3802?selectedFrequencyId=1

The receivers can be switched from Home Assistant (I am using the Telldus Live integration for that), and the thermostats can be adjusted from Home Assistant automations (I am using the Samsung SmartThings integration for that). I could also use Telldus Live to adjust the thermostats but the integration doesn’t have support for climate entities so the whole thing is a faff to get it working nicely as I needed a whole bunch of event triggers in Telldus Live to to the switching.

To do the load balancing I have set up a few scenes in Home Assistant. Note these are heating scenes, not lighting scenes. These scenes reflect what I want to set in high and low energy consumption scenarios:

Extract from my scenes.yml (note I have shortened the file, this is just for illustration)

- id: '1610369717606'
  name: Cozy comfort
  entities:
    climate.xx1_kitchen_thermostat:
...
      temperature: 19
      friendly_name: XX1 kitchen thermostat
    climate.xx1_livingroom:
...
      temperature: 21
      friendly_name: XX1 livingroom

    switch.xx1_downstairs_hall_heater:
      friendly_name: XX1 Downstairs Hall Heater
      assumed_state: true
      state: 'on'
- id: '1610369802789'
  name: Comfort
  entities:
    climate.xx1_kitchen_thermostat:
...
      temperature: 17

    climate.xx1_livingroom:
...
      temperature: 20


- id: '1610369867367'
  name: Background
  entities:
    climate.xx1_kitchen_thermostat:
...
      temperature: 17

    climate.xx1_livingroom:
...
      temperature: 17

    switch.xx1_downstairs_hall_heater:
      assumed_state: true
      state: 'off'
- id: '1610370588171'
  name: Bathroom comfort
  entities:
    climate.xx1_downstairs_bathroom:
...
      temperature: 21
.
    climate.xx1_upstairs_bathroom:
...
      temperature: 21
.
- id: '1610370643067'
  name: Bathroom background
  entities:
    climate.xx1_upstairs_bathroom:
...
      temperature: 19

    climate.xx1_downstairs_bathroom:
...
      temperature: 19

- id: '1610370680604'
  name: Middle bathroom background
  entities:
    climate.xx1_middle_bathroom:
...
      temperature: 19

As you see we now have some scenes for “Comfort” and “background” for different rooms. The main principle of the load balancing (primarily aimed at reducing the energy use peaks) is to switch from Comfort to background when the energy use is too high.

In addition there are a number of automations that control this. First of all some automations to set an energy use state. For example in automations.yml I set 2 different high usage levels and 2 different low usage levels:

- id: '1612438851388'
  alias: Load balance activate power_high_1
  description: Emergency off
  trigger:
  - platform: numeric_state
    entity_id: sensor.real_time_consumption_xx1
    above: '5000'
  condition: []
  action:
  - service: input_boolean.turn_on
    data: {}
    entity_id: input_boolean.power_high_1
  - service: input_boolean.turn_off
    data: {}
    entity_id: input_boolean.power_low_1
  - service: input_boolean.turn_off
    data: {}
    entity_id: input_boolean.power_low_2
  mode: single
- id: '1612438911499'
  alias: Load balance activate power_high_2
  description: Emergency off
  trigger:
  - platform: numeric_state
    entity_id: sensor.real_time_consumption_xx1
    above: '8000'
  condition: []
  action:
  - service: input_boolean.turn_on
    data: {}
    entity_id: input_boolean.power_high_2
  - service: input_boolean.turn_off
    data: {}
    entity_id: input_boolean.power_low_1
  - service: input_boolean.turn_off
    data: {}
    entity_id: input_boolean.power_low_2
  mode: single
- id: '1612439134922'
  alias: 'Load balance activate power_low_1 '
  description: Power levels low
  trigger:
  - platform: numeric_state
    entity_id: sensor.real_time_consumption_xx1
    below: '5000'
    for: 00:01:00
  condition: []
  action:
  - service: input_boolean.turn_off
    data: {}
    entity_id: input_boolean.power_high_1
  - service: input_boolean.turn_off
    data: {}
    entity_id: input_boolean.power_high_2
  - service: input_boolean.turn_on
    data: {}
    entity_id: input_boolean.power_low_1
  mode: single
- id: '1612439181945'
  alias: Load balance activate power_low_2
  description: Power levels low
  trigger:
  - platform: numeric_state
    entity_id: sensor.real_time_consumption_xx1
    below: '2000'
    for: 00:05:00
  condition: []
  action:
  - service: input_boolean.turn_off
    data: {}
    entity_id: input_boolean.power_high_1
  - service: input_boolean.turn_off
    data: {}
    entity_id: input_boolean.power_high_2
  - service: input_boolean.turn_on
    data: {}
    entity_id: input_boolean.power_low_2

There are also some input_booleans that you can see above, but I wont show you how to configure those (just read the documentation).

So for a really low energy consumption to count it has to have been in that state for 5 minutes - then we can risk turning things back on again. My main fuse was rated at 13KW but we couldn’t let the energy use get that high and risk another blackout so there are 2 thresholds at 5KW and 8KW where we start turning things off. We had seen the fuse trip at a use level as low as 7KW if it stayed at that level for a few minutes. These automations don’t do any actual switching, they just set some states that are used below:

In this load balance automation, I never let 2 panel heaters at 2KW each be on at the same time. The 2KW living room heater is on a zwave room thermostat and we just let that run, but if the heater comes on, we turn off another in a different part of the house (yes it did get a bit cold for a while!).

- id: '1612179757384'
  alias: 'Load balance #1 livingroom - downstairs'
  description: ''
  trigger:
  - platform: device
    type: turned_on
    entity_id: switch.xx1_downstairs_hall_heater
    domain: switch
    for:
      hours: 0
      minutes: 10
      seconds: 0
      milliseconds: 0
  - platform: device
    type: turned_on
    entity_id: switch.xx1_livingroom_heater
    domain: switch
  condition: []
  action:
  - type: turn_off
    entity_id: switch.xx1_downstairs_hall_heater
    domain: switch
  mode: single

In this example we switch off as much as we can when the power use hits a peak:

- id: '1612182803892'
  alias: 'Load balance #3 Emergency off'
  description: Emergency off
  trigger:
  - platform: state
    entity_id: input_boolean.power_high_2
    from: 'off'
    to: 'on'
  condition: []
  action:
  - type: turn_off
    entity_id: switch.xx1_hot_water
    domain: switch
  - type: turn_off
    entity_id: switch.xx1_downstairs_hall_heater
    domain: switch
  - service: climate.set_temperature
    data:
      entity_id: climate.xx1_kitchen_thermostat
      temperature: 10
  - service: input_boolean.turn_off
    data: {}
    entity_id: input_boolean.kitchen_state
  mode: single

There are also some automations to turn stuff back on again when the high use period is over.
For example the automation that was already in place based on time triggers can also be conditionally triggered when the energy use drops to acceptable levels:

- id: '1610370458895'
  alias: Activate comfort
  description: moderate comfort level starts
  trigger:
  - platform: time
    at: 06:01:00
  - platform: time
    at: '17:00:00'
  - platform: state
    entity_id: input_boolean.power_high_2
    to: 'off'
    from: 'on'
  condition:
  - condition: time
    after: 06:00:00
    before: '23:55:00'
  action:
  - scene: scene.comfort
  mode: single

A typical case might be that the living room is set to be warm and cosy, but then it’s time to make dinner and the oven and cooker come on. The automations instantly register this and turn down the thermostat in the living room and turn off anything that can be switched directly.

Whilst this was a situation I never want to be in again, the automations did work and we never had another blackout in the few very cold days we had to wait for some maintenance to be carried out. The house was in general a little colder than normal and more unevenly heated. However I liked the load balance idea so much that we still have it running, this helps us save a little bit of energy, but it also reduces peaks so that there is less load on the system and less load on the grid!

This is a list of some of the documentation for the configurations in this post:

4 Likes

GREAT stuff!

One add, if anyone don’t want to go the Tibber route and get their energy company to open the HAN port (if they even will do that), it’s doable for anyone to achieve the same thing with an ESP32 and a Photoresistor in less than an hour:

This tutorial was really helpful for minimizing the amount of data I’m recording.

However, I’m not clear how you are showing the past hour consumption with sensor.real_time_consumption.xx1 if you are not recording history data for that sensor?

The way I understood is that the real-time data is excluded from the recorder while the accumulated consumption (with window of 1 minute) is still being record.
These datapoints are used for the utilitymeter.

Just reworked my config and added the filtered entities where I use the utility_meter.
Excluding all energy entities that where replaced by a filter or I didn’t use the history off.
What an improvement!!

I have a smart meter (DSMR) at home, connected with an esp8266 sending MQTT messages where i only send every 15s (instead of every second).
I still have recorder history of the actual power consumption & actual power delivery (every 15s).

template:
  - sensor:
      - name: Total power usage
        unit_of_measurement: 'kWh'
        state: >
          {% set low = states('sensor.p1_consumption_low_tariff') | float %}
          {% set high = states('sensor.p1_consumption_high_tariff') | float %}
          {{ (low + high) | round(3) }}
        availability: >
                {% if is_state('sensor.p1_consumption_low_tariff', '0') or is_state('sensor.p1_consumption_high_tariff', '0') %} 
                  false
                {% else %}
                  true
                {%- endif %}
        
      - name: Total power production
        unit_of_measurement: 'kWh'
        state: >
          {% set low = states('sensor.p1_return_delivery_low_tariff') | float %}
          {% set high = states('sensor.p1_return_delivery_high_tariff') | float %}
          {{ (low + high) | round(3) }}
        availability: >
                {% if is_state('sensor.p1_return_delivery_low_tariff', '0') or is_state('sensor.p1_return_delivery_high_tariff', '0') %} 
                  false
                {% else %}
                  true
                {%- endif %}

Where I then applied the utilitymeters to filtered entities:

sensor:
- platform: filter
    name: "filtered total power usage"
    entity_id: sensor.total_power_usage
    filters:
      - filter: time_throttle
        window_size: "00:02"
        
  - platform: filter
    name: "filtered total power production"
    entity_id: sensor.total_power_production
    filters:
      - filter: time_throttle
        window_size: "00:02"

My dashboards are fast again! Thanks @jeremyc for the filter suggestions!

The tibber integration seems to have changed since you wrote this topic @jeremyc ?
I get a separate sensor enity for each value from the api, not one sensor with several attributes.

I would like to follow your example and exclude all theese realtime sensors from filling up the database. I guess I just keep adding them under the exclude: line in the recorder: paragraph?

recorder:
  commit_interval: 120
  purge_keep_days: 7
  auto_purge: true
  exclude:
    entities:
     - sensor.real_time_consumption_xx1
     - sensor.power_myaddress
     - sensor.accumulated_consumption_myaddress
     - sensor.accumulated_consumption_current_hour_myaddress
     

Will i need to do this for all the other sensors from Tibber? Like the L1, L2, L3 voltage, the wifi signal sensor and so on?

Also, how do i rewrite the filtering & templating part, and the utility_meter integrations when theres no attributes, but separate sensors?

This line in particular:

 value_template: "{{ (state_attr('sensor.real_time_consumption_xx1','accumulatedConsumption') |float * 1000)|round(2)}}"
        unit_of_measurement: 'Wh'

I would like kWh, not Wh, as im using alot of power here…

I use Sonoff S31 and I only have watts, I don’t have a ‘accumulatedConsumption’
Is there a way I can get a bar chart with daily kWh usage?

I use DIN devices from Tongou. Black one are the one with a meetering option.
Screenshot_20230312_193853
https://www.aliexpress.com/store/1100052?spm=a2g0o.order_list.order_list_main.26.21ef5e5bZx14xa
They have models up to 63A. Works fine.
The current is reported in mA but HA expect A. I correct it with a simple template in order for the Energy dashbord to work.
Using that base I have implemented a load balancing facility in Python which works fine.
GitHub - dominig/pyscript_power_load_balance: Pyscript for Home Automation enabling a power dispatch via power balancing

  - sensor:
    - name: enedis_total
      unit_of_measurement: "kWh"
      device_class: energy  
      state_class: total_increasing
      state: >-
        {{ (states('sensor.meter_general_summation_delivered') | int / 100) }}
    - name: enedis_amp
      unit_of_measurement: "A"
      device_class: energy  
      state_class: measurement
      state: >-
        {{ (states('sensor.meter_general_rms_current') | int / 1000) }}

1 Like

For anyone reading this thread, please not that most of what I have described in my project has been superseded by other work.

Check out the Home Assistant energy dashboard, and the Node-red “power saver” https://powersaver.no/ for some better solutions. I have been using these instead of my earlier solution.