Reduce recorded amount of data of Shelly devices

Configuration
1

Hello,

over the time I installed quite a few Shelly devices with some of them for recording the energy consumption. Those devices seem to record data with an very high rate, which caused a significant growth of my database within one month (compressed backup size grow from 1.7 GB to 2.1 GB within one month).

I found out that I could reduce the time between captures with a custom sensor and the filter, like shown here.

The problem is that I have some devices that might turn on for only 10 seconds - 60 seconds and than stay turned of for hours. So their energy consumption might not be recorded when reducing the update frequency. On the other hand, updating such a device every 5 seconds will cause a lot of useless data.

So I was looking for something like you could do with the filter in ESPHome and combine a throttle and delta filter. This will e.g. record only one data point every hour, except when the change in data is larger than 1 (watt).

filters:
  - or:
    - throttle: "1h"
    - delta: 1

Is there any way to achieve this with an Home Assistant filter as well?

2

Data is data.
You would have to slow the data from coming in or turn the recorder off completely and collect none of it.
So you have to tell the device to not generate the data.

3

Well no, it seems like you can do this. Of course the device will still generate the data, but you can exclude it from the Home Assistant recorder by e.g. only including entities that shall be recorded. All others will not be recorded.

Then you need to setup one template sensor like this:

template:
	- trigger:
	  # one record every 10 minutes
	  - platform: time_pattern
		minutes: "/10"

	  # additional one record on a big enough change of the sensor value of sensor.shelly_power
	  - platform: state
		entity_id: sensor.shelly_power

	  sensor:
	  - name: "Power filter"
		unique_id: power_filter
		unit_of_measurement: "W"        # unit of measurement
		device_class: power
		state: >
		  {% set source = 'sensor.shelly_power' %}

		  {% if trigger.platform == 'state'
				and trigger.from_state is not none
				and trigger.to_state is not none %}

			{% set old = trigger.from_state.state | float(0) %}
			{% set new = trigger.to_state.state | float(0) %}
			{% set delta = (new - old) | abs %}

			{# Only record when change of data was >= 5 #}
			{% if delta >= 5 %}
			  {{ new | round(0) }}
			{% else %}
			  {{ this.state if this.state is defined else new }}
			{% endif %}

		  {# When the trigger was the time_pattern, record the current data #}
		  {% else %}
			{{ states(source) }}
		  {% endif %}
1 Like
4

Thanks for posting your solution!

5

I don’t think this is going to have any effect in long term, because long term statistics are already aggregated and only data per 15 minutes periods is saved. So the only effect is for the data from the last 10 days (by default).

Not saying your solution is wrong, just adding this for context. I think for most people, if storage space is not critical, this is not necessary.

2 Likes
6

Is there any way to change this 15 minute period for the historical data? 15 minutes might be even too much for most sensors. But what about smaller spikes that only last less 5 - 10 minutes? It would be nice if those would be kept in the historical data.

Storage on my HA server is not critical in my case, but the backup also grew pretty much and that is where I’m limited.

At this point I’m not sure if the data polling of the Shellys is really the problem. That’s why I opened another thread in the configuration-forum.

7

I don’t think this can be manipulated, or at least not easily.

By the way I found that I was wrong, long term stats are aggregated in 1 hour periods, not 15 minutes ones.

They do store max, min and average (for measurement type of data, like most sensors), so small peaks are not completely gone.

Documentation of the this in more detail is here:

8

So I was looking for something like you could do with the filter in ESPHome and combine a throttle and delta filter.

I opened a feature request to add a HA Delta filter a few months ago for the same reason (excessive verbosity of Shelly power reports), in case you want to give it an upvote (and wouldn’t have done yet).

I also ended up using a template sensor to achieve the same purpose, but believe it would be a great addition to have this natively in HA.