Data filtering by extending `recorder` config options

Feature Requests
1

Hi everyone,
our growing amount of sensors are producing huge amounts of data at very high frequency. This frequency can often times not be adjusted, meaning that sensors update every second or every few seconds without pushing any relevant information.

I personally have been filtering some of my sensors by publishing to a dedicated MQTT topic, subscribing to that using Node-Red, filtering the data there and then publishing the filtered data to an MQTT topic that Home Assistant is subscribed to.

However, that is only possible if the sensors are running a custom firmware such as Tasmota. Even ESPHome does not allow filtering when you use it as a Bluetooth Proxy (luckily Tasmota does).

Would it not be possible to add an interface that would allow processing of the data Home Assistant is about to write to the database?

Right now, Home Assistant already allows selecting which entities to write to the database by adjusting the recorder config ( Recorder - Home Assistant (home-assistant.io) ). Could this maybe be extended to allow templating or simple filterting?

A good starting point would be something simple like the ESPHome sensor delta filtering ( Sensor Component — ESPHome ).

Would it be possible to add this to the Home Assistant recorder?

Thank you for your consideration.
Alex

Possible config approach:

recorder:
  include:
    entities:
      - sensor.abc_temperature*
        filters:
         - delta: 2.0  #this would only write to database if temperature changes by +-2.0 °C
      - sensor.abc_power*
        filters:
         - delta: 5%  #this would only write to database if power consumption changes by +-5%
      - sensor.abc_something_else*
        filters:
          - or:
            - throttle: 10min     #sensor data will be written to database either if delta is 5 OR if the last write was more than 10 min ago
            - delta: 5.0
2

Poor man’s approach:

#  To throttle a sensor down to 1 value per minute, the window_size should be set to "00:01”.
- platform: filter
  name: "filtered smlreadertotalenergyconsumption"
  entity_id: sensor.smlreadertotalenergyconsumption
  filters:
    - filter: time_throttle
      window_size: "00:01"

Note that sensor.smlreadertotalenergyconsumption is the original sensor. In my record config, I’ve added sensor.filtered_smlreadertotalenergyconsumption.

2 Likes
3

Nice idea :slight_smile: , but unfortunately it is just another big workaround like the tasmota+node-red approach I am using, for something that would really be beneficial if it had a real implementation into recorder.

With your suggested approach you quickly create hundreds of fake sensors, just for a bit of filtering (because each hardware sensor will have multiple entities and each entity will need its own filter).

So the idea is very good if only one or two sensors are affected, but in our growing home automation environments the smartest approach would be a modification of the recorderto allow such additional filtering, IMHO.