Recorder include and exclude on area-based label

Recorder - Home Assistant (home-assistant.io)

Like to include or exclude sensor based on area or label that is just added to HA.

Excluding by label (to avoid listing them all out in configuration. yaml) would make it so much easier to discard data you know you will never want to see again other than in real time. I know my DB would be a lot smaller if I had this ability.

On top of excluding, it would also be nice if one could set a shorter retention period for entities based on a label.

1 Like

There are so many popular threads about database activity and size, that the inclusion, exclusion process should really be made more user-friendly and accessible.

However, whilst I wait for this long-overdue feature to be implemented…

I have recently implemented the basics of getting this done directly in HA. I am currently just needing to write some automations, but the framework is in place.

Here is an example…

I have a macro that returns an entity_id list of all entities matching any of the labels I provide

This macro lives in /config/custom_templates/labels.jinja

{#-
# Returns: a comma separated list of *ANY* entities matching any of the labels provided in label_list
# Usage: {{label_entities_any(['Temperature-Climate','Humidity-Climate']).split(",")}}
-#}
{%- macro label_entities_any(label_list) -%}
    {{- label_list | map('label_entities') | sum(start=[]) | sort | join(",") -}}
{%- endmacro -%}

I have another macro that uses this first macro.

It returns me a list of every entity_id I want to not record.

You could just use a single label ‘No-Recorder’ and tag everything you want excluded.

{%- macro recorder_excluded_entities() -%}
    {%- set excluded = [
        'Weather-Wind-Speed',
        'Weather-Wind-Gust',
        'Weather-Wind-Gust-Max',
        'Weather-Air-Quality',
        'Weather-Temperature-Feels-Like',
        'Weather-Temperature',
        'Weather-Humidity',
        'Weather-Dew-Point',
        'Weather-Rain-Hourly',
        'Weather-Rain-Daily',
        'Weather-Rain-Weekly',
        'Weather-Rain-Monthly',
        'Weather-Rain-Yearly',
        'Weather-Relative-Pressure',
        'Weather-UV-Index',
    ] -%}
    {%- set entities = label_entities_any(excluded)
        .split(",")
        | list
        | sort
        | unique -%}
    {{- entities| join(",") -}}
{%- endmacro -%}

I have a YAML package (recorder_exclude.yaml) that takes that list of entities and writes it out to a YAML file (which is read in by my configuration).

So for changes to take effect…

  • you must first run the script (I have an automation that does this every 6 hours and on shutdown)
  • you need to restart HA
# ############################################################
# Notify file service - enables writing to a file (it appends)
# ############################################################
notify:
- platform: file
  name: file_log_weather_recorder_test
  filename: /config/packages/climate/weather_recorder_test.yaml
  timestamp: false

# ############################################################
# Shell-Commands to manage the recorder configuration file
# ############################################################
shell_command:
  # Remove the old file 
  recorder_exclude_remove: rm /config/db1138/climate/weather_recorder_test.yaml
  # Create the replacement file
  recorder_exclude_create: touch /config/db1138/climate/weather_recorder_test.yaml
  # Remove the first two lines from the file 'Because STUPID' @see below
  recorder_exclude_clean: sed -i '1,2d' /config/packages/climate/weather_recorder_test.yaml

# ############################################################
# Script that creates the YAML file
# ############################################################
script:
  write_recorder_exclude:
    alias: write_recorder_exclude
    mode: single

    variables:
      recorder_entities: >
        {%- from "labels.jinja" import recorder_excluded_entities -%}
        {%- set recorder_entities = recorder_excluded_entities().split(",") -%}
        {{- recorder_entities -}}
      recorder_output: >
        {%- set e = "" -%}
        {%- set sep = "\n    - " -%}
        {%- if recorder_entities | count -%}
            {%- set e = sep + recorder_entities | join(sep) -%}
        {%- endif -%}        
        {%- set rv = [
          '',
          '# Generated by "script.write_recorder_exclude"',
          'recorder:',
          '  exclude:',
          '    entities:' + e,
        ] -%}
        {{- rv | join("\n") -}}

    sequence:
    - if:
      - condition: template
        value_template: "{{- (recorder_entities | count) > 0 -}}"
      then:
      - service: shell_command.recorder_exclude_remove
        data: { }
        alias: 'Clears the contents of the recorder file'
      - delay: "00:00:01"
      - service: shell_command.recorder_exclude_create
        data: { }
        alias: 'Creates a new empty recorder file'
      - delay: "00:00:01"
      - service: notify.file_log_weather_recorder_test
        data:
          message: |
            {{- recorder_output -}}
        alias: 'Write the contents to the recorder file'
      - delay: "00:00:01"
      # Home Assistant notifications (Log started: 2024-05-26T07:53:15.380423+00:00)
      # --------------------------------------------------------------------------------
      - service: shell_command.recorder_exclude_clean
        data: { }
        alias: 'Removes the STUPID first two lines that get added to all notify.file files'


1 Like

You can modify the macros and script above to collect all entities with a ‘Record-24-Hours’ label and then run the recorder.purge_entities service with the results.

Manual:

service: recorder.purge_entities
data:
  keep_days: 1
  entity_id:
    - sensor.weather_daily_rain
    - sensorweather_dew_point
    - sensor.weather_feels_like

Automated: (make sure to add a condition to check you have some entities in recorder_excluded_entities() before calling the service)

service: recorder.purge_entities
data:
  keep_days: 1
  entity_id: >
        {%- from "labels.jinja" import recorder_excluded_entities -%}
        {%- set recorder_entities = recorder_excluded_entities().split(",") -%}
        {%- set purge_entities = "" -%}
        {%- set sep = "\n    - " -%}
        {%- if recorder_entities | count -%}
            {%- set purge_entities = sep + recorder_entities | join(sep) -%}
        {%- endif -%}       
        {{- e -}}
1 Like

Hello @teskanoo

I had added the label purge to all the entities that I don’t want to keep so I thought why not use the following approach:

{% set list = label_entities("purge")%}
{% set todo = "- " + list | join("\n- ")%}
{{-todo-}}

the above returns:

- sensor.adguard_home_average_processing_speed
- sensor.adguard_home_dns_queries

so running the action:

action: recorder.purge_entities
data:
  keep_days: 5
  entity_id: >
    {% set list = label_entities("purge")%}
    {% set todo = "- " + list | join("\n- ")%}
    {{-todo-}}

fails with the following message:

Failed to perform the action recorder.purge_entities. Entity ID - sensor.adguard_home_average_processing_speed - sensor.adguard_home_dns_queries is an invalid entity ID for dictionary value @ data['entity_id']. Got None

What am I doing wrong?

Looks like you’re trying to turn your list into how it appears when it’s pure YAML text

What you want is a simple comma separated list

action: recorder.purge_entities
data:
  keep_days: 5
  entity_id: >
    {{-  label_entities("purge") | join(",") -}}
1 Like

@teskanoo thank you so much. That was it. Your code worked!

I assumed that my script should have worked since the manual format is:

action: recorder.purge_entities
data:
  keep_days: 1
  entity_id:
    - sensor.mfc_l8690cdw_status
    - sensor.mfc_l8690cdw_page_counter
    - sensor.mfc_l8690cdw_b_w_counter
    - sensor.mfc_l8690cdw_color_counter
    - sensor.mfc_l8690cdw_duplex_unit_pages_counter
    - sensor.mfc_l8690cdw_drum_remaining_life
    - sensor.mfc_l8690cdw_belt_unit_remaining_life
    - sensor.mfc_l8690cdw_fuser_remaining_life
    - sensor.mfc_l8690cdw_laser_remaining_life
    - sensor.mfc_l8690cdw_pf_kit_1_remaining_life
    - sensor.mfc_l8690cdw_pf_kit_mp_remaining_life
1 Like