Fix entitiy bloat

Problem

Synopsis

Currently we have a lot of entities in our Home Assistan, which are only interfere in everyday usage. But can’t be removed or disabled, because other entities or automation are counting on them.

Example

For example I have battery-powered ESPHome project. And since I am tinkering with it periodically, I have sensors for:

  • Battery-Voltage
  • Battery-Procentage (Template from Battery-Voltage)
  • Battery-Procentage-filtered (filter from Battery-Procentage)
  • Battery-Procentage-change (trend from Battery-Procentage-filtered)

I want to see those graphs for a day to know, how good or bad my change ESPhome project is. But I don’t need to store them for 10 days, as all other sensors, which I want for statistics and graphs. And I don’t want to see them in all the lists, when I creating new automation, or changing Dashboards.
And, I think, any user who uses Home Assistant heavily have a lot of entities like that: temperature trends, boolean helpers, etc.

What is the problem here

We need to have ability to divide entities, and treat them differently. This will help with system usability and DB size. Don’t see less important sensors everywhere and not store them in Database for long time.

Solution

As developer from different sphere, my solutions can be not in the spirit of HA. Because of that I made this feature request, to understand, if it is something, what community wants. Maybe I will try to help integrate them, if this changes are desirable.

Here is multilevel solution. We can stop on any level, and it still will be good solution for this problems.

Level 1 (Separate type of entities)

Add ability to divide sensor/entity viability type. As example, it can be shadow:true property. Or, for easier future features visibility:shadow or visibility:hidden or any other way to distinguish second class entities.

Example code
  - platform: template
    name: "My Tempelate"
    id: my_tempelate
    shadow: true
    lambda: |-
      ...

or

  - platform: template
    name: "My Tempelate"
    id: my_tempelate
    visibility: shadow
    lambda: |-
      ...

    Level 1.1 (Hide them)

Separate those new entities on separate page, or make them not appear without special checkbox. In that case “Development Tools”, “Entities page” and fronted configuration will feel cleaner.

    Level 1.2 (Store them less time)

Store those entities less time. We need them for filters, or other stuff, but don’t need to store 10 days of them.

Example code
recorder:
  auto_purge: true
  purge_keep_days: 10
  purge_shadow_keep_days: 1

Level 2 (Combine sensors)

Lot of times we need sensor from integration only to calculate our own sensor from them.

    Level 2.2 (Substitute integration entities)

Add ability to modify integration sensor, without keeping original value. Like, adding filters or templates. It will be just more deep customization.

Example code
homeassistant:
  # ...
  customize:
    # Change C to F and change only once every 2 minutes
    sensor.bathroom_temperature:
      entity_value: '{{ (value -32 ) * 5/9 }}'
      filters:
      - filter: time_simple_moving_average
        window_size: "00:02"
        precision: 1

    Level 2.2 (Separate storage period for each sensor)

Then we also can change how long each sensor is stored. We will be able to greatly reduce DB size by storing each sensor only as long, as it needed, but not all sensor for the same duration.

Example code
homeassistant:
  # ...
  customize:
    sensor.bathroom_temperature:
      purge_keep_days: 3

Level 3 (Make entities into attribute)

And last, but not least — just make some of our entities — attributes for the other entities. No one need separate entity for temperature trends. It should be just added attribute to existing temperature sensor. And same goes to a lot of entities which are bloating everything. I don’t think this will reduce DB size, but ability to attach entities as attributes will make navigation inside more pleasant.

P.S.

I totally understand, that Home Assistant is open source project, and everything I mentioned can already be done, with AppDeamons, code tinkering, etc. But I think having such things in native functionality will be really good for the project.

If you have the software skills to implement your Feature Request, you can submit your proposal in the Discussions section of the Architecture repository where the development team can review it and respond.


I believe you already know that any entity can be excluded from the recorder integration. What would be convenient is the ability to exclude it via the UI as opposed to the current method which is to edit the recorder’s YAML configuration.

You do if you want to see a History Chart because it only graphs an entity’s state and not any of its attributes.

Good luck with your discussion in the Architecture repository.

2 Likes

This is similar to my earlier FR (feel free to up-vote if you agree):

I agree that there needs to be more granularity in the Recorder than one keep_days for all data, or exclude things entirely.

Anything which improves this problem would be great. You have my vote.

1 Like

Yes, but not all of them should be removed from DB entirely. How I wrote in second sentence:

But can’t be removed or disabled, because other entities or automation are counting on them.

You can, just not with native graph card. I am using ApexCharts Card, so was not thinking about limitation of native graph.

Thank you! Didn’t know about that! But it is too big task to start implementing by myself. I better start HA development with something less impactful, as frontend for example

I have several entities excluded from recorder but continue to reference them in automations.

Excluding an entity from recorder simply stops recording its state-changes and doesn’t “remove or disable” it. Automations can continue to reference it.


What I believe you are proposing is, instead of a single purge period, multiple purge periods. In other words, the history for some entities could be set to a shorter period than the default value.

Yes, you can use current value in automatons. But you can’t use them anywhere, where you need stored values. For example in filter time_simple_moving_average

That’s true but nothing you have proposed would change that. An entity’s states are either recorded or not; there’s no middle ground.

Anyway, save your explanations for the development team; I don’t have any say in what architectural changes are permitted.