Best ways to filter entities

[Ildar_Gabdullin]

Assume we need to list some entities based on some criteria.
There are at lease these possible ways:

1.Using some pattern in entity_id:

states | selectattr("entity_id","search","xyz") | ...

and here you have to follow some rules of naming for entities.

2.Using groups:

expand('group.xyz') | ...

and here you need to manage groups (manually or automatically - and in latter case you again have to filter entities using some way).

3.Using areas / integration - if you need to list entities related to them.

4.Using custom attributes added either by customization or (for template sensors) inside template sensors:

states |
  selectattr('attributes.attr_1','defined') |
  selectattr('attributes.attr_1','eq','xyz') |  ...

but - if I am not mistaken - every record for this entity in database will store a static field “attr_1” = which is not good.

I would like to hear about other people’s experience for this…

[Ildar_Gabdullin]

Hi, does anyone has an advice?

[petro]

You are mistaken. selectattr is a filter and only produces results that are actually accessed.

The states object accesses all objects and forces updates on all state changes. I believe this is what you’re getting confused by. The template itself is not accessing all that memory.

Using expand() limits your template to only be updated based on what’s found in the template. So, it would update less, unless you have a very chatty entity in your expand. Then it would be no different.

i.e. using states is the problem, not select, selectattr, reject, and rejectattr. Using expand limits your initial set of entities before using something to filter the list down.

[Ildar_Gabdullin]

@petro

You are mistaken.

Sorry, that was a very brief and unclear answer.
Do you mean that for this sensor (“template”, for instance):

template:
  - sensor:
      - name: my_sensor
        state: ...
        attributes:
          attr_1: xxxxxxxxxxxxxxxxxxxxxxxxxx
          attr_2: yyyyyyyyyyyyyyyyyyyyyyyyyy

the database will not keep values “xxxxxxxxxxxxxxxxxxxxxxxxxx” & “yyyyyyyyyyyyyyyyyyyyyyyyyy” for EACH sensor’s change?
I.e. these static values are stored once for this sensor and do not repeat for each state change?

@petro

selectattr is a filter and only produces results that are actually accessed.

And how is that confirms or disprove my post? Could you elaborate?

@petro

The states object accesses all objects and forces updates on all state changes. I believe this is what you’re getting confused by. The template itself is not accessing all that memory.

Yes, I understand that the “states” stores all entities. But where is my confusion? Unclear…

@petro

Using expand() limits your template to only be updated based on what’s found in the template. So, it would update less, unless you have a very chatty entity in your expand. Then it would be no different.

Sorry, again unclear.
Assume I have a group - and I believe that this group already have all needed entities.
Expanding this group gives me a list of these entities.
Surely I need to manage my groups, update them periodically - and I mentioned this a possible problem (a not updated in time group may contain a wrong list of entities).

[petro]

@Ildar_Gabdullin

the database will not keep values “xxxxxxxxxxxxxxxxxxxxxxxxxx” & “yyyyyyyyyyyyyyyyyyyyyyyyyy” for EACH sensor’s change?

It’s not a database. It’s the state machine. It’s a collection of state objects that house the current state object for all entities. This is persistent at all times. State objects do not magically gain attributes when you ask for an attribute that doesn’t exist in the state object.

@Ildar_Gabdullin

And how is that confirms or disprove my post? Could you elaborate?

Generator’s only give you what you asked for. If you didn’t ask for it, it doesn’t give it to you. When you say “I only want entities with xyz”, you’re only going to get entities with ‘xyz’. Nothing else is altered or presented on entities that do not have ‘xyz’.

@Ildar_Gabdullin

Yes, I understand that the “states” stores all entities. But where is my confusion? Unclear…

No, it does not store anything. It accesses. Store means “makes new memory”. Accessing means, “Uses the already existing memory”.

@Ildar_Gabdullin

Assume I have a group - and I believe that this group already have all needed entities.
Expanding this group gives me a list of these entities.
Surely I need to manage my groups, update them periodically - and I mentioned this a possible problem (a not updated in time group may contain a wrong list of entities).

I don’t understand what you’re asking here.

[petro]

FYI the state machine with all state objects exists at all times. That’s how everything is displayed in home assistant. When you use states, you’re accessing them all. When you use expand you’re accessing a subset of the entities that are in states. That’s all you really need to understand.

[Ildar_Gabdullin]

@petro

It’s not a database

Stop.
I wrote:

@Ildar_Gabdullin

but - if I am not mistaken - every record for this entity in database will store a static field “attr_1” = which is not good.

That was about using attributes like “attr_x” to select needed entities.
You answered:

@petro

You are mistaken

So I concluded that your answer is regarding my assumption above - and asked about database again.
And you finally wrote “it’s not a database”, I am confused.

@petro

State objects do not magically gain attributes when you ask for an attribute that doesn’t exist in the state object.

Yes, but why are you saying this? I see a misunderstanding.

And regarding all the rest - I totally do not understand how is this related to my 1st post.
Probably I am missing something.
Or may be you understood my question in a wrong way.

[petro]

@Ildar_Gabdullin

So I concluded that your answer is regarding my assumption above - and asked about database again.
And you finally wrote “it’s not a database”, I am confused.

Because it’s not a database. A database is very specific in software. This is not that. This is an object that is holding the current state of all entities in HA. It’s a collection. A list. A lookup. It’s not a database.

@Ildar_Gabdullin

Yes, but why are you saying this? I see a misunderstanding.

You asked:

@Ildar_Gabdullin

if I am not mistaken - every record for this entity in database will store a static field “attr_1” = which is not good.

Nothing is stored. When you use selectattr('attributes.attr_1','defined'), nothing is stored. It’s simply looking to see if a state object has that attribute. I.e. Nothing is added or stored, it’s simply looked at.

[petro]

Imagine you’re in a room with 10 people in a line. You quickly glance at all 10 peoples shoes and you ask the people wearing red shoes to step forward. So only 3 people with red shoes step forward. Did the people without red shoes do anything? Or did they just sit there doing nothing? The only “work” being done is the 3 people stepping forward.

That’s how selectattr, select, reject, and rejectattr work. Except it’s like that with state objects. You only get & use what you ask for. So all that matters is the initial set that you start with.

[Ildar_Gabdullin]

Sorry, you misunderstood my answer.
Info which you are providing is definitely valuable - but this is not what I asked for.

Once again: sometimes we need to list a set of entities - to do smth with them, to list in auto-entities card etc.
I posted 4 known methods to select (“filter”) needed entities:

1. List entities which have a defined entity_id: “sensor.voltage_1”, “sensor.voltage_2”, “sensor.voltage_xyz”. So here we need to follow some “naming conventions”.
2. List entities by expanding a group - and this group must contain needed entities (and we again need to create these groups based on some filters).
3. List entities based on area, integration, device_class etc. This way may be used as a PART of a desired filter.
4. List entities based on previously added attributes. And here I have an assumption (possibly wrong): if some entity has an attribute used ONLY for filtering - it occupies a place in DB which is not good (same static value in each record).

A question was - what method of filtering do you recommend?

[petro]

Sorry, but maybe this is a language barrier thing, but your original question is not as clear as this post.

Personally, I use 2 only. I pair that with an automation that runs periodically to automatically generate group so that I don’t have to maintain any lists of entities.

I still place random attributes on all my template sensors because I filter my recorded database to only things that I care about.

If you’re worried about CPU and ‘heavy’ templates, using 2, or 3 are the most optimized.

Here’s an example of an autogenerated list and template it’s using:

alias: Create Group
trigger:
- platform: event
  event_type: event_template_reloaded 
- platform: homeassistant
  event: start
action:
- service: group.set
  data:
    object_id: my_group
    entities: >
      {{ filter to a list of entities that you want using method 1, 3, or 4 }}

Then

template:
- sensor:
  - name: blah
    state: >
      {{ expand('group.my_group') | .... }}

This will automatically update your group whenever you reload your template entities or when HA starts.

---

EDIT: I’m super lazy with my configuration, I do not like maintaining lists of things. So this method will get you the ‘most hands off’ approach to lists of entities. It’s also optimized to only perform the heavy templating on restarts or reloads. And all the lightweight templates are performed regularly when entities update.

[Mariusthvdb (Marius)]

really a bit puzzled by your post here Ildar. What is the purpose of your quest? you mention auto-entities, and then only a very select set of the available filter options, which Thomas has listed here

so, yeah, those can be useful, depending on the output you desire, there’s no preference other than the requirement set by the user.

additional to those, you have the template option. with the available (de)selectors. Based on the collection of entities we startup with.

As that last aspect in fact your actual question? how to define the base set of entities, to work with ?

[Mariusthvdb (Marius)]

@petro

event_type: template_reloaded

btw Petro, you sure about that event? I have event_template_reloaded listed here, and functional upon template reload,

where template_reloaded does not fire in the Events listener at all

[Ildar_Gabdullin]

@petro

language barrier thing, but your original question is not as clear as this post

No problem, we are all intelligent people and should always find a way to understand each other.

@petro

runs periodically to automatically generate group

I do it for some cases, and here is a risk: sometimes a group may contain an outdated list…
Your solution with “template_reloaded” may not sufficient enough in cases like “I need entities with some desired state”; so it may be used as a PART of filtering (as all other 3 methods) - expand a group then analyse each entity.

As for using “attributes for filtering”: it was not my idea, probably I heard it from you or Taras.
I find this method rather good in some situations when you do not want to deal with groups or entity names.

@petro

I still place random attributes

What do you mean by “random”?
Is it some kind of random unique value?
Could you describe a use-case for this?

Also, I am still worried about my assumption -

@Ildar_Gabdullin

if some entity has an attribute used ONLY for filtering - it occupies a place in DB which is not good (same static value in each record).

surely it is an off-topic here - but do you know smth about this, do these static user-defined attributes occupy a place for every entity’s state change?

[petro]

I don’t remember what it was, I pulled that example from memory

[petro]

@Ildar_Gabdullin

I do it for some cases, and here is a risk: sometimes a group may contain an outdated list…
Your solution with “template_reloaded” may not sufficient enough in cases like “I need entities with some desired state”; so it may be used as a PART of filtering (as all other 3 methods) - expand a group then analyse each entity.

As for using “attributes for filtering”: it was not my idea, probably I heard it from you or Taras.
I find this method rather good in some situations when you do not want to deal with groups or entity names.

yes, but the beauty of an automation like that is you can make it update whenever you want by adding additional triggers.

@Ildar_Gabdullin

What do you mean by “random”?
Is it some kind of random unique value?
Could you describe a use-case for this?

So, when I create a template entity, I always add a template: ... attribute that basically describes where it came from so I can quickly filter it in developer tools → states page.

e.g.

template:
- sensor:
  - name: Outside Temperature
    state: "{{ ... }}"
    attributes:
      template: temperature

@Ildar_Gabdullin

surely it is an off-topic here - but do you know smth about this, do these static user-defined attributes occupy a place for every entity’s state change?

Yes, but from what I can tell, the most recent database changes only store them when they change. bdraco might know better.

[Mariusthvdb (Marius)]

sure, np there. the docs confirmed: Template - Home Assistant

[Ildar_Gabdullin]

@petro

by adding additional triggers.

Agree, this makes this way flexible. And may be cumbersome - in some cases, but this is again a flexibility of HA - do whatever you want)).

@petro

describes where it came from

Exactly same way I am using it, thank you.

@petro

bdraco might know better

Thanks, I will ask him.

Update: got an answer from bdraco.