So I’m going to lead by saying that if you are interested in this topic, you should vote for this and/or this. Because my solution is such a hack. It definitely works, I’ve been using it for a while but its absolutely a hack and I wish this was native.
Ok, now that that’s out of the way, let’s get started. When I migrated my automation logic from Node RED to HA the experience was mostly positive but I ran into one major dealbreaker - the complete inability to use regexes and wildcards in entity_id in triggers.
I almost never hard-code entity ID lists anywhere. I am very diligent about following naming patterns so that I can easily add or remove devices and have my automations just handle that change. To give an example I made an automation maybe 2 years ago that tied together a door sensor to a closet light. Since then I’ve added door sensors and smart lights to 3 more closets and never once had to change the automation. As soon as I added the devices to HA the closet lights turned on and off with the door. I have similar patterns like this all over.
So I was completely unwilling to migrate until I solved this. I did and at this point its been running smoothly for like 6 months so I’m pretty confident in it. It may be a hack but at least its a hack that works well. Here’s the package you can use then I’m going to talk about some of the key pieces and instructions. If you don’t know how to use a package, see here.
And before anyone asks, no this cannot be a blueprint. Its not just one automation, there’s supporting pieces. Sorry.
automation:
- id: 5db0e93431054d038971552f904ac3b9_update_entity_lists
alias: Update entity include lists
description: List of all entity ID patterns used in triggers. This keeps files up to date with those lists for !includes.
mode: single
max_exceeded: silent
variables:
domains:
#### TODO: REPLACE WITH YOUR LIST OF DOMAINS
- alert
- light
- person
- update
matches:
#### TODO: REPLACE WITH YOUR LIST OF MATCHES
# Cannot use `options` here or it will break. Options is taken by input_select
battery_level: sensor[.].+_battery_level
closets/inputs/lights: switch[.].+_closet_light
closets/inputs/sensors: binary_sensor[.].+_closet_(door_contact|motion_sensor_occupancy)
closets/timers: timer[.].+_closet
couch_occupied: binary_sensor.family_room_couch_.+_seat_contact
detected_activity_processed: sensor[.].+_detected_activity_processed
expected_travel_mode: select[.].*_expected_travel_mode
guest_toggles: switch[.]guest_staying_.+
is_traveling: binary_sensor[.].+_is_traveling
lock_weekday_access: input_boolean[.]front_door_lock_weekday_access_.+
phone_battery_level: sensor[.].+_phone_battery_level
protect_status: binary_sensor[.]nest_protect_.+(co|heat|smoke)_status
reached_a_destination: switch[.].+_reached_a_destination
recurring_task: input_datetime[.]recurring_.+
reminders/appointments: input_datetime[.]recurring_appointment_.+
reminders/days: sensor[.]reminder_.+
reminders/tasks: input_datetime[.]recurring_task_.+
request_location_update: button[.].+_request_location_update
room_illuminance: sensor[.].+_light_sensor_illuminance_lux
room_illuminance_stats: sensor[.].+_illuminance_lux_stats
room_presence/inputs/controllers: select[.]presence_lights_.+
room_presence/inputs/detectors: binary_sensor[.]person_detected_.+
room_presence/inputs/light_sensors: binary_sensor[.]bright_.+
room_presence/off_scenes: scene[.]room_off_.+
room_presence/on_scripts: script[.]room_on_.+
sleep_sensors: binary_sensor[.][a-z]+_is_asleep(_in_.+)?
trigger:
- platform: homeassistant
event: start
- platform: event
event_type:
- entity_registry_updated
- automation_reloaded
action:
- parallel:
- sequence:
- alias: Update domain lists
repeat:
for_each: "{{ domains }}"
sequence:
- alias: Get existing and new list
variables:
path: "domain/{{ repeat.item }}"
comment: "Entities with domain `{{ repeat.item }}`"
old: &old-list "{{ state_attr('input_select.auto_lists', path) | default([]) }}"
new: >-
{{ states | selectattr('entity_id', 'search', '^' ~ repeat.item ~ '[.]')
| map(attribute='entity_id') | list }}
- &record-changes
alias: If list changed, record it
if: "{{ old != new }}"
then:
parallel:
- alias: Update the file
service: notify.update_entity_list
data:
message: |
{{ path }}
# {{ comment }}
- {{ new | join('\n- ') }}
- alias: Update the select
service: input_select.set_options
data:
entity_id: input_select.auto_lists
options: "{{ state_attr('input_select.auto_lists', 'options') + [path] }}"
- sequence:
- alias: Update match lists
repeat:
for_each: "{{ matches | dictsort }}"
sequence:
- alias: Get existing and new list
variables:
path: "{{ repeat.item[0] }}"
regex: "{{ repeat.item[1] }}"
comment: "Entities with IDs matching `{{ regex }}`"
old: *old-list
new: >-
{{ states | selectattr('entity_id', 'match', regex)
| map(attribute='entity_id') | list }}
- *record-changes
- alias: Stop if no files were updated
condition: "{{ state_attr('input_select.auto_lists', 'options') | count > 1 }}"
- alias: Update directory file
service: notify.update_entity_list_directory
data:
message: |
# Directory file. !include in customization to make an entity with all lists as its attributes
{{ ( domains | map('regex_replace', '^(.*)$', 'domain/\\1') | list
+ matches | dictsort | map(attribute=0) | list
) | map('regex_replace', '^(.*)$', '\\1: !include \\1.yaml') | join('\n') }}
- parallel:
- alias: Notify about updates
service: persistent_notification.create
data:
notification_id: entity_lists_updated
title: Entity lists updated
message: "Modified: {{ state_attr('input_select.auto_lists', 'options')[1:] | join(', ') }}"
- alias: Reload customizations
service: homeassistant.reload_core_config
- alias: Reload input selects to reset auto_lists
service: input_select.reload
- alias: Reload scripts
service: script.reload
- alias: Reload groups
service: group.reload
- alias: Reload automations
service: automation.reload
input_select:
auto_lists:
name: Auto lists
icon: mdi:view-list
initial: listing
options: ['listing']
homeassistant:
customize:
input_select.auto_lists: !include /config/common/auto/.directory.yaml
notify:
- platform: command_line
name: Update entity list
command: >-
read path
&& path="/config/common/auto/${path}.yaml"
&& cat /dev/stdin > "$path"
- platform: command_line
name: Update entity list directory
command: cat /dev/stdin > /config/common/auto/.directory.yaml
If you import this, add input_select.auto_lists to your list of entities to exclude in recorder. I cannot do that for you in the package because of how package import work. But definitely do this or you will get a lot of SQL errors from this huge entity
Ok so, how does this work? First it doesn’t actually enable triggers like this, it’s not possible for a package to change that:
trigger:
platform: state
entity_id: update.*
What it does instead is write out YAML files that can be used via !include. So for example if you were actually interested in all update entities what you would do is add update to the list of domains in the variables section of the automation (note the giant TODO comment):
variables:
domains:
- update
And then you could use this as a trigger in your automation:
trigger:
platform: state
entity_id: !include /config/common/auto/domain/update.yaml
Or let’s say you wanted to make an automation that sent a notification any time any battery sensor was low. To do that you can add something like this to variables.matches in the automation (note the second giant TODO comment):
variables:
matches:
battery_level: sensor[.].+_battery_level
And then you could use a trigger like this in your automation:
trigger:
platform: state
entity_id: !include /config/common/auto/battery_level.yaml
Matches need to be regex. Glob would be nicer I know but I can’t do that. There’s no glob support in templates, only regex support. If you really hate regex here’s the bare minimum you need to know to use this:
-
.means any character. If you want to actually match.then put[.](or\\.I just dislike the double slash) - If you want to do a wildcard match (like putting a
*in a glob filter) then put.*
Pretty cool right? This also isn’t limited to automations since you can !include anywhere. Want to make a group with all your switch entities?
switch:
- platform: group
entities: !include /config/common/auto/domain/switch.yaml
Or want to use the same list of entities in a script?
sequence:
- alias: Turn on all the things!
service: switch.turn_on
data:
entity_id: !include /config/common/auto/domain/switch.yaml
And the automation itself is listening for any changes to the entity registry as well as Home Assistant restart. So if you add an entity to the system or change the entity ID of one the automation will immediately pick up that change, see if any lists need to be updated, update them and reload all your automations and scripts. It also listens for the automation_reloaded event so if you change the lists it re-runs and updates the files.
Some might ask “what about templates?” Its true, !include does not work in templates. So if you wanted to use all your battery sensors in a template or template sensor this trick won’t work. However, I’ve got you covered, this is where input_select.auto_lists comes in. For every list you make there is an matching attribute on this entity. So if you want to make a template binary sensor that turns on anytime anytime any battery level sensor is below a threshold you can do this:
template:
binary_sensor:
name: Low battery in house
state: >-
{{ expand(state_attr('input_select.auto_lists', 'battery_level'))
| map(attribute='state') | map('float', 0)
| select('lt', 20) | count > 0 }}
Seriously, exclude input_select.auto_lists from recorder. It’s going to be massive.
The automation does not directly reload template entities but it does reload input_select entities. So template entities will see that state chnage and react to it like normal.
That’s pretty much it. I’m going to put a bit more of a technical deep dive below for anyone interested in questions like “why an input select” and “what’s the command line stuff”. But for those that just want to use it, feel free. Also I left all my domains and matches in there just to provide some examples for people. You should delete or comment out all of them and put in what you want.
Note that if you don’t like my folder choice (/config/common/auto) feel free to change it. Just find and replace /config/common/auto within the package to whatever you want.
Also I should probably note that the automation does not clean up after itself. If you remove a match or a domain then it won’t update that file anymore but it won’t delete it. You have to do that manually.
Technical details
What’s up with the input select?
For the most part it could’ve been any entity at all. You can see the real guts of it is in customization
homeassistant:
customize:
input_select.auto_lists: !include /config/common/auto/.directory.yaml
That’s how it gets all those attributes. The automation also writes out a directory which is a giant dictionary of includes.
The reason its an input select is purely for the automation internals. The automation goes through each domain and match you specified. It finds a list of entities that match that query and compares it to what the input select has. If its different it adds that domain/match to the list of options in the input select so the automation knows what changed. Then it can re-write those particular files and notify you what changed. That couldn’t be done with automation variables because they are scoped, you can’t add anything to a top-level variable in loop steps.
It never has any options except during the automation. You should not modify it, just treat it as read-only.
Why do you have command line notifies?
Those write out the files. They are notifies as opposed to shell commands because shell commands don’t accept any inputs and command line notifies do.
Shell commands require you to stuff values in helpers and then read out from those. Gross, I basically never use them. Command line notifies are the same thing except the message comes in as stdin so there’s no helpers required.
Wish there were multiple inputs though. This isn’t really ideal:
read path
&& path="/config/common/auto/${path}.yaml"
&& cat /dev/stdin > "$path"
Basically the first line is the first argument and everything else is the second. Would be nice to just have multiple arguments.
What’s going on with matches that have names like room_presence/inputs/controllers?
An advanced feature I put in. If the name of your match includes slashes then those become folders. So to include that you would would do this:
!include /config/common/auto/room_presence/inputs/controllers.yaml
The reason for this is for those sometimes I wanted to specifically use the room presence controllers and sometimes I wanted to include the controllers, detectors and light sensors (i.e. everything in room_presence/inputs). Putting them in folders means you can do that like this:
!include_dir_merge_list /config/common/auto/room_presence/inputs
Could I use this to make a trigger that fires anytime any entity with a particular attribute changes?
Absolutely. Or whatever criteria for finding entities you want. I’m not going to write that for you but here’s an overview on how to do it.
Just add a new list variable to variables and then add a new sequence to handle it in the first parallel step of the automation like this:
- sequence:
- alias: Update <criteria> lists
repeat:
for_each: "{{ <criteria> }}"
sequence:
- alias: Get existing and new list
variables:
path: "<criteria>/{{ repeat.item }}"
comment: "Entities with <criteria> `{{ repeat.item }}`"
old: &old-list "{{ state_attr('input_select.auto_lists', path) | default([]) }}"
new: >-
{{ states | selectattr('entity_id', 'search', '^' ~ repeat.item ~ '[.]')
| map(attribute='entity_id') | list }}
- *record-changes
Fill in the blank on <criteria> and replace the template in new with however you find entities based on the current loop item for <criteria>. Everything else will take care of itself.
Feel free to ask any other questions below!