Priority queue for state changes

[sammousa (Sam)]

Introduction

When writing automations you currently always need to have a global view of your whole home in mind. People who ask about this are told to “change their logic” by either merging their automations into one automation, or making all automations interdependent. As someone already pointed out in a few topics on mutexes, these tricks and workarounds do not actually solve the problem; even though they will significantly reduce the occurrence of some of the problems.

Problem
Imagine I have a light in my home, it’s an RGBW light and I use it as part of a scene when I’m on my couch.
I want to use this light as a notification light for all kinds of things:

1. When the doorbell rings, it should go green for 3 seconds.
2. When a smoke detector goes off, it should flesh red until the alarm settles.
3. When I receive an email it should flash blue for 1 second.
4. Imagine I have 10 more notifications like this.

This use case is very complicated to do with HA because there we lack the right data structures. I can store the current state in an automation, I can store a value in one of the input helpers and that is about it.
Of course, if I want to have clean code, all these cases above should live in their own automation, and since they have nothing to do with eachother they should not have any references to the others.

Proposed solution
A simple solution to support these kinds of automations would be to support something like stacks or priority queues.
Instead of telling the light to go blue for 1 second and then go back, I instead push something onto a priority queue and HA takes care of executing it.
The queue can be filled from multiple threads, but commands on it are always only executed by a single thread.

So each automation can now be very simple:
– Notify doorbell

1. Push the following message to the queue: { entity: light1, color: green, duration: 3, priority: 50 }

– Notify email

1. Push the following message to the queue: { entity: light1, color: blue, duration: 1, priority: 40 }

– Alarm

1. Push the following message to the queue: { entity: light1, color: red, animation: flash, duration: -1, priority: 1000, label: alarm }
2. Wait for template / condition until alarm is off.
3. Remove the messages with label alarm from the queue.

From an implementation perspective (I think) this doesn’t have to be extremely complicated:

• Every entity has it’s own queue structure
• Each entity belongs to 1 queue runner
• An automation entry could choose to use the queue or not use it
• Changing the status directly uses some default value for priority, this allows you to choose whether your notification should override manual control. (For example, don’t turn off the alarm with the light switch, but maybe do turn off the email notification with the light switch)

I believe this could greatly increase the power of automations while reducing their complexity.

Alternative solution
If you have an alternative solution please make sure it satisfies the requirements below. I am aware that currently this problem is not unsolvable, it is just unworkable at bigger scale. (Imagine the light from the example being part of a scene, imagine some notifications also applying to other lights that are part of different scenes etc)

Requirements for a clean solution:

• Automations should not be aware of eachother by name / id
• Changing or adding a new notifications should not require changes to existing automations
• Automations should be separate

[templeton_nash]

Could you do this with scripts?

You could have a script that takes in any parameters required for your light. The script would need it’s mode set to ‘queued’.

Each automation could then call the script and pass in the required parameters.

[sammousa (Sam)]

Hmm, for light that could work, how would we generalize it to any service call?

Edit: Actually, the queued script would not be able to use priorities.
We need priorities to indicate which notifications are more important, otherwise we might have a flashing email notification while the house is on fire

[sammousa (Sam)]

In theory a hacky solution could be:

1. Use dynamic scenes to describe changes.
2. Create webhook trigger
3. Create script triggered by 2 that:

• Snapshots given entities from trigger data
• Applies the scene from trigger data
• Wait for duration
• Restores from snapshot

Still missing the priority system, but if using webhooks this could possibly be outsourced to something external.
Note if we don’t require the priorities we can use a queued script with scenes for a generalized approach that works for lights and other platforms.

[tom_l]

I still don’t see the issue. All of this can be done with current automation syntax.

You can save scenes on the fly and restore them, you can prioritise using conditions.

[Tinkerer (aka DubhAd on GitHub)]

You forgot to vote for your own request.

[sammousa (Sam)]

Actually you cannot prioritise using conditions.
Using conditions you get the downsides I explicitly mentioned:

• All automations need to be aware of all others to do prioritization
• There are race conditions since your condition and action are not atomic

There’s a good example on race conditions here: Race condition - Wikipedia and what you’re proposing (using conditions) maps perfectly to the bugs mentioned in the article.

[anon63427907]

adding two cents;

• not all integrations are playing well with scenes, they either lack of exposing their current state as scene or applying the scene on the entities.
• it would be nice to have a mechanism to override the queue such as I am not worried about my telegram notification when there is a fire alarm

[tom_l]

@sammousa

All automations need to be aware of all others to do prioritization

What if you use an input number/select helper to indicate the currently running priority?

First action is to set this to the automation’s priority, last action is to set it to 0.

Each automation also compares their own hard coded priority to this helper in a condition.

[salsaman]

I agree ! Home assistant needs things like this to keep pace as automations become ever more complex and intertwined. Consider the current model - a state change or similar triggers an automation, The automation checks various conditions and exits or performs an appropriate action. This is a very linear approach and as pointed out, makes it difficult to react to and deal with multiple / parallel actions. Some kind of list / queue helper would be a good start. I think a lot of these issues can actually be resolved within the existing framework by add some new helper types or extending existing ones. In relation to this I have made a number of feature requests, which have sadly all been ignored.

How about timers that can be actually useful for measuring durations and sequencing events ?

Make the Timer helper actually useful

I suggesting adding the following services timer.reset timer.freeze timer.unfreeze make the state of the timer be the current time. Reset sets the current time to 0, freeze stops the timer and unfreeze starts it again. Then we can make nice triggers / conditions based on the numeric state of a timer N.B I had a very brief glance at the current timers. It seems the only service currently existing is Cancel. This can easily be replaced by freeze followed by reset. To get 5 stars, make it s…

A mutex helper so automations can obtain a lock on critical resources

Add a "mutex" automation helper

Final solution: Acquiring a lock is the only action that is serialized via a script Releasing the lock is done by writing an empty value to the input_text helper The script you can find here, the forum is buggy and posting code doesn’t work: HA Scripted Mutex · GitHub

Want to make an automation react based on the mean value of a sensor ? Sorry we can’t do that. You can look at a pretty chart though.

Make a Statistics helper

Can we please have a helper which can do at least mean average value of a sensor over a given period (e.g. last 24 hours). This would be really useful for automations. For bonus points you could select the statistic you want (e.g. min, max). You could add other statistics like mode, median, standard deviation, statistical range (similar to what we have in the combine sensor). This could also simplify things by replacing both the derivative and integral helpers. The UI could be something like: …

And if we want to move to a more holistic view, it is going to be a lot easier if we can combine various sensors in more flexible way. Exactly what I proposed here Make the Combine Sensors helper be super useful

Sorry for going a bit off topic, but the one thing all these requests have in common is that they start to move away from the simplistic trigger/condition/action model which is starting to look a bit obsolete. The same way we moved from controlling individual lights to lighting groups, we need to stop considering all things in isolation and add in these kinds of things which allow a more joined up view.

[petro]

all of those FRs can already be done in HA and it seems like you’re ingoring responses in the FRs. Not sure why.

Tom already replied to your timer, you somehow missed all the service calls that are built into the system.

For your mutex, you can use an input boolean in combination with a wait_template.

And your statistics helper already exists. The integration is literally called statistics.

[tcj (Taylor)]

I found this addon that tries to add background state and priorities to at least lighting. It looks promising, and is a lot easier to wrap my head around coming from theatrical lighting compared to trying to actively push the correct state every time anything changes.

[salsaman]

I was mistaken about timers, my apologies, but regarding the other FRs, I don’t believe there is currently a way to replicate all the requested functionality. Regardless of this, I believe the point is still valid, the current design seems still too focused on individual devices/scripts and maybe ought to be rethought.

[tom_l]

Or, you could re-think the way you automate. Nearly everything should be state driven. If you have one entity who’s state drives multiple automations, consider combining them into one automation.

[sammousa (Sam)]

Sorry Tom, but I can’t agree with this reasoning.

If you have one entity who’s state drives multiple automations, consider combining them into one automation.

Imagine I have an entity who tells you whether someone is home, many things will relate to this, alarm, heating, lighting. Obviously maintaining all of that in a single automation would be maintenance hell.

Automations should be small and specific, similar to how you’d prefer to write software code they should obey the Single Responsibility Principle.

[tom_l]

That example sounds like conditions (is the person home), not triggers (person arrives home).

You can use the conditions wherever you want but there should only be one automation that triggers on the person arriving home.

@sammousa

they should obey the Single Responsibility Principle.

Exactly. One trigger → one automation.

[mchangsp]

@sammousa

Proposed solution
A simple solution to support these kinds of automations would be to support something like stacks or priority queues.

I am thinking of using a local todo list as ‘the queue’ to store items. Each item contains a ‘unit of work’ (turn on light, wait 1 sec, etc).
HA already has services to:

• add an item to the list
• delete an item from the list

I am missing service calls for:

• “sort the to do list according to some value (due date, priority) and give me the first item”
• “get me the first item off the list + delete that item from the list”

If I have the missing services I can implement all kind of queues like:

• first in first out (FIFO)
• last in first out (LIFO)

And because the queue is a todo list I can display the list in a dashboard.

@sammousa What do you think of this idea?

[mchangsp]

Here is a use case in which I need a queue.
I have a home thermostat entity which sometimes is unavailable (for hours). In scripts/automations I change the thermostat value. The error handling in case the thermostat is not available is cumbersome. How do I remember what needs to be done while the thermostat is unavailable?
My idea is to store all thermostat state changes in a queue. The queue runner has the option to execute all state changes or decide to take only the last one.

Of course I can create a queued script to change the thermostat value. I am not sure of the limits.

• How many script executions can be queued?
• And I cannot “see” how many scripts are queued.
• What happens when Home Assistant is restarted and there are queued scripts?

Using a “todo list” as a queue as the advantages:

• I can see the todo list / queue of outstanding wok to be done
• a todo list survies a restart
• number of items in a to do list is greater then the maximum number of queued scripts

[petro]

You should fix your thermostat instead of trying to find a use for a tool that doesn’t exist.

[mchangsp]

hahaha, I agree. In this case it is a Nefit thermostat with a cloud component.
The cloud component is out of my reach, so I want to queue up all the commands until the thermostat becomes available again (compensating for the fact that Nefit has an uptime of less than 100%).

And there are other use cases where queues are handy. Home Assistant already caters for the specific ‘light state’ use case by creating a scene on the fly. But there are other entities for which you may want to queue up commands (eg an entity object wich sends out notifications and you want to merge messages to reduce the number of messages sent within a certain time period)