I have a couple of template sensors that average the measurements of multiple sensors and use an exponentially weighted moving average (EWMA) to smooth the results. Examples include indoor and outdoor temperature, wind speed, and light level.
I would like to address a few issues and appreciate your assistance:
Code simplification with macros:
Is there a way to move the majority of the code to a macro? This would shorten the template sensor itself and keep the logic centralized for all template sensors.
Using variables in triggers and availability:
Can the sensors defined in variables also be used for the trigger and availability? I want to avoid maintaining three separate lists of the sensors used in calculations. I know yaml anchors exist but haven’t figured out their use in this context.
Trigger simplification:
Is it necessary to define the sensors in state_changed? Can triggers be shortened to update the template sensor every x minutes and additionally on sensor updates?
Refactoring suggestions:
Do you have any other suggestions to make the sensor more robust, faster, and readable? I’m not an expert in jinja, and parts of my code are based on various snippets from the forums.
Here’s the current yaml configuration for reference:
- trigger:
- id: time_pattern
platform: time_pattern
minutes: "/1"
- id: state_changed
platform: state
entity_id:
- sensor.temperature1
- sensor.temperature2
- sensor.temperature3
- sensor.temperature4
action:
variables:
tau: 300
sensor_names:
- sensor.temperature1
- sensor.temperature2
- sensor.temperature3
- sensor.temperature4
sensor:
- name: "Outdoor temperature EWMA"
unique_id: outdoor_temperature_ewma
icon: mdi:thermometer
availability: >
{{ [
states('sensor.temperature1'),
states('sensor.temperature2'),
states('sensor.temperature3'),
states('sensor.temperature4'),
] | reject('in', ['unavailable', 'unknown', 'none']) | list | count > 0 }}
attributes:
last_sampled: >-
{{ now().isoformat() }}
state: >-
{% set sensor_values = sensor_names | map('states') | map('float') | list %}
{% set current_timestamp = now() | as_timestamp %}
{# Determine the next state based on the triggering sensor or average of all sensors #}
{% if trigger.entity_id and states(trigger.entity_id) not in ['unavailable', 'unknown', 'none'] %}
{% set current_measurement = states(trigger.entity_id) | float %}
{% else %}
{% set current_measurement = (sensor_values | sum) / (sensor_values | length) if sensor_values | length > 0 else 0 %}
{% endif %}
{# Fetch the previous state and the time it was last changed #}
{% if this.state not in ['unavailable', 'unknown', 'none'] %}
{% set previous_measurement = this.state | float %}
{% else %}
{% set previous_measurement = current_measurement %}
{% endif %}
{% set previous_timestamp = this.attributes.last_sampled | default(this.last_changed) | as_timestamp(now()) %}
{# Calculate the time difference and the weight for EWMA #}
{% set delta_time = current_timestamp - previous_timestamp %}
{% set weight = e ** (-delta_time / tau) %}
{# Calculate the new EWMA state #}
{{ previous_measurement * weight + current_measurement * (1 - weight) }}
unit_of_measurement: "°C"
state_class: measurement
Macros can be created for Jinja2 code. Reusing templates
This particular example doesn’t benefit (much) from YAML Anchors and Aliases. A macro might be useful if you have several of these Trigger-based Template Sensors and they all use virtually the same state template.
It’s my understanding that variables defined in action are only available to the state option (I suggest you test it). EDITIt appears that it is available in other options as well.
If you’re using a State Trigger then you must list the entities it should monitor (State Trigger does not support templates).
Why do believe there’s a need to periodically update the sensor (with a Time Pattern Trigger)? The State Trigger alone will update it whenever an entity’s value changes. Why periodically compute a result using unchanged values?
Thanks. As for the time pattern trigger: The sensors have different sensitivity. Some update very frequently, others only update every 30 minutes or so when the difference to the previous measurement is large enough to justify the broadcast. Without the time pattern trigger the combined sensor would mainly reflect the noisier input sensors. Using the time pattern ensures that the less noisy sensors are used in the calculation.
Regarding macros, variables and anchors: I hope someone can suggest some improvements, because the current situation does not feel ideal. And yes, I do have about 10 template sensors using this exact state template. I just haven’t figured out how to implement a macro as probably I would need to pass this.state and this.attributes to it?
I just figured out how to move the state template to a macro, the trick is to pass this.state and this.attributes instead of this. Now I wonder where to define the sensor names so I can use them in multiple places in the template sensor. Maybe I creating a group for these sensors might be the solution. Still working on it.
The variable preceded by a & is the anchor. The same variable name preceded by a * is the alias. In the linked example, &area_info contains six lines of YAML (and associated Jinja2 code). Wherever the alias appear, that where the anchor’s code will be duplicated.
As you already know, YAML and Jinja2 are two different programming concepts and each one is handled by a separate processor. Anchors and aliases is a YAML feature so it’s not applicable within a Jinja2 template.
BTW, have you confirmed that a variable defined in action is accessible in more than just the state option? Because if it is (although I doubt it) then this would work:
Surprisingly, it works. Here is my updated template sensor. Thanks for your help. Now the only issue is how to deal with noisy data of sensors, which is why I am now only using a time based trigger:
{% macro calculate_ewma(sensor_names, tau, trigger_entity_id, current_state, attributes) %}
{# Get valid sensor values as floats #}
{% set valid_sensor_values = sensor_names
| map('states')
| reject('in', ['unavailable', 'unknown', 'none'])
| map('float')
| list
%}
{# Get the current timestamp #}
{% set current_timestamp = now() | as_timestamp %}
{# Determine the current measurement based on the triggering sensor or average of all valid sensors #}
{% if trigger_entity_id and states(trigger_entity_id) not in ['unavailable', 'unknown', 'none'] %}
{% set current_measurement = states(trigger_entity_id) | float %}
{% else %}
{% set total = valid_sensor_values | sum %}
{% set count = valid_sensor_values | length %}
{% set current_measurement = total / count if count > 0 else 0 %}
{% endif %}
{# Fetch the previous measurement and the timestamp when it was last changed #}
{% if current_state not in ['unavailable', 'unknown', 'none'] %}
{% set previous_measurement = current_state | float %}
{% else %}
{% set previous_measurement = current_measurement %}
{% endif %}
{% set previous_timestamp = attributes.last_sampled | default(attributes.last_changed) | as_timestamp(now()) %}
{# Calculate the time difference and the EWMA weight #}
{% set time_difference = current_timestamp - previous_timestamp %}
{% set ewma_weight = e ** (-time_difference / tau) %}
{# Calculate and return the new EWMA state #}
{{ previous_measurement * ewma_weight + current_measurement * (1 - ewma_weight) }}
{% endmacro %}
It’s still unclear to me why you’re making a distinction between “noisy” and “less noisy” sensors.
If the goal is to compute the average value of all sensors, whenever any of them changes its value, then simply trigger on state-changes (no need for periodic calculations). It doesn’t matter which sensors experience state-changes more or less frequently, the calculation is an average of all of their values.
The only reason to perform the calculation periodically (and not on state-changes) is if you want the average value to be less responsive to frequent state-changes. For example, if some sensor values change every second but you don’t want them reflected in the average value, then it might be preferable to trigger periodically and not on state-changes.
That alone will cause spikes when a sensor goes offline for a brief moment (as it will then not be included in the average). So my prior solution was to use that filter and then to apply a smoothing algorithm on that. But this leads to having a interim sensor which I don’t use. I therefore adapted the template sensor smoothing code to also handle multiple sensors as input.
A EWMA (exponential weighted moving average) only keeps a single prior value in memory and weighs the new measurement based on tau and the delta in time between the prior and current measurement. So a noisy sensor would trigger this recalculation multiple times per minute, another sensor might only trigger it once every 15 minutes. This gives the noisy sensor much more relevance. Here’s a graphical example of two outdoor temperature sensors that can handle rain and low temperatures. One is the Eve Weather (Homekit), the other Inkbird IBS-TH2 (BLE). The Eve Weather occasionally goes unavailable, the Inkbird is noisy:
I pair these two sensors with two internet based weather providers, giving me a total of 4 sensors. I expect at least 2 to be available all the time, usually 3 if not all 4. But as frequently one or two drop out, I need to smooth the spikes.
If you can suggest an alternative algorithm that will average and smooth measurements, I’d happily implement it as alternative.
You can configure the State Trigger to not trigger when one of its monitored entities changes its state to unavailable (and/or unknown). It’s done using the not_to option. That’ll ensure the average is only computed for valid numeric state-changes.
Alternatively you can report the Template Sensor’s existing value if one of the sensor’s values is non-nominal.
Or you can use the availability option to disallow evaluating the state template unless all sensor values are nominal. The Template Sensor’s reported value will be unavailable but that’s simply ignored when Home Assistant graphs historical data.
Lots of options for ensuring calculations are performed only when correct data is available (without the use of periodic triggering).
FWIW, additional processing can be performed with the Filter and Statistics integrations.
Sorry for the confusion. My goal is redundancy. I have 4 sensors (two local, two internet based) to ensure I always have a outdoor temperature available which I can use to control e.g. a glasshouse heating system. Plants shouldn’t die in a cold night just because a sensor goes offline and heating energy shouldn’t be wasted when the temperature is warm enough.
So you’re saying I could use the built in filters to a) only update at numeric state changes, b) compute an average of the currently available sensors and c) compute a moving average of the measurements of the past x minutes? All in a single step, without sensors inbetween these steps (e.g. one does the averaging, then another sensor for the smoothing)? Because then in fact I really wouldn’t need my own template sensor and EWMA macro.
At this point in discussion, you have received a substantial amount of information and advice so the next step is to explore it and use whatever suits your needs.
Before writing my own template sensor I did try to solve the issue with the filter integration but failed to find an acceptable solution. I wouldn‘t ask for help here without having done my homework first.
Good to know. Not everyone is as diligent so it’s challenging to determine what they have or haven’t tried beforehand.
With the application of YAML Anchors and aliases, and the use of script variables in both the state and availability options, at least half of your requirements were addressed.
Certainly. I’m just confused as you suggested I wouldn’t need this template sensor at all as I could implement it with built in integrations instead. As failing to do so led me to writing this template sensor a few months ago, I wonder what I was missing. I cannot see a way to average multiple sensors and smooth the result within a single sensor and also not act on nun-number state changes. Based on my attempts, this isn’t possible, making this template sensor the viable option instead.
I said “additional processing” can be performed by those other integrations. In other words, the Template Sensor can perform basic data gathering/averaging and other integrations can be used to process it even more.
I didn’t suggest completely replacing the Template Sensor because I don’t know if either of the other two integrations can fulfill all your requirements. For example, the Statistics integration offers several ways to process an entity’s data but it can only do so for one entity (fails to meet your need to monitor multiple entities).
