Template to add values of all sensors with certain name ending

[pnbruckner (Phil)]

Ok, I figured it out. I tried a similar script on my system and added more logging. Turns out += doesn’t work in the sandboxed environment python_script’s run in. Argh! So, this should work:

total_power = 0
count = 0
for entity_id in hass.states.entity_ids('sensor'):
    if entity_id.endswith('actueel'):
        state = hass.states.get(entity_id)
        logger.debug('entity_id = {}, state = {}'.format(entity_id, state.state))
        try:
            total_power = total_power + float(state.state)
        except:
            pass
        else:
            count = count + 1
hass.states.set(
    'sensor.total_sensors', total_power, {
        'unit_of_measurement': 'Watt',
        'count': count})

[Mariusthvdb (Marius)]

@Mariusthvdb

{{ states.sensor|selectattr( ‘entity_id’,‘in’,state_attr(‘group.iungo_verbruik_actueel’,‘entity_id’)) |map(attribute=‘state’)|map(‘int’)|sum }}

bingo!

now add the count to the jinja…and we’re off! so cool this is,
a huge thx!

[pnbruckner (Phil)]

Next you’re going to need a way to run this script (let’s call it python_script.update_total_sensors for this example.) I think something like this should work for you:

- alias: Update sensor.total_sensors
  trigger:
    platform: event
    event_type: state_changed
    event_data:
      new_state:
        domain: sensor
  condition:
    condition: template
    value_template: "{{ trigger.event.data.entity_id.endswith('actueel') }}"
  action:
    service: python_script.update_total_sensors

EDIT: Added further filtering to only run the script if the domain is sensor. Not 100% sure this will work. If not, then the domain filtering can definitely be done in the template condition.

[Mariusthvdb (Marius)]

yes, thought about that.
the thing is, there’s a continuous state changing going on, so i fear the automation would hog the system.
I have several other python scripts that ‘listen’ to state changes, and now settled on running them time based;

- alias: 'Timed Python scripts per 20 secs'
  id: 'Timed Python scripts per 20 secs'
#  initial_state: 'on'
  trigger:
    platform: time
    seconds: '/20'
  action:
    - delay: 00:00:02
    - service: python_script.whats_on
    - delay: 00:00:02
    - service: python_script.summary

I’ll test the listener script for state_change first, and if it doesn’t compromise leave it dedicated like that. If not, i’ll add another time based script for 10 sec, or simply add the script to the existing automation.

2 details left for now…
how can we round the sensor to 2 decimals? tried total_power(round(2)) and round(total_power,2) but this errors out

how can i add the count of counted sensors tot the jinja template?

[pnbruckner (Phil)]

@Mariusthvdb

there’s a continuous state changing going on

There are continuous state changes going on all the time anyway. Every second there is a time changed event. And with every state change the system has to do some amount of processing to figure out which triggers have fired. I would write the automation the way that makes the most sense. To me that’s with the event trigger the way I wrote it.

Also to consider, with my trigger/condition, sensor.total_sensors will be updated immediately whenever one of the sensors changes that goes into it’s state. Whereas if you run the script every 20 seconds you’ll have up to a 20 second delay, and you’ll be running the script many times where it doesn’t need to run. To me that seems like a bigger, unnecessary load on the system.

@Mariusthvdb

how can we round the sensor to 2 decimals? tried total_power(round(2)) and round(total_power,2) but this errors out

Where? In a template, or in the python_script. If the latter, the correct syntax would be round(total_power, 2). However, like I said, these scripts are run in a fairly restricted/limited environment, and it could be that the round function is not available. Since a state is always a string anyway, if that doesn’t work then maybe you could have the state argument be '{:.1f}'.format(total_power) instead.

BTW, when something “errors out”, it’s helpful to include the error message. Often that gives clues to why it doesn’t work.

@Mariusthvdb

how can i add the count of counted sensors tot the jinja template?

What jinja template? In any case, try {{ state_attr('sensor.total_sensors', 'count') }}.

[Mariusthvdb (Marius)]

@pnbruckner

There are continuous state changes going on all the time anyway. Every second there is a time changed event. And with every state change the system has to do some amount of processing to figure out which triggers have fired. I would write the automation the way that makes the most sense. To me that’s with the event trigger the way I wrote it.

Also to consider, with my trigger/condition, sensor.total_sensors will be updated immediately whenever one of the sensors changes that goes into it’s state. Whereas if you run the script every 20 seconds you’ll have up to a 20 second delay, and you’ll be running the script many times where it doesn’t need to run. To me that seems like a bigger, unnecessary load on the system.

i completely agree with you, and thats the way i had my script triggered in the first place. But ever since the Hue implementation changed to asyncio, I had disconnects from the Hue hub, whenever a listener to the lights was active. Both my python_scripts checked the light states…

see: All my hue lights randomly becoming unavailable

maybe you have thoughts on that?

in the end I settled for timed automations, and the Hue lights have calmed down disconnecting , albeit still not perfect…

Ill start with your automation though, since this is no lights listener, I shouldn’t do any harm there.

@pnbruckner

Where? In a template, or in the python_script. If the latter, the correct syntax would be round(total_power, 2). However, like I said, these scripts are run in a fairly restricted/limited environment, and it could be that the round function is not available. Since a state is always a string anyway, if that doesn’t work then maybe you could have the state argument be '{:.1f}'.format(total_power) instead.

round(total_power,2) works after all, I forgot a ‘,’ , hence the syntax error I got… thx

@pnbruckner

'{:.1f}'.format(total_power)

what does this mean exactly?

@pnbruckner

What jinja template? In any case, try {{ state_attr('sensor.total_sensors', 'count') }} .

the one that started it all…

{{ states.sensor|selectattr( 'entity_id','in',state_attr('group.iungo_verbruik_actueel','entity_id'))
   |map(attribute='state')|map('int')|sum }}

so now i use:

total_power = 0
count = 0
for entity_id in hass.states.entity_ids('sensor'):
    if entity_id.endswith('actueel'):
        state = hass.states.get(entity_id)
        logger.debug('entity_id = {}, state = {}'.format(entity_id, state.state))
        try:
            total_power = total_power + float(state.state)
        except:
            pass
        else:
            count = count + 1
hass.states.set(
    'sensor.total_sensors', round(total_power,2), {
        'unit_of_measurement': 'Watt',
        'count': count})

which would be the best place in the loop to have the condition checked for state being >0?

this seems to be working, not sure if its correct 100%, please have a look?

total_power = 0
count = 0
for entity_id in hass.states.entity_ids('sensor'):
    if entity_id.endswith('actueel'):
        state = hass.states.get(entity_id)
        if state.state > '0':
            logger.debug('entity_id = {}, state = {}'.format(entity_id, state.state))
            try:
                total_power = total_power + float(state.state)
            except:
                pass
            else:
                count = count + 1

adding to the ninja’s, this give a nice set of monitoring instruments

{{ states.sensor|selectattr( 'entity_id','in',state_attr('group.iungo_verbruik_actueel','entity_id'))
   |map(attribute='state')|map('int')|sum }}

{%-for state in states.sensor 
   if  state.entity_id.endswith('actueel') and state.state >'0' %}
{{state.name}} : {{state.state}}
{%-endfor%}

{{ state_attr('sensor.total_sensors', 'count') }}

so cool this templating, learning each day, thx!

[pnbruckner (Phil)]

@Mariusthvdb

'{:.1f}'.format(total_power)

what does this mean exactly?

See format strings. Basically it means format as a float with one decimal place.

@Mariusthvdb

the one that started it all

I’m confused. I thought the python_script was replacing that template.

@Mariusthvdb

which would be the best place in the loop to have the condition checked for state being >0?

total_power = 0
count = 0
for entity_id in hass.states.entity_ids('sensor'):
    if entity_id.endswith('actueel'):
        state = hass.states.get(entity_id)
        logger.debug('entity_id = {}, state = {}'.format(entity_id, state.state))
        try:
            power = float(state.state)
        except:
            continue
        if power > 0:
            total_power = total_power + power
            count = count + 1
hass.states.set(
    'sensor.total_sensors', round(total_power,2), {
        'unit_of_measurement': 'Watt',
        'count': count})

[Mariusthvdb (Marius)]

@Mariusthvdb

{{ states.sensor|selectattr( ‘entity_id’,‘in’,state_attr(‘group.iungo_verbruik_actueel’,‘entity_id’)) |map(attribute=‘state’)|map(‘int’)|sum }} {%-for state in states.sensor if state.entity_id.endswith(‘actueel’) and state.state >‘0’ %} {{state.name}} : {{state.state}} {%-endfor%} {{ state_attr(‘sensor.total_sensors’, ‘count’) }}

there;s 1 entity i need to exclude: ‘group.iungo_verbruik_actueel’

thought to do that this way, but now the automation doesn’t trigger at all. What am I doing wrong here??

- alias: Update sensor.total_power
  trigger:
    platform: event
    event_type: state_changed
    event_data:
      new_state:
        domain: sensor
  condition:
    condition: template
    value_template: >
      {{ trigger.event.data.entity_id.endswith('actueel') and
      trigger.event.data.entity_id != 'group.iungo_verbruik_actueel'}}
  action:
    service: python_script.total_power

–edit–
nevermind, its for domain sensor… duh, no group is checked at all…still, the automation should have run?
–end edit----

under event_data you have added the new_state:
what does that do? I have several other state change listeners, but they don’t use the new_state. Isn’t a state change per definition a new state…?

[pnbruckner (Phil)]

@Mariusthvdb

under event_data you have added the new_state:
what does that do? I have several other state change listeners, but they don’t use the new_state . Isn’t a state change per definition a new state…?

See here and here.

As I said, I wasn’t sure if that trigger would work. So try this instead:

- alias: Update sensor.total_sensors
  trigger:
    platform: event
    event_type: state_changed
  condition:
    condition: template
    value_template: >
      {{ trigger.event.data.entity_id.startswith('sensor.') and
         trigger.event.data.entity_id.endswith('actueel') }}
  action:
    service: python_script.update_total_sensors

[Mariusthvdb (Marius)]

@pnbruckner

See here and here.

As I said, I wasn’t sure if that trigger would work. So try this instead:

yes thanks, I’ve checked that, but it doesn’t explain thoroughly what the new_state is for, or when that triggers though.

Ive changed the former into this state change listener, only left out the new_state, and now it works perfectly:

- alias: 'Update sensor.total_power'
  id: 'Update sensor.total_power'
#  hide_entity: True
#  initial_state: 'on'
  trigger:
    platform: event
    event_type: state_changed
    event_data:
      domain: sensor
  condition:
    condition: template
    value_template: >
      {{ trigger.event.data.entity_id.endswith('actueel')}}
  action:
    service: python_script.total_power

will copy your second version anyhow cause that might come in very conveniently in other situations also, so thx again!

btw this sensor only gets a state when started and never updates:

totaal_actueel_sensors:
  friendly_name: Totaal actueel
  unit_of_measurement: 'Watt'
  value_template: >
    {{ states.sensor|selectattr( 'entity_id','in',state_attr('group.iungo_actueel_verbruik','entity_id'))
    |map(attribute='state')|map('int')|sum }}

while this one does:

count_totaal_actueel_sensors:
  friendly_name: Count sensors actueel
  unit_of_measurement: 'Count'
  value_template: >
    {{ state_attr('sensor.total_sensors', 'count') }}

any reason why that would be? would it be the fact thats it triggers from the group, and the group itself doesn’t change its group members. If so, id need a way to change that filter to point to the sensors like tried before, or it simply won’t work ever…

[pnbruckner (Phil)]

@Mariusthvdb

yes thanks, I’ve checked that, but it doesn’t explain thoroughly what the new_state is for, or when that triggers though.

Ive changed the former into this state change listener, only left out the new_state, and now it works perfectly

Every state change has a from/old state and a to/new state. That’s what makes it a change. E.g., with the state trigger, you can specify from: and/or to:. If you look at a state_changed event recording in the log, you’ll see it calls the two states old_state and new_state. Also, in conditions and actions you can use trigger.from_state and trigger.old_state. They’re all pretty much the same thing, just different ways of referring to them. (The main difference is from: and to: specify state strings, whereas the others are state objects.)

The fact that it works when you remove new_state: is interesting. I’ll have to look at the code to see how this works because that’s a bit surprising.

Regarding the updating, or lack thereof, of the two ways of showing the count, yes, basically template sensors only update when an entity it references updates. In the case of the group, it will not necessarily change state just because an entity within the group changes. This group’s entities’ states are numbers, so the group will probably never have a state change (other than at startup.) And since that’s the only entity directly referenced in the template, it will never change. You can explicitly list entities that will cause the template sensor to update, so you could list all the entities of the group, but then that kind of defeats the purpose of having the group in the first place. You could also list sensor.time (if you have it configured), but that will only force the template sensor to update every minute (because sensor.time changes every minute.) But, again, the python_script and the new count template sensor were created to replace that first template sensor, so you should just be able to get rid of it now, right?

[Mariusthvdb (Marius)]

@pnbruckner

But, again, the python_script and the new count template sensor were created to replace that first template sensor, so you should just be able to get rid of it now, right?

indeed , and yes those 2 would suffice. Good to understand why there is no updating.

But since we successfully use the endswith filter now, I thought it might be possible to use it in this sensor value_template too somehow. I thought that might be an option since we’re not using a loop here, and, since you’ve successfully created a value_template using the endswith in the automation’s condition:

 value_template: >
      {{ trigger.event.data.entity_id.startswith('sensor.') and
         trigger.event.data.entity_id.endswith('actueel') }}

totaal_actueel_sensors:
  friendly_name: Totaal actueel
  unit_of_measurement: 'Watt'
  value_template: >
    {{ states.sensor|selectattr( 'entity_id','endswith','actueel')
    |map(attribute='state')|map('int')|sum }}

unfortunately this errors out: Error rendering template: TemplateRuntimeError: no test named 'endswith'
would be cool…probably my imagination taking the better of me again…?

or:

{{ states.sensor |map(endswith='actueel')|map(attribute='state')|map('int')|sum }}

error: Error rendering template: FilterArgumentError: map requires a filter argument

or take the endswith out of the filter mappings and try it as a condition:

{% if states.entity_id.endswith('actueel') %}
{{ states.sensor |map(attribute='state')|map('int')|sum }}
{%endif%}

would be nice too… but nothing happens (of course…) anyways, you get my drift…

something else:

this is a very nice template:

map_totaal_actueel_sensors:
  friendly_name: Overzicht sensors actueel
  unit_of_measurement: 'Watt'
  value_template: >
    {%-for state in states.sensor 
      if state.entity_id.endswith('actueel') and state.state >'0' %}
    {{state.name}} : {{state.state}}
    {%-endfor%}
  {{ '-'.ljust(30, '-') }} {{ '-'.ljust(30, '-') }}

based on @skalavala 's nrs 1 and 4 jinja, but there no way I can get it to show in the frontend (max 255 character count), and even with the custom-ui state card value_only, which i use in other places with sensors or log file like this, it doesn’t show up, either giving the Unknown, or stopping the system in a permanent loop it seems.

Could you see why that is?

[pnbruckner (Phil)]

@Mariusthvdb

unfortunately this errors out: Error rendering template: TemplateRuntimeError: no test named 'endswith'
would be cool…probably my imagination taking the better of me again…?

Would be nice, and I even looked for something like that, but as you can see here there is no such test available.

@Mariusthvdb

Could you see why that is?

Yes, state strings are limited to 255 characters. So you’d have to compact it until it was no more than that. Still, I don’t see how this would stop the system.

BTW, state.state >'0' - that is doing a string comparison, not a number comparison. That will not work in many cases. E.g., '0.0' > '0' is True. But '0.0'|float > 0 is False.

[Mariusthvdb (Marius)]

@pnbruckner

BTW, state.state >'0' - that is doing a string comparison, not a number comparison. That will not work in many cases. E.g., '0.0' > '0' is True. But '0.0'|float > 0 is False.

ah, thats why my stove reporting 0.0 is still showing…
cool: this does the trick: state.state|float > 0 !

still the 255 limit should be circumvented by the custom card. Somehow it doesn’t work here. Ive set the customization:

sensor.map_totaal_actueel_sensors:
  custom_ui_state_card: state-card-value_only

this will help: {{state.name[:-8]}} : {{state.state}}

[Mariusthvdb (Marius)]

@pnbruckner

Yes, state strings are limited to 255 characters. So you’d have to compact it until it was no more than that. Still, I don’t see how this would stop the system.

well it does…
The system startsup, I can see all my startup notifications but cant reach the frontend in the browser nor app.
I’ve compacted it by adding the {{state.name[:-8]}} : {{state.state}}

this is flooding the log, apparently hogging the system.

2018-08-19 16:47:04 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 282, in async_update_ha_state
    self.entity_id, state, attr, self.force_update, context)
  File "/usr/local/lib/python3.6/site-packages/homeassistant/core.py", line 851, in async_set
    context)
  File "/usr/local/lib/python3.6/site-packages/homeassistant/core.py", line 622, in __init__
    "State max length is 255 characters.").format(entity_id))
homeassistant.exceptions.InvalidStateError: Invalid state encountered for entity id: sensor.map_totaal_actueel_sensors. State max length is 255 characters.
ountered for entity id: sensor.map_totaal_actueel_sensors. State max length is 255 characters.
ttr, self.force_update, context)
  File "/usr/local/lib/python3.6/site-packages/homeassistant/core.py", line 851, in async_set
    context)
  File "/usr/local/lib/python3.6/site-packages/homeassistant/core.py", line 622, in __init__
    "State max length is 255 characters.").format(entity_id))
homeassistant.exceptions.InvalidStateError: Invalid state encountered for entity id: sensor.map_totaal_actueel_sensors. State max length is 255 characters.
.map_totaal_actueel_sensors. State max length is 255 characters.
t)
  File "/usr/local/lib/python3.6/site-packages/homeassistant/core.py", line 851, in async_set
    context)
  File "/usr/local/lib/python3.6/site-packages/homeassistant/core.py", line 622, in __init__
    "State max length is 255 characters.").format(entity_id))

[pnbruckner (Phil)]

Well that sounds like a bug to me. Sure, it’s an error for an entity’s state string to be longer than 255 characters, but it shouldn’t have that effect.

[klogg]

I am following this with interest it is a bit of an education
However is it possible that this…

@pnbruckner

Sure, it’s an error for an entity’s state string to be longer than 255 characters

…is related to my error here?

Help please understanding an error

I am getting the following error which seems to be giving some very useful information but does anyone know precisely what it means so that I can trouble shoot and prevent it? Error sending message: Can't parse entities: can't find end of the entity starting at byte offset 284. Args: (REDACTED1, "REDACTED2"), kwargs: {'parse_mode': 'Markdown', 'disable_notification': False, 'disable_web_page_preview': None, 'reply_to_message_id': None, 'reply_markup': None, 'timeout': None} It is generated by …

The sensor in question for me is the custom component Places. I only ask because the error refers to a ‘byte offset 284’.

(I don’t mean to hijack this thread, I will not continue this here but if you think it might be connected perhaps you could respond in my original thread. Thanks).

[Mariusthvdb (Marius)]

i fear for you this has nothing to do with each other. My 255 character error is a known issue, where the state of the sensor is 255+ characters, which the system does not allow.

Im trying to overcome that with a dedicated ‘text-only’ custom card, but am lost for the moment how to…

sorry, this doesn’t help you…

[Mariusthvdb (Marius)]

maybe we can overcome this too using the Python.
I need an attribute ‘text’ with the outcome of the sensor as value. So where the regular jinja template sensor is:

map_totaal_actueel_sensors:
  friendly_name: Overzicht sensors actueel
  unit_of_measurement: 'Watt'
  value_template: >
    {%-for state in states.sensor 
    if state.entity_id.endswith('actueel') and state.state|float > 0 %}
    {{state.name[:-8]}}: {{state.state}}
    {%-endfor%}
    {{ '-'.ljust(30, '-') }} {{ '-'.ljust(30, '-') }}
   'Count sensors :' {{ state_attr('sensor.total_sensors', 'count') }} 'totaal: ' {{ states.sensor|selectattr( 'entity_id','in',state_attr('group.iungo_actueel_verbruik','entity_id'))
    |map(attribute='state')|map('int')|sum }} 

id need that to be calculated in Python, and be the value for text.

I can then create the sensor in Python as follows:

hass.states.set(
‘sensor.map_totaal_actueel_sensors’, ‘’, {
‘custom_ui_state_card’: ‘state-card-value_only’,
‘text’: outcome-of-template-here
})

I think an attribute ‘text’ cant be set in a regular sensor? If so that would be even easier to try.

Would you please have another go for me? Seems we have almost all already, the count, the total power usage, and the states.

Only thing left in this nicely printed overview is the formatting.

 ##########################################################################################
# map_total_sensors.py
# reading all sensors.*_actueel using power (>0), listing and counting them, and
# calculate summed power consumption
# by @Mariusthvdb  and big hand Phil, @pnbruckner
##########################################################################################
sensor_list = []
total_power = 0
count = 0
for entity_id in hass.states.entity_ids('sensor'):
    if entity_id.endswith('actueel'):
        state = hass.states.get(entity_id)
        logger.debug('entity_id = {}, state = {}'.format(entity_id, state.state))
        sensor = '{} : {}'.format(entity_id, state.state)
        try:
            power = float(state.state)
        except:
            continue
        if power > 0:
            total_power = round((total_power + power),2)
            count = count + 1

            sensor_list.append(sensor)

summary = '*============== Sensors ==============\n' \
          '----- Sensor ------------ Power -----\n' \
          '${}\n' \
          '*=============== Sumary ==============\n' \
          '$  Sensors    {}  :  Total Power  {}\n' \
          '*=====================================\n' \
          .format(sensor_list,
                  count,
                  total_power)

hass.states.set('sensor.map_total_sensors', '', {
        'custom_ui_state_card': 'state-card-value_only',
        'text': summary
        })
##########################################################################################
# map_total_sensors.py
##########################################################################################

showing as:

quite the result already, and updating immediately, both per sensor, as the total.

changed is somewhat to do away with the opening and closing list brackets, and the apostrophes around each sensor. As often the case, one needs intermediate variables, and a bit of playing with join, append and format…

This way each sensor gets and stays on its own line too, which is very nice when changing the browser window. (before it filled out, since the sensor section was seen as 1 entity.)

changed code:

sensor_list = []
total_power = 0
count = 0
for entity_id in hass.states.entity_ids('sensor'):
    if entity_id.endswith('actueel'):
        state = hass.states.get(entity_id)
        logger.debug('entity_id = {}, state = {}'.format(entity_id, state.state))
        #sensor_id = '{}'.format(entity_id)
        #sensor_power = '{}'.format(state.state)
        #sensor = '{}   -   {}\n'.format(sensor, senor_power)
        sensor = '{}   -   {}\n'.format(entity_id[7:-8], state.state)
        try:
            power = float(state.state)
        except:
            continue
        if power > 0:
            total_power = round((total_power + power),2)
            count = count + 1

            sensor_list.append(sensor)

sensor_map = '\n'.join(sensor_list)

summary = '*============== Sensors ==============\n' \
          '$----- Sensor ------------ Power -----\n' \
          '{}\n' \
          '*=============== Sumary ==============\n' \
          '$  Sensors    {}  :  Total Power  {}\n' \
          '*=====================================\n' \
          .format(sensor_map,
                  count,
                  total_power)

Of course ill need to reformat the rest of the map too, using some of the color options in the custom card, and better justification, as in the original jinja template.

[pnbruckner (Phil)]

Looking good. Do you still need any help, with formatting, or whatever, or are you good now?