Template sensor check if string is NULL

[Robufz]

I have the following serial device strings

"PRESS,,f,65,M,,F*3F"
"PRESS,,f,,M,,F*3F"

and the following YAML

- trigger:
    - platform: state
      entity_id: sensor.serial_gauge
    - platform: time_pattern
      minutes: "/10"
  sensor:
    - name: pressure
      unique_id: pressure_id
      state: >-
        {% set d = trigger.to_state.state.split(',') %}
        {{ d[3] if d | count > 3 and d[0] == 'PRESS' else this.state }}

the code works great with the 1st string
BUT the code aborts if d[3] is null like in the 2nd string

So I tried this

- trigger:
    - platform: state
      entity_id: sensor.serial_gauge
    - platform: time_pattern
      minutes: "/10"
  sensor:
    - name: pressure
      unique_id: pressure_id
      state: >-
        {% set d = trigger.to_state.state.split(',') %}
        {{ d[3] if (d | count > 3 and d[0] == 'PRESS' and d[3] is not none)  else this.state }}

But it does not work.
How can I stop this trigger aborting if d[3] is null ??

[Didgeridrew]

@Robufz

and d[3] is not none

and d[3] != ''

But your time pattern trigger won’t have a to_state, so the sensor is going to error out every 10 minutes.

[Robufz]

@Didgeridrew

and d[3] != ‘’

Ok. I will try this as well.

@Didgeridrew

But your time pattern trigger won’t have a to_state, so the sensor is going to error out every 10 minutes.

Why is that? It has been working but I would like to know your reasoning.
I put the time pattern in because if a sensor goes offline, ie. No data is received, then it will keep its last value. In this case I want it to report unavailable. And the time pattern I presume forces it to evaluate and the result will be unavailable.

[Didgeridrew]

@Robufz

Why is that? It has been working but I would like to know your reasoning.

The Available Trigger Data for a Time Pattern trigger includes the following:

Available variables
trigger.id	The id of the trigger.
trigger.idx	Index of the trigger. (The first trigger idx is 0.)
trigger.platform	Hardcoded: time_pattern
trigger.now	DateTime object that triggered the time_pattern trigger.

If you check your logs, you will likely find an “Undefined Error” every 10 minutes because trigger.to_state is undefined.

Every firing of the time pattern trigger will set the sensor’s value to “unavailable”. In the following I used an analog to the sensor as you have it configured except I switched it to a 3 minute interval so the pattern is more apparent:

The longer gaps are when I followed a time pattern trigger with a input where d[3] was blank… If your serial device is streaming data in regularly enough you may not notice the little blips of “unavailable”, but they’re likely there.

@Robufz

I put the time pattern in because if a sensor goes offline, ie. No data is received, then it will keep its last value.

You could guard against that with a more complex template, but it’s not necessary.
With a properly configured State trigger, when no data is received from the serial gauge, the trigger will not fire, and the sensor keeps it’s last value anyway. One thing you could do to make sure your state trigger doesn’t cause issues is to reject “unavailable” and “unknown” as follows:

- trigger:
    - platform: state
      entity_id: sensor.serial_gauge
      not_to:
        - unknown
        - unavailable

In any case, the time pattern trigger isn’t doing anything except adding bad data.

[Robufz]

@Didgeridrew

If your serial device is streaming data in regularly enough you may not notice the little blips of “unavailable”, but they’re likely there.

You are absolutely correct! Thank you. So basically I am setting all of my sensors, I have 15, to be unavailable every 10 minutes just to catch out the sensor that does not report data anymore. However to keep reporting the last known value is not acceptable for me. How do you suggest I can detect if a sensor stops reporting data?

@Didgeridrew

- trigger:
    - platform: state
      entity_id: sensor.serial_gauge
      not_to:
        - unknown
        - unavailable

I am not sure how this will work as it will only stop firing the trigger if sensor.serial_gauge is Unknown or Unavailable. In my case there will always be data, but only some fields my be null.

[Didgeridrew]

@Robufz

… However to keep reporting the last known value is not acceptable for me.

I must have misunderstood your goal. For clarity, what was the purpose of the this.state in your original if not to keep reporting last known value as current value?

@Robufz

… How do you suggest I can detect if a sensor stops reporting data?

… In my case there will always be data, but only some fields may be null.

If the state of the gauge sensor being “unavailable” or “unknown” is not sufficient, what other criteria need to be considered to define “sensor stops reporting data”? For example, if d[3] is null for 1 minute ( or 5, 10, 20…), should that be considered reporting or not?

How do you want to be shown that the gauge sensor is not reporting? You can use further templating to set the value of attributes. Or you could set an availability template for sensor.pressure… we need to know what the goal is.

[Robufz]

@Didgeridrew

For clarity, what was the purpose of the this.state in your original if not to keep reporting last known value as current value?

The entity_id: sensor.serial_gauge receives information and I create 15 different sensors based on this information.So in my original code, under this 1 trigger I have 15 sensors defined. Each new line is information about an individual sensor based on the text value of d[0]. In the example 1 sensor is “PRESS”. But I have 14 others. So each of these 15 sensors get triggered each time and I uses the this.state to keep the same value in the case when d[0] does not match. Otherwise it will lose its value. I hope this explanation makes sense. Please tell me if there is another way to achieve this.

@Didgeridrew

If the state of the gauge sensor being “unavailable” or “unknown” is not sufficient,

Unavailable and Unknown is very sufficient. This is there reason I used a time pattern. But If a sensor, for example the “PRESS” one stops reporting the trigger will never have a chance to to set it to Unavailable or Unknown because d[0] will never match “PRESS” again as it is not being reported.

say my serial data is

"PRESS,,f,65,M,,F*3F"
"WEGT",12,1,F,5,4F
"PRESS,,f,60,M,,F*3F"
"WEGT",14,1,F,5,4F
"WEGT",11,1,F,5,4F
"WEGT",12,1,F,5,4F

Here "PRESS" has stopped reporting  but this.state will stay as the last reported value "60"
But here I want to report PRESS as unavailable.

Much appreciated your help. 

[Didgeridrew]

So how do we detect when something doesn’t happen…?

Do the “topics” always come in the same order i.e. does PRESS always follow WEGT in normal operation? If so, you could do something like the following:

- trigger:
    - platform: state
      entity_id: sensor.serial_gauge
  sensor:
      - name: pressure_2
        unique_id: pressure_2_id
        state: >-
          {% set current = this.state | default(0) %}
          {% set d = trigger.to_state.state.split(',') %}
          {% if trigger.to_state is search('WEGT') and not trigger.from_state.state is search('PRESS') %}
          ERROR
          {% else %}
          {{ d[3] if (d | count > 3 and d[0] == 'PRESS' and d[3] != '') else current }}
          {% endif %}

[Robufz]

@Didgeridrew

Do the “topics” always come in the same order

Unfortunately no. The serial device is a combination of other devices on a communication bus. So there is no control of the order. The only thing certain is that each device reports at least every 30 seconds.

I am really stumped to find a way to solve this. At worst I will need to live with a device offline still showing its last known value. But this is far from ideal.

Is there a way I can use the time pattern that I had but stop the sensor failing momentarily?
Maybe I can set a timestamp attribute for each sensor and if it is more than 60 Secs old then the gauge would show Unavailable? Is this actually possible in YAML?

I spent 20 years of my life coding but trying to solve problems with YAML drives me a bit crazy!

Any other suggestions? Many Thanks.

[Didgeridrew]

@Robufz

Is there a way I can use the time pattern that I had but stop the sensor failing momentarily?

It’s possible to pull the current state of the serial sensor when the time pattern trigger fires, but the issue becomes that, as long as it’s one of the normal strings, there’s no way to know whether the state that is captured at that moment indicates a failure of a specific sensor.

@Robufz

I am really stumped to find a way to solve this. At worst I will need to live with a device offline still showing its last known value. But this is far from ideal.

… Maybe I can set a timestamp attribute for each sensor and if it is more than 60 Secs old then the gauge would show Unavailable? …

Any other suggestions?

I think you’re going to have to go with a timestamping approach and maybe a binary sensor that indicates errors either for each individual sensor or the group… here’s a rough draft I came up with (just using the value sequence you provided earlier).

template:
  - trigger:
      - platform: state
        entity_id: sensor.serial_gauge
        variables:
          d: "{{ trigger.to_state.state.split(',') }}"
    sensor:
     - name: pressure
        unique_id: pressure_id
        device_class: pressure
        state: |-
          {{ d[3] if (d | count > 3 and d[0] == 'PRESS' and d[3] != '')  else this.state }}
        attributes:
          timestamp: |-
            {{ now() if d[0] == 'PRESS' else this.attributes.timestamp | default(now()) }}

      - name: weight
        unique_id: weight_id
        device_class: weight
        state: |-
          {{ d[1] if (d[0] == 'WEGT' and d[1] != '') else this.state }}
        attributes:
          timestamp: |-
            {{ now() if d[0] == 'WEGT' else this.attributes.timestamp | default(now()) }}

  - binary_sensor:
      - name: Serial Sensor Error
        state: |-
          {{ expand('sensor.weight', 'sensor.pressure_3')
          | selectattr('attributes.timestamp', 'defined')
          | map(attribute='attributes.timestamp')
          | map('as_datetime') | select('lt', now() - timedelta(seconds=45))
          | list | count > 0 or not has_value('sensor.serial_gauge') }}

It seems to mostly work…

FWIW, you can get a line graph and statistical output by adding unit_of_measurement: and state_class: measurement, respectively, to the numeric sensors. I left it off to make it easier to see all three together in the History tool.

[Robufz]

Brilliant. Thanks. I will try this out in the morning.
And also the state_class tip.
Thanks also for showing the history logbook. I never released I could trace the yaml execution like this. I have been running blind.

You have a high level of understanding of yaml. How did you learn it and what resources are there available? I am ok but am using trial and error alot because I cannot find any good resources.

[Didgeridrew]

@Robufz

How did you learn it and what resources are there available? I am ok but am using trial and error alot because I cannot find any good resources.

Mostly, I learned from other members on this forum, the official docs, and various Youtube channels.

YAML basics:
YAML for Non-Programmers by Thomas Loven

Templating:
Home Assistant Templating : This is the the most comprehensive and up-to-date source for HA-specific functions.

Jinja Docs: These docs are not geared toward new/casual users, but they contain a lot of important information.

Jinja for Ninjas: @skalavala has put together a well organized tutorial and group of examples. It’s been a while since this was updated, so there are better/easier ways to do some of the things shown, but it’s still a good resource.

Since Jinja is built on Python, many Python methods also work in templates. I’ve never seen a complete list of which methods work and which don’t… But the Official Python docs have come in handy for me multiple times to figure out what’s going on in templates posted by some of the more advanced users on this forum.

[Robufz]

@Didgeridrew

and d[3] != ‘’

I have just spent a few hours going around in circles. But I have just proved something strange.

Basically if my template sensor specifies a unit_of_measurement: “Bar” then the trigger will fail if there is an unexpected null value. If I then put in the "and d[3] != " it will also fail but the sensors afterward will not be Unknown, they will just keep that same value forever.

BUT if remove the unit_of_measurement from the sensor that has a null value everything works well. I do not even need to perform the "and d[3] != " comparison. The trigger will not fail at that sensor and will continue evaluating the other sensors.

I repeated this 3 times. Put in unit_of_measurement and the following sensors are stuck with the same value. Remove the unit_of_measurement and it all works.

Is this a bug?

It is as if a null value is always an error if unit_of_measurement is specified. I have also tried different units like m and mph. Same result.

[Robufz]

@Didgeridrew

state_class: measurement

same if I also just put in
state_class: measurement

[Didgeridrew]

Working with the, admittedly limited, testing method I used previously I cannot reproduce what you have described. Post a screen shot of one of the sensor configurations that being problematic as well as any history or other info you think is pertinent.

This is what I’m getting:

It may not be readily apparent in the image, but I edited the sensor configuration and reloaded the integration throughout the test to include unit_of_measurement and state_class both individually and together. All three configurations showed up exactly the same.

[Robufz]

It is incredible.
I go to sleep and now the next day I cannot reproduce this either now. It now failing again on the null 3rd element. Yesterday it was working all fine.
I will try again. There seems to be something strange happening when there is a null value.
In your test did you put in a null value to replicate what I have?
eg.
“PRESS,f,65,M,F3F"
"PRESS,f,M,F3F”

The 2nd line has a null value in d[3]

[Robufz]

So now it is working again. All I did was re arrange the order of the sensors in the trigger. I put the one, waterdepth, that has a null value last so that the following 2 sensors work. And then I put the original code back with waterdepth first and it keeps working.
I restart HA each time via developer tools.
Is it possible restart does not clear all of the state values?

Here is the original code and which is working again even though a value is null.

- trigger:
    - platform: state
      entity_id: sensor.serial_instruments
      variables:
        d: "{{ trigger.to_state.state.split(',') }}"
  sensor:
    - name: waterdepth
      unique_id: water_depth_id
      state: >-
        {{ d[3] if (d | count > 3 and d[0] | regex_match('^\\$\\w{2}DBT') ) else this.state }}
    - name: watertemperature
      unique_id: water_temperature_id
      state: >-
        {{ d[1] if (d | count > 1 and d[0] | regex_match('^\\$\\w{2}MTW') ) else this.state }}
    - name: waterpressure
      unique_id: water _pressure_id
      state: >-
        {{ d[3] if (d | count > 3 and d[0] | regex_match('^\\$\\w{2}VHW') ) else this.state }}

[Robufz]

And now I put in a unit_of_measurement and it stops working. The following sensors are not reevaluated and are keeping their previous values.
Actually I think it has nothing to do with unit_of_measurement. It is probably some other state that is being disturbed by changing the code.

- trigger:
    - platform: state
      entity_id: sensor.serial_instruments
      variables:
        d: "{{ trigger.to_state.state.split(',') }}"
  sensor:
    - name: waterdepth
      unique_id: water_depth_id
      unit_of_measurement: "m"
      state: >-
        {{ d[3] if (d | count > 3 and d[0] | regex_match('^\\$\\w{2}DBT') ) else this.state }}
    - name: watertemperature
      unique_id: water_temperature_id
      state: >-
        {{ d[1] if (d | count > 1 and d[0] | regex_match('^\\$\\w{2}MTW') ) else this.state }}
    - name: waterpressure
      unique_id: water _pressure_id
      state: >-
        {{ d[3] if (d | count > 3 and d[0] | regex_match('^\\$\\w{2}VHW') ) else this.state }}

[Robufz]

And now I take out the unit_of_measurement and the following 2 sensors are reporting again. Well watertemperature is quite stable but the pressure is changing.

- trigger:
    - platform: state
      entity_id: sensor.serial_instruments
      variables:
        d: "{{ trigger.to_state.state.split(',') }}"
  sensor:
    - name: waterdepth
      unique_id: water_depth_id
      state: >-
        {{ d[3] if (d | count > 3 and d[0] | regex_match('^\\$\\w{2}DBT') ) else this.state }}
    - name: watertemperature
      unique_id: water_temperature_id
      state: >-
        {{ d[1] if (d | count > 1 and d[0] | regex_match('^\\$\\w{2}MTW') ) else this.state }}
    - name: waterpressure
      unique_id: water _pressure_id
      state: >-
        {{ d[3] if (d | count > 3 and d[0] | regex_match('^\\$\\w{2}VHW') ) else this.state }}

[Didgeridrew]

@Robufz

Actually I think it has nothing to do with unit_of_measurement. It is probably some other state that is being disturbed by changing the code.

It might be the combination… If the state value is non-numeric (except ‘unavailable’/‘unknown’ IIRC) it will cause an error with a sensor with a unit_of_measurement.