Template sensor check if string is NULL

Robufz · July 1, 2023, 9:19am

Unfortunately no. The serial device is a combination of other devices on a communication bus. So there is no control of the order. The only thing certain is that each device reports at least every 30 seconds.

I am really stumped to find a way to solve this. At worst I will need to live with a device offline still showing its last known value. But this is far from ideal.

Is there a way I can use the time pattern that I had but stop the sensor failing momentarily?
Maybe I can set a timestamp attribute for each sensor and if it is more than 60 Secs old then the gauge would show Unavailable? Is this actually possible in YAML?

I spent 20 years of my life coding but trying to solve problems with YAML drives me a bit crazy!

Any other suggestions? Many Thanks.

Didgeridrew · July 1, 2023, 4:53pm

It’s possible to pull the current state of the serial sensor when the time pattern trigger fires, but the issue becomes that, as long as it’s one of the normal strings, there’s no way to know whether the state that is captured at that moment indicates a failure of a specific sensor.

I think you’re going to have to go with a timestamping approach and maybe a binary sensor that indicates errors either for each individual sensor or the group… here’s a rough draft I came up with (just using the value sequence you provided earlier).

template:
  - trigger:
      - platform: state
        entity_id: sensor.serial_gauge
        variables:
          d: "{{ trigger.to_state.state.split(',') }}"
    sensor:
     - name: pressure
        unique_id: pressure_id
        device_class: pressure
        state: |-
          {{ d[3] if (d | count > 3 and d[0] == 'PRESS' and d[3] != '')  else this.state }}
        attributes:
          timestamp: |-
            {{ now() if d[0] == 'PRESS' else this.attributes.timestamp | default(now()) }}

      - name: weight
        unique_id: weight_id
        device_class: weight
        state: |-
          {{ d[1] if (d[0] == 'WEGT' and d[1] != '') else this.state }}
        attributes:
          timestamp: |-
            {{ now() if d[0] == 'WEGT' else this.attributes.timestamp | default(now()) }}

  - binary_sensor:
      - name: Serial Sensor Error
        state: |-
          {{ expand('sensor.weight', 'sensor.pressure_3')
          | selectattr('attributes.timestamp', 'defined')
          | map(attribute='attributes.timestamp')
          | map('as_datetime') | select('lt', now() - timedelta(seconds=45))
          | list | count > 0 or not has_value('sensor.serial_gauge') }}

It seems to mostly work…

FWIW, you can get a line graph and statistical output by adding unit_of_measurement: and state_class: measurement, respectively, to the numeric sensors. I left it off to make it easier to see all three together in the History tool.

Robufz · July 1, 2023, 11:23pm

Brilliant. Thanks. I will try this out in the morning.
And also the state_class tip.
Thanks also for showing the history logbook. I never released I could trace the yaml execution like this. I have been running blind.

You have a high level of understanding of yaml. How did you learn it and what resources are there available? I am ok but am using trial and error alot because I cannot find any good resources.

Didgeridrew · July 2, 2023, 4:23am

Mostly, I learned from other members on this forum, the official docs, and various Youtube channels.

YAML basics:
YAML for Non-Programmers by Thomas Loven

Templating:
Home Assistant Templating : This is the the most comprehensive and up-to-date source for HA-specific functions.

Jinja Docs: These docs are not geared toward new/casual users, but they contain a lot of important information.

Jinja for Ninjas: @skalavala has put together a well organized tutorial and group of examples. It’s been a while since this was updated, so there are better/easier ways to do some of the things shown, but it’s still a good resource.

Since Jinja is built on Python, many Python methods also work in templates. I’ve never seen a complete list of which methods work and which don’t… But the Official Python docs have come in handy for me multiple times to figure out what’s going on in templates posted by some of the more advanced users on this forum.

Robufz · July 2, 2023, 7:06pm

I have just spent a few hours going around in circles. But I have just proved something strange.

Basically if my template sensor specifies a unit_of_measurement: “Bar” then the trigger will fail if there is an unexpected null value. If I then put in the "and d[3] != " it will also fail but the sensors afterward will not be Unknown, they will just keep that same value forever.

BUT if remove the unit_of_measurement from the sensor that has a null value everything works well. I do not even need to perform the "and d[3] != " comparison. The trigger will not fail at that sensor and will continue evaluating the other sensors.

I repeated this 3 times. Put in unit_of_measurement and the following sensors are stuck with the same value. Remove the unit_of_measurement and it all works.

Is this a bug?

It is as if a null value is always an error if unit_of_measurement is specified. I have also tried different units like m and mph. Same result.

Robufz · July 2, 2023, 7:12pm

same if I also just put in
state_class: measurement

Didgeridrew · July 2, 2023, 8:28pm

Working with the, admittedly limited, testing method I used previously I cannot reproduce what you have described. Post a screen shot of one of the sensor configurations that being problematic as well as any history or other info you think is pertinent.

This is what I’m getting:

It may not be readily apparent in the image, but I edited the sensor configuration and reloaded the integration throughout the test to include unit_of_measurement and state_class both individually and together. All three configurations showed up exactly the same.

Robufz · July 3, 2023, 9:14pm

It is incredible.
I go to sleep and now the next day I cannot reproduce this either now. It now failing again on the null 3rd element. Yesterday it was working all fine.
I will try again. There seems to be something strange happening when there is a null value.
In your test did you put in a null value to replicate what I have?
eg.
“PRESS,f,65,M,F3F"
"PRESS,f,M,F3F”

The 2nd line has a null value in d[3]

Robufz · July 3, 2023, 9:28pm

So now it is working again. All I did was re arrange the order of the sensors in the trigger. I put the one, waterdepth, that has a null value last so that the following 2 sensors work. And then I put the original code back with waterdepth first and it keeps working.
I restart HA each time via developer tools.
Is it possible restart does not clear all of the state values?

Here is the original code and which is working again even though a value is null.

- trigger:
    - platform: state
      entity_id: sensor.serial_instruments
      variables:
        d: "{{ trigger.to_state.state.split(',') }}"
  sensor:
    - name: waterdepth
      unique_id: water_depth_id
      state: >-
        {{ d[3] if (d | count > 3 and d[0] | regex_match('^\\$\\w{2}DBT') ) else this.state }}
    - name: watertemperature
      unique_id: water_temperature_id
      state: >-
        {{ d[1] if (d | count > 1 and d[0] | regex_match('^\\$\\w{2}MTW') ) else this.state }}
    - name: waterpressure
      unique_id: water _pressure_id
      state: >-
        {{ d[3] if (d | count > 3 and d[0] | regex_match('^\\$\\w{2}VHW') ) else this.state }}

Robufz · July 3, 2023, 9:38pm

And now I put in a unit_of_measurement and it stops working. The following sensors are not reevaluated and are keeping their previous values.
Actually I think it has nothing to do with unit_of_measurement. It is probably some other state that is being disturbed by changing the code.

- trigger:
    - platform: state
      entity_id: sensor.serial_instruments
      variables:
        d: "{{ trigger.to_state.state.split(',') }}"
  sensor:
    - name: waterdepth
      unique_id: water_depth_id
      unit_of_measurement: "m"
      state: >-
        {{ d[3] if (d | count > 3 and d[0] | regex_match('^\\$\\w{2}DBT') ) else this.state }}
    - name: watertemperature
      unique_id: water_temperature_id
      state: >-
        {{ d[1] if (d | count > 1 and d[0] | regex_match('^\\$\\w{2}MTW') ) else this.state }}
    - name: waterpressure
      unique_id: water _pressure_id
      state: >-
        {{ d[3] if (d | count > 3 and d[0] | regex_match('^\\$\\w{2}VHW') ) else this.state }}

Robufz · July 3, 2023, 9:50pm

And now I take out the unit_of_measurement and the following 2 sensors are reporting again. Well watertemperature is quite stable but the pressure is changing.

- trigger:
    - platform: state
      entity_id: sensor.serial_instruments
      variables:
        d: "{{ trigger.to_state.state.split(',') }}"
  sensor:
    - name: waterdepth
      unique_id: water_depth_id
      state: >-
        {{ d[3] if (d | count > 3 and d[0] | regex_match('^\\$\\w{2}DBT') ) else this.state }}
    - name: watertemperature
      unique_id: water_temperature_id
      state: >-
        {{ d[1] if (d | count > 1 and d[0] | regex_match('^\\$\\w{2}MTW') ) else this.state }}
    - name: waterpressure
      unique_id: water _pressure_id
      state: >-
        {{ d[3] if (d | count > 3 and d[0] | regex_match('^\\$\\w{2}VHW') ) else this.state }}

Didgeridrew · July 3, 2023, 9:50pm

It might be the combination… If the state value is non-numeric (except ‘unavailable’/‘unknown’ IIRC) it will cause an error with a sensor with a unit_of_measurement.

Robufz · July 3, 2023, 9:52pm

So now I am at the same situation as 24hours ago.
Works without unit_of_measurement. But not with.
But then why does it just stop working the next day.
The serial data is more or less the same.
And waterdepth I have kept disconnected.

Robufz · July 3, 2023, 9:55pm

That is what I thought yesterday.
But what I cannot explain is why does 24 hours later change its behavior.
I even restarted the hardware, Rpi 4. But no difference.
I will put the null condition check again and see if that improves it.

Didgeridrew · July 3, 2023, 10:06pm

Script I used for testing

alias: zTest Repeat For testing
sequence:
  - repeat:
      for_each:
        - PRESS,,f,61,M,,F*3F
        - WEGT,11,1,F,5,4F
        - PRESS,,f,62,M,,F*3F
        - WEGT,12,1,F,5,4F
        - WEGT,13,1,F,5,4F
        - PRESS,,f,,M,,F*3F
        - WEGT,14,1,F,5,4F
        - PRESS,,f,63,M,,F*3F
        - WEGT,15,1,F,5,4F
        - PRESS,,f,64,M,,F*3F
        - WEGT,16,1,F,5,4F
        - WEGT,17,1,F,5,4F
        - WEGT,18,1,F,5,4F
        - PRESS,,f,65,M,,F*3F
        - WEGT,19,1,F,5,4F
        - PRESS,,f,66,M,,F*3F
        - PRESS,,f,67,M,,F*3F
        - WEGT,20,1,F,5,4F
        - WEGT,21,1,F,5,4F
        - PRESS,,f,68,M,,F*3F
        - WEGT,22,1,F,5,4F
        - PRESS,,f,69,M,,F*3F
        - WEGT,23,1,F,5,4F
        - WEGT,24,1,F,5,4F
        - PRESS,,f,,M,,F*3F
        - WEGT,25,1,F,5,4F
        - PRESS,,f,70,M,,F*3F
        - WEGT,26,1,F,5,4F
        - PRESS,,f,71,M,,F*3F
        - WEGT,27,1,F,5,4F
        - WEGT,28,1,F,5,4F
        - WEGT,29,1,F,5,4F
        - PRESS,,f,72,M,,F*3F
        - WEGT,30,1,F,5,4F
        - PRESS,,f,73,M,,F*3F
        - PRESS,,f,74,M,,F*3F
        - WEGT,31,1,F,5,4F
        - WEGT,32,1,F,5,4F
      sequence:
        - service: input_text.set_value
          data:
            value: "{{ repeat.item }}"
          target:
            entity_id:
              - input_text.ztest_text_test
        - delay: 30
mode: single

Is that a typo (I think the forum software will auto-corrects double commas)? Previously you posted it as follows: "PRESS,f,,M,F*3F”

Robufz · July 4, 2023, 8:21am

Yes must be some cut past error or typo I made. 65 is replaced with null.
PRESS,f,65,M,F3F
PRESS,f,M,F3F

The 3rd element on the 2nd line is null.
I added quotes just to show it is a string. The actual serial data does not include quotes.

Robufz · August 20, 2023, 6:11pm

I have figured out what the problem is.
If I define a unit_of_measurement: “m” then the template sensor MUST return a numerical value.
If it tries to return none or “” or ‘’ it will error out and not return anything.

So what can I set it If want to return a null value and I have a unit_of_measurement: “m” defined??
It works if I return a 0 but that is not correct in my case.

Didgeridrew · August 20, 2023, 9:31pm

The short answer is you can’t…

What is the purpose of returning a null value?

If the goal is to determine that the sensor isn’t updating, you can and should set up an availability. You will need to figure out how you want to differentiate between when the template needs to report null/unavailable and when it should use the previous state. You will likely need to define attributes or other standalone sensors.

Robufz · August 21, 2023, 11:37am

I figured it out! I used availability. And also got the timestamp working that we discussed a while back.
here is the code.

    - name: waterdepth
      unique_id: water_depth_id
      unit_of_measurement: "m"
      state: >-
        {% set new_data_condition = (d | count > 3 and d[0]|length >= 6 and d[0][0] == 'X' and d[0][3:6] == 'WBT') %}
        {{ d[3].split('*')[0] if new_data_condition else this.state }}
      availability: >-
        {% set new_data_condition = (d | count > 3 and d[0]|length >= 6 and d[0][0] == 'X' and d[0][3:6] == 'WBT') %}
        {% set time_difference_seconds = as_timestamp(now()) - as_timestamp(this.attributes.updated_timestamp | default(now())) %}
        {{ not (new_data_condition and d[3] == '') and (time_difference_seconds <= 60) }}
      attributes:
        updated_timestamp: >-
          {% set new_data_condition = (d | count > 3 and d[0]|length >= 6 and d[0][0] == 'X' and d[0][3:6] == 'WBT') %}
          {{ now() if new_data_condition else this.attributes.updated_timestamp | default(now()) }}

It would be nice not to have to repeat the new_data_condition logic 3 times. But I could not see how to set it to have scope for the whole sensor. Seems to only have scope per section.
Any ideas?
Since I now have 500 sensors I cant really declare a 500 different variables at the trigger level.

Didgeridrew · August 21, 2023, 5:26pm

There has been a Feature Request thread for variables in template sensors since before there were this or trigger variables, 2 variants of trigger variables, or custom template macros…

You may be able to reduce the size by using more specific triggers instead of relying on a single, general trigger that you then have to parse over and over.