Building a sensor from an API that returns a '-' (hyphen) more often than a number

Bhrudwo · March 14, 2024, 7:11am

I am trying to build a sensor that is sourced from my local city council’s public API that reports on the local creek/river height, as it can rise quite quickly and I want to build a sensor to alarm off of.
APIs are extremely new to me, and only been tinkering with HA for about 6 months.

I have built the sensor so far as

sensor:
- platform: rest
resource: https://www.data.brisbane.qld.gov.au/data/api/3/action/datastore_search?resource_id=78c37b45-ecb5-4a99-86b2-f7a514f0f447&fields=E1702,_id&limit=1&sort=_id%20desc
method: GET
name: riverheight_pineapple_st
value_template: "{{ value_json.result.records[0].E1702 | float(0.0) }}"
scan_interval: 300

The data it returns is either

a hyphen ‘-’, or
a number

The float bit, I think, is working. When I look at the sensor in the Entities page, I see this, and can see a reading of 9.18m (the purple sections are showing numbers).

But I was the sensor to report a number all the time.
How do I handle the fact that more often than not the value/state will be ‘-’ and not a number? but I want it to basically read as a number all the time? Like: IF value = ‘-’ THEN use last good value, ELSE value = current read value.

Thanks in advance.

tom_l · March 14, 2024, 7:47am

This will keep the last value if a hyphen is received:

value_template: >
  {{ this.state if value_json.result.records[0].E1702 == '-' else value_json.result.records[0].E1702 }}

There’s no point using the float filter. States are strings.

Bhrudwo · March 14, 2024, 9:22am

Thanks, I will give that a try. Will have to wait for the data to not be a ‘-‘ to check that it is reading a number and keeps it.
I was thinking I had to push the number in as a float, as it appears to be a string from the json, even when it is a number

tom_l · March 14, 2024, 9:35am

All you have to do to have it be treated as a number is give it a unit_of_measurement.

Bhrudwo · March 14, 2024, 9:42pm

Thanks, I added the unit_of_measurement as “m”

  - platform: rest
    resource: https://www.data.brisbane.qld.gov.au/data/api/3/action/datastore_search?resource_id=78c37b45-ecb5-4a99-86b2-f7a514f0f447&fields=E1702,_id&limit=1&sort=_id%20desc
    method: GET
    name: riverheight_pineapple_st_2
    #value_template: "{{ value_json.result.records[0].E1702 | float(0.0) }}"
    value_template: >
      {{ this.state if value_json.result.records[0].E1702 == '-' else value_json.result.records[0].E1702 }}
    scan_interval: 300
    unit_of_measurement: "m"

And finally got a new reading other than “-” overnight, and this is what I now see, which is a huge step forward, I have an actual value

Is there any way to get the numerical reading to continue to be the reading until a new value comes from the API?
Kind of like, if reading = “-” then use previous reading, else reading?
I think this is what you had suggested in the value_template, but it doesn’t appear to be doing that.
Cheers

tom_l · March 14, 2024, 10:07pm

It should be doing that. this.state is the previous value:

Are there other non-numeric messages than “-”?

Also check Settings → System → Logs for errors about this sensor.

Bhrudwo · March 14, 2024, 11:17pm

No, it is either a number of a ‘-’; but when ‘-’ and check the State of the sensor, it says unknown

The sensor got a second reading (number) this morning, so the first was 9.18m and the second is 9.16m

tom_l · March 14, 2024, 11:19pm

Bhrudwo · March 14, 2024, 11:22pm

Sorry, forgot to do that.
I am seeing this in the logs

2024-03-15 09:13:42.408 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: ‘this’ is undefined when rendering ‘{{ this.state if value_json.result.records[0].E1702 == ‘-’ else value_json.result.records[0].E1702 }}’

2024-03-15 09:18:42.441 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: ‘this’ is undefined when rendering ‘{{ this.state if value_json.result.records[0].E1702 == ‘-’ else value_json.result.records[0].E1702 }}’

tom_l · March 14, 2024, 11:46pm

Ok try this:

value_template: >
      {{ this.state|default(0) if value_json.result.records[0].E1702 == '-' else value_json.result.records[0].E1702 }}
    scan_interval: 300

Then your sensor should be 0 until it updates to a number the first time. Then it will always keep the last state or be the new number.

Bhrudwo · March 15, 2024, 12:14am

So I updated it and now the log is showing:
Logger: homeassistant.helpers.template
Source: helpers/template.py:2345
First occurred: 10:12:30 AM (1 occurrences)
Last logged: 10:12:30 AM

Template variable error: ‘this’ is undefined when rendering ‘{{ this.state|default(0) if value_json.result.records[0].E1702 == ‘-’ else value_json.result.records[0].E1702 }} #{{ this.state if value_json.result.records[0].E1702 == ‘-’ else value_json.result.records[0].E1702 }}’

I see that it is including the previous commented out line (I commented it out rather than deleting at this stage.

Troon · March 15, 2024, 7:24am

Then use:

{{ (this|default({'state':0})).state if value_json.result.records[0].E1702 == '-' else value_json.result.records[0].E1702 }}

Feels like a bit of a bodge but it works

Bhrudwo · March 16, 2024, 12:53am

Working, as in I am getting numbers now - 0 or a reading, so a huge step forward.

Bhrudwo · March 16, 2024, 3:39am

Any thoughts on why the if statement is not feeding back the previous value?

Bhrudwo · March 27, 2024, 1:00am

So I have this kind of working, although I am getting errors in HA Logs.

I am playing with two ways of importing it

  - platform: rest
    resource: https://www.data.brisbane.qld.gov.au/data/api/3/action/datastore_search?resource_id=78c37b45-ecb5-4a99-86b2-f7a514f0f447&fields=E1702,_id&limit=1&sort=_id%20desc
    method: GET
    name: riverheight_pineapple_st
    value_template: "{{ value_json.result.records[0].E1702 }}"
    scan_interval: 300
    unit_of_measurement: "m"

  - platform: rest
    resource: https://www.data.brisbane.qld.gov.au/data/api/3/action/datastore_search?resource_id=78c37b45-ecb5-4a99-86b2-f7a514f0f447&fields=E1702,_id&limit=1&sort=_id%20desc
    method: GET
    name: riverheight_pineapple_st_2
    value_template: >
      {{ (this|default({'state':0})).state if value_json.result.records[0].E1702 == '-' else value_json.result.records[0].E1702 }}
    scan_interval: 300
    unit_of_measurement: "m"

and then using a template to try and prevent the value updating on a bad read.

template:
  - trigger:
      - platform: state
        entity_id: sensor.riverheight_pineapple_st
        not_to:
          - unknown
          - unavailable
  - trigger:
      - platform: state
        entity_id: sensor.riverheight_pineapple_st_2
        not_to:
          - unknown
          - unavailable
          - "0"

Logs when the value is “-” for the first sensor.

Logger: homeassistant.helpers.entity
Source: helpers/entity.py:945
First occurred: March 26, 2024 at 8:21:49 AM (612 occurrences)
Last logged: 7:06:49 AM

Update for sensor.riverheight_pineapple_st fails
Update for sensor.riverheight_pineapple_st_exp2 fails
Update for sensor.riverheight_pineapple_st_exp fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 663, in state
    numerical_value = int(value)
                      ^^^^^^^^^^
ValueError: invalid literal for int() with base 10: '-'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 666, in state
    numerical_value = float(value)
                      ^^^^^^^^^^^^
ValueError: could not convert string to float: '-'

The above exception was the direct cause of the following exception:

But for the second sensor, it still drops to zero when the reading is “-”, even with the template.
So even though the first sensor (and others I am experimenting with) are creating errors in the log files, they are at least preventing a new/false reading, and keeps the old good value.

It is raining quiet heavily at the moment, for the last few days, so I actually have some data…

Anyone have thoughts on why my template with the “0” in the not_to doesn’t work?

Troon · March 27, 2024, 7:39am

This works for me — read in the raw value with no unit of measurement in the rest sensor, sort out what to do in the template sensor:

sensor:
  - platform: rest
    resource: https://www.data.brisbane.qld.gov.au/data/api/3/action/datastore_search?resource_id=78c37b45-ecb5-4a99-86b2-f7a514f0f447&fields=E1702,_id&limit=1&sort=_id%20desc
    name: riverheight_pineapple_raw
    value_template: "{{ value_json['result']['records'][0]['E1702'] }}"
    scan_interval: 300

template:
  - sensor:
      - name: Pineapple river height
        state: >
          {% if states('sensor.riverheight_pineapple_raw')|float('x') is number %}
            {{ states('sensor.riverheight_pineapple_raw') }}
          {% else %}
            {{ this.state }}
          {% endif %}
        unit_of_measurement: "m"

Has no value until the first numeric measurement comes in. Then I mess about with the raw value using Set State, and it ignores non-numeric and holds numeric value as they come in:

Bhrudwo · April 3, 2024, 7:44am

Thanks heaps for that, it appears to work without the errors my method was giving me.