Simple xml REST api as sensor data does not work

[Probing]

hi,

I have a rest api that gives an xml like this:

<?xml version="1.0"?>
<blubb>
    <foo/>
    <bar/>
    <zone1>16</zone1>
    <zone2>1</zone2>
    <zone3>3</zone3>
</blubb>

I now want to import that data (16, 1 and 3) into Home Assistant, but my attempts failed since hours of trying.

my configuration.yaml looks like this now:

# Loads default set of integrations. Do not remove.
default_config:

# Text to speech
tts:
  - platform: google_translate

automation: !include automations.yaml
script: !include scripts.yaml
scene: !include scenes.yaml

# Sensors of Domain1
rest:
  - scan_interval: 30
    resource: https://www.domain1.com/api.php
    verify_ssl: false
    sensor:
      - name: "Name of Sensor"
        value_template: "{{ value_json.blubb.zone3 }}"
        unit_of_measurement: "Amount"
        state_class: "measurement"
        unique_id: "domain1.apiofthissensor.sensorname"

What am I doing wrong?

[tom_l]

Assuming all the data is less than 255 characters, start by using just:

value_template: "{{ value }}"

What is the state of that sensor?

[Troon]

@Probing

gives an xml like this

Can we see the actual XML? When people make up replacement data, we often end up chasing a problem in the source data we can’t see. For example, if any of the keys in the real data starts with a digit, then you need bracket notation:

{{ value_json['blubb']['3rd_zone'] }}

and if any of the values are non-numeric, you cannot specify unit_of_measurement or state_class.

@Probing

my attempts failed

How? Do you see the sensor but it is unknown? Do you not see the sensor at all (and if so, have you restarted HA to load up the rest integration?)?

Are there any errors in your logs?

[tom_l]

Also some XML → json puts an @ in front of the key, but I can’t remember the rule. Which was Why I wanted to see the converted data with {{value}}.

[Probing]

Thank you all for taking your time!

@Troon

How? Do you see the sensor but it is unknown? Do you not see the sensor at all (and if so, have you restarted HA to load up the rest integration?)?

Are there any errors in your logs?

I did not get that there are additional logs hidden in the settings. I thought the “logbook” (translated) in the main menu would be all the logs I get.
Now the real logs are something I can work with. Thank you for hinting me to that.

So with the original yaml of the original post i get:

Template variable error: 'value_json' is undefined when rendering '{{ value_json.blubb.zone3 }}' 

Seems to me that the file can not be parsed to json. I would also be fine with XQuery/XPath that is usually used for xml data, but this seems to be not available in homeassistant. At least I could not find anything about that in the docs.

@tom_l

Assuming all the data is less than 255 characters, start by using just:

value_template: "{{ value }}"

When i use value_template: "{{ value }}" i get this error:

ValueError: Sensor sensor.nameOfSensor has device class 'None', state class 'measurement' unit 'Amount' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: '<?xml version="1.0"?>
<blubb>
    <foo/>
    <bar/>
    <zone1>496</zone1>
    <zone2>231</zone2>
    <zone3>528</zone3>
</blubb>' (<class 'str'>)
2024-02-06 14:20:39.682 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up rest platform for sensor
Traceback (most recent call last):
  File "/lsiopy/lib/python3.11/site-packages/homeassistant/components/sensor/__init__.py", line 644, in state
    numerical_value = float(value)  # type:ignore[arg-type]
                      ^^^^^^^^^^^^
ValueError: could not convert string to float: '<?xml version="1.0"?>\n<blubb>\n    <foo/>\n    <bar/>\n    <zone1>496</zone1>\n    <zone2>231</zone2>\n    <zone3>528</zone3>\n</blubb>'

So at least the data is there.

@Troon

Can we see the actual XML? When people make up replacement data, we often end up chasing a problem in the source data we can’t see. For example, if any of the keys in the real data starts with a digit, then you need bracket notation:

{{ value_json['blubb']['3rd_zone'] }}

and if any of the values are non-numeric, you cannot specify unit_of_measurement or state_class.

I would rather not to be honest. But im promise that the data looks exactly like that. No digits in the xml elements. No special characters either. Data is always numerical.
I also tried some variants of the JSONPath that would work in an online evaluator, but without any luck.

@tom_l

Also some XML → json puts an @ in front of the key, but I can’t remember the rule. Which was Why I wanted to see the converted data with {{value}}.

Seems not to be the case here. There goes something wrong in the conversion from XML → JSON.

@tom_l

What is the state of that sensor?

What does this mean? The “sensor” in this case is an online API.

[tom_l]

@Probing

When i use value_template: "{{ value }}" i get this error:

Remove the state_class option for this test. And unit too.

rest:
  - scan_interval: 30
    resource: https://www.domain1.com/api.php
    verify_ssl: false
    sensor:
      - name: "Name of Sensor"
        value_template: "{{ value }}"
        unique_id: "domain1.apiofthissensor.sensorname"

[Troon]

I’ve just published this on my website:

test.xml:

<blubb>
    <foo/>
    <bar/>
    <zone1>496</zone1>
    <zone2>231</zone2>
    <zone3>528</zone3>
</blubb>

and this sensor:

- scan_interval: 30
  resource: https://MY_DOMAIN/public/test.xml
  verify_ssl: false
  sensor:
      - name: "Name of Sensor"
        value_template: "{{ value_json.blubb.zone3 }}"
        unit_of_measurement: "Amount"
        state_class: "measurement"
        unique_id: "domain1.apiofthissensor.sensorname"

gives this:

So there’s nothing wrong with the configuration as you have presented it. My guess is that it’s something in your headers, or an oddity like an invisible Byte Order Mark, or (despite your protestations to the contrary) something in the real data that is throwing it. Try tom_l’s suggestion — if the XML you’re reading is over 255 characters, use {{ value[:250] }} as the template.

[Probing]

@tom_l

@Probing

When i use value_template: "{{ value }}" i get this error:

Remove the state_class option for this test. And unit too.

rest:
  - scan_interval: 30
    resource: https://www.domain1.com/api.php
    verify_ssl: false
    sensor:
      - name: "Name of Sensor"
        value_template: "{{ value }}"
        unique_id: "domain1.apiofthissensor.sensorname"

Then i get the whole xml as data in the sensor.

@Troon

I’ve just published this on my website:

test.xml:

<blubb>
    <foo/>
    <bar/>
    <zone1>496</zone1>
    <zone2>231</zone2>
    <zone3>528</zone3>
</blubb>

and this sensor:

- scan_interval: 30
  resource: https://MY_DOMAIN/public/test.xml
  verify_ssl: false
  sensor:
      - name: "Name of Sensor"
        value_template: "{{ value_json.blubb.zone3 }}"
        unit_of_measurement: "Amount"
        state_class: "measurement"
        unique_id: "domain1.apiofthissensor.sensorname"

gives this:

So there’s nothing wrong with the configuration as you have presented it. My guess is that it’s something in your headers, or an oddity like an invisible Byte Order Mark, or (despite your protestations to the contrary) something in the real data that is throwing it. Try tom_l’s suggestion — if the XML you’re reading is over 255 characters, use {{ value[:250] }} as the template.

Thank you for going far beond what can be expected to test this.
Could you also add the xml header and try again? Maybe this is what makes the jjson parser to hickup?

[tom_l]

@Probing

Then i get the whole xml as data in the sensor.

Hmmm. What about if you use:

value_template: “{{ value_json }}”

And actually post the results.

[Troon]

My test above fails if I rename the file to end in .php, as my web server sends it with different headers.

value in that situation is the full XML but value_json is undefined, just as you’re seeing: because of the headers, HA doesn’t run the conversion.

You will either need to get your PHP script / server to send as text/xml if you want to use HA’s XML-to-JSON conversion (the correct solution); or you will have to botch it by manually scraping the XML, something like this:

- scan_interval: 30
  resource: https://www.domain1.com/api.php
  verify_ssl: false
  sensor:
      - name: "Name of Sensor"
        value_template: >
          {{ value|regex_findall("zone3>(\d*)<")|first }}

Following a DM with the OP, it turns out that the real server in question will respond with application/xml if the request ends in .xml rather than .php. So this:

rest:
  - scan_interval: 30
    resource: https://www.domain1.com/api.xml
note change ------------------------------^^^
    verify_ssl: false
    sensor:
      - name: "Name of Sensor"
        value_template: "{{ value_json.blubb.zone3 }}"
        unit_of_measurement: "Amount"
        state_class: "measurement"
        unique_id: "domain1.apiofthissensor.sensorname"

should work. UPDATE: I’ve just tried it myself and it does work.

[Probing]

@tom_l

@Probing

Then i get the whole xml as data in the sensor.

Hmmm. What about if you use:

value_template: “{{ value_json }}”

And actually post the results.

Then i get this error in the logs:

[homeassistant.helpers.template] Template variable warning: 'value_json' is undefined when rendering '{{ value_json }}'

What do you mean with “results”. I then cann search the sensor on the “overview” tab of the webinterface. And there i can see its value. Is there a better way doing this?

@Troon

My test above fails if I rename the file to end in .php, as my web server sends it with different headers.

value in that situation is the full XML but value_json is undefined, just as you’re seeing: because of the headers, HA doesn’t run the conversion.

You will either need to get your PHP script / server to send as text/xml if you want to use HA’s XML-to-JSON conversion (the correct solution); or you will have to botch it by manually scraping the XML, something like this:

- scan_interval: 30
  resource: https://www.domain1.com/api.php
  verify_ssl: false
  sensor:
      - name: "Name of Sensor"
        value_template: >
          {{ value|regex_findall("zone3>(\d*)<")|first }}

Thank you for testing even further and finding out what the problem was.
Could this be considered a bug, or is this behavour expected?

Since I am not in control over the API I am not able to change the header unfortunately. But i am happy with some regex magic.
It is now workign as expected. Big thank you to both of you for your time!

One last slightly off topic question:
I have set this unique id in the sensor …

unique_id: "this.isaverylong.unique.name"

… but in the webinterface the entitiy-id shows up as sensor.name. Am I doing something wrong here, or do i get the concept wrong?

[Troon]

@Probing

Could this be considered a bug, or is this behavour expected?

It could be considered a bug in the source server that it sends out XML with the wrong headers. HA behaviour is expected.

@Probing

Am I doing something wrong here, or do i get the concept wrong?

The latter. The sensor entity ID comes from the name by default, so:

  sensor:
      - name: "Blubb Zone 3"

would be sensor.blubb_zone_3. The unique_id is an internal thing, needed for allowing customisation and assigning to areas. You can rename the entity ID in the UI later if you want, but best not to.

I use this tool for generating unique IDs.

[Probing]

@Troon

Following a DM with the OP, it turns out that the real server in question will respond with application/xml if the request ends in .xml rather than .php. So this:

rest:
  - scan_interval: 30
    resource: https://www.domain1.com/api.xml
note change ------------------------------^^^
    verify_ssl: false
    sensor:
      - name: "Name of Sensor"
        value_template: "{{ value_json.blubb.zone3 }}"
        unit_of_measurement: "Amount"
        state_class: "measurement"
        unique_id: "domain1.apiofthissensor.sensorname"

should work. UPDATE: I’ve just tried it myself and it does work.

Oh wow, the API also does respond on an “.xml” ending? I did not think that this is doable, so I did not try.

Thanks that makes it even easier! You’re a hero.

[Troon]

@Probing

Oh wow, the API also does respond on an “.xml” ending? I did not think that this is doable, so I did not try.

That is not a general solution that will work in all cases. I tried it on the off-chance, because I’ve been doing this for a long time.

troon@laptop:~$ curl -v https://[...].php
* processing: [...]
< content-type: text/html; charset=UTF-8
<?xml version="1.0"?>
...
troon@laptop:~$ curl -v https://[...].xml
* processing: [...]
< content-type: application/xml; charset=utf-8
<?xml version="1.0"?>