Need help scraping temperature data from my Weishaupt heat pump (usr-cgi / xml source)

chrismaddahl · September 5, 2022, 5:07am

Hi all,

I am new to home assistant and in the process of my first setup.
I have the following problem:

My heatpump with a Dimplex Controller has the option to provide over usr-cgi / xml the data from all sensors.
I am interested in several temperature values.
For starters I tried to scrape only the outdoor temperature.

This adress:
http://192.168.0.27/usr-cgi/xml.cgi?A|1|10

provides the following output:

I am want the value under index 2

I goggled a lot and tried several configs
I also searched here in the community.

So far my config looks like this:

  - platform: scrape
    resource: http://192.168.0.27/usr-cgi/xml.cgi?A|1|10
    name: Temp Outdoor
    select: "/html/body/table/tbody/tr[10]/td[2]/text()"
    unit_of_measurement: °C
    value_template: '{{ value | replace (",", ".") | float }}'
    device_class: temperature
    state_class: measurement
    scan_interval: 60

Sensor is displayed in the dashboard but with value “unknown”

I think the problem is that the temperature value in the HTML source is not between tags but like free text:

So how can I scrape values like that?

tom_l · September 5, 2022, 5:32am

You should be able to use a restful sensor, but I need to put the XML through an XML to JSON conversion tool to work out the value template for you.

Can you post the XML from http://192.168.0.27/usr-cgi/xml.cgi?A|1|10 as formatted text please?

(generally posting images of text is a bad idea)

chrismaddahl · September 5, 2022, 5:40am

Oh, sorry for that.
Here is the text version.

<PCOWEB>
<PCO>
<ANALOG>
<VARIABLE>
<INDEX>1</INDEX>
<VALUE>13.5</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>2</INDEX>
<VALUE>21.0</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>3</INDEX>
<VALUE>-999.9</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>4</INDEX>
<VALUE>-8.7</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>5</INDEX>
<VALUE>22.3</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>6</INDEX>
<VALUE>22.5</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>7</INDEX>
<VALUE>22.0</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>8</INDEX>
<VALUE>43.2</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>9</INDEX>
<VALUE>-999.9</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>10</INDEX>
<VALUE>-999.9</VALUE>
</VARIABLE>
</ANALOG>
</PCO>
</PCOWEB>

tom_l · September 5, 2022, 5:42am

Unfortunately you have stripped off all the indentation. It is important to include this.

chrismaddahl · September 5, 2022, 5:53am

is this better?
I saved it as .XML, opened it with notepad++ and copy/pasted it here as preformatted text

<PCOWEB>
<PCO>
<ANALOG>
<VARIABLE>
<INDEX>1</INDEX>
	<VALUE>13.4</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>2</INDEX>
	<VALUE>21.0</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>3</INDEX>
	<VALUE>-999.9</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>4</INDEX>
	<VALUE>-8.7</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>5</INDEX>
	<VALUE>22.3</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>6</INDEX>
	<VALUE>22.5</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>7</INDEX>
	<VALUE>22.0</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>8</INDEX>
	<VALUE>43.2</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>9</INDEX>
	<VALUE>-999.9</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>10</INDEX>
	<VALUE>-999.9</VALUE>
</VARIABLE>
</ANALOG>
</PCO>
</PCOWEB>

tom_l · September 5, 2022, 5:55am

Nope. It still does not look like this:

chrismaddahl · September 5, 2022, 5:56am

But I can’t copy the indentations, it only selects the text:

tom_l · September 5, 2022, 5:59am

Is there an option to save it as “raw”.

chrismaddahl · September 5, 2022, 6:06am

I can only save as XML in chrome or edge, than i get what was posted before.

if i use View Source there is even less indentations:

Or i could manually add the identations in notepad++

chrismaddahl · September 5, 2022, 6:44am

I tried your suggestion with XML to JSON myself and got the following results:

I found the correct JSONpath:

this is my new config:

  - platform: rest
    json_attributes_path: "$.PCOWEB.PCO.ANALOG.VARIABLE[:1].VALUE"
    json_attributes:
      - VALUE
    resource: http://192.168.0.27/usr-cgi/xml.cgi?A|1|10
    value_template: '{{ value_json.VALUE | replace (".", ",") }}'
    name: "Temp Outdoor X"
    method: GET
    unit_of_measurement: °C
    device_class: temperature
    state_class: measurement
    scan_interval: 60

senor gets recognized, but it shows value 0°C

if I add “| float” to the value_template, the sensor disappears from dashboard without error message

Any idea why?

tom_l · September 5, 2022, 6:50am

Share the text you used. There are a number of things wrong with what you have done.

chrismaddahl · September 5, 2022, 7:08am

Here is the JSON text:

{
  "PCOWEB": {
    "PCO": {
      "ANALOG": {
        "VARIABLE": [
          {
            "INDEX": "1",
            "VALUE": "13.5"
          },
          {
            "INDEX": "2",
            "VALUE": "21.0"
          },
          {
            "INDEX": "3",
            "VALUE": "-999.9"
          },
          {
            "INDEX": "4",
            "VALUE": "-8.7"
          },
          {
            "INDEX": "5",
            "VALUE": "22.3"
          },
          {
            "INDEX": "6",
            "VALUE": "22.5"
          },
          {
            "INDEX": "7",
            "VALUE": "22.0"
          },
          {
            "INDEX": "8",
            "VALUE": "43.2"
          },
          {
            "INDEX": "9",
            "VALUE": "-999.9"
          },
          {
            "INDEX": "10",
            "VALUE": "-999.9"
          }
        ]
      }
    }
  }
}

tom_l · September 5, 2022, 7:18am

Sigh. No. The XML.

chrismaddahl · September 5, 2022, 7:22am

It is the same as before:

<PCOWEB>
<PCO>
<ANALOG>
<VARIABLE>
<INDEX>1</INDEX>
	<VALUE>13.4</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>2</INDEX>
	<VALUE>21.0</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>3</INDEX>
	<VALUE>-999.9</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>4</INDEX>
	<VALUE>-8.7</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>5</INDEX>
	<VALUE>22.3</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>6</INDEX>
	<VALUE>22.5</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>7</INDEX>
	<VALUE>22.0</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>8</INDEX>
	<VALUE>43.2</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>9</INDEX>
	<VALUE>-999.9</VALUE>
</VARIABLE>
<VARIABLE>
<INDEX>10</INDEX>
	<VALUE>-999.9</VALUE>
</VARIABLE>
</ANALOG>
</PCO>
</PCOWEB>

tom_l · September 5, 2022, 8:11am

Put this in your configuration.yaml file and then run a config check, restart if it passes.

rest:
  - resource: http://192.168.0.27/usr-cgi/xml.cgi?A|1|10
    scan_interval: 60
    sensor:
      - name: "Index 2"
        json_attributes_path: "$.response.system"
        value_template: "{{ value_json['PCO']['ANALOG'][1]['VALUE'] }}"

chrismaddahl · September 5, 2022, 8:26am

it didn’t show any errors after config check, but it shows only value unknown after restart:

what is the meaning of json_attributes_path: “$.response.system” ?

and could it be a problem that the value in JSON is also in double quotes?

tom_l · September 5, 2022, 8:41am

Probably because of the indentation issues with your XML.

Copy and paste mistake. It shouldn’t stop your sensor form working, though you should use:

rest:
  - resource: http://192.168.0.27/usr-cgi/xml.cgi?A|1|10
    scan_interval: 60
    sensor:
      - name: "Index 2"
        value_template: "{{ value_json['PCO']['ANALOG'][1]['VALUE'] }}"

No, single line templates must be enclosed in quotes.

chrismaddahl · September 5, 2022, 10:20am

Why is indentation so important, I thought whitespace is not relevant for parsing XML?

chrismaddahl:

and could it be a problem that the value in JSON is also in double quotes?

No, single line templates must be enclosed in quotes.

No, I meant this double quotes:

Could is be a problem, because the REST is returning the value “13.5” and not 13.5?
And because of that it is only displaying 0°C or unknown?

tom_l · September 5, 2022, 10:29am

No that is not the issue.

chrismaddahl · September 5, 2022, 11:13am

Ok, than maybe a different approach, there is a second option to extract the data.

I can create my own HTML file on the heat pump and specify which values to output.

This is the HTML file on server:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<!--tagparser="/pcotagfilt"-->
<head>
 <meta content="text/html; charset=ISO-8859-15" http-equiv="content-type"> 
 <meta http-equiv="refresh" content="15" >
</head>
<body>
<h1><%var(0,2,1)%></h1>
</body>
</html>

This is the HTML file when I open it with browser and view source:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<!--tagparser="/pcotagfilt"-->
<head>
 <meta content="text/html; charset=ISO-8859-15" http-equiv="content-type"> 
 <meta http-equiv="refresh" content="15" >
</head>
<body>
<h1>23.7</h1>
</body>
</html>

This is my config:

  - platform: scrape
    resource: http://192.168.0.27/temp2.html
    name: "Scrape Test"
    select: "body > h1"
    unit_of_measurement: °C
    value_template: '{{ value | replace (".", ",") | float }}'
    device_class: temperature
    state_class: measurement
    scan_interval: 60

Now I get this result:

What am I doing wrong?