Another XML Scrape help question

Roughlee · July 17, 2022, 12:46pm

I want to get the temp and a few others on to a dashboard.
Help please.

This is the xml page

<?xml version="1.0" encoding="windows-1250" ?>
<?xml-stylesheet type="text/xsl" href="HOME.XSL?U=1588753443" version="1.0" ?>
<PAGE TITLE="                                  ">
<ACCESS PAGE_LEVEL="0" USER_LEVEL="0" UCID="1588753443" />
<INPUT NAME="__R3189_TIME_Thh:mm:ss" VALUE="02:00:00" />
<INPUT NAME="__R37954_TIME_Thh:mm:ss" VALUE="00:10:00" />
<INPUT NAME="__R4008.0_BOOL_i" VALUE="1" />
<INPUT NAME="__R79565.0_BOOL_i" VALUE="0" />
<INPUT NAME="__R79565.1_BOOL_i" VALUE="0" />
<INPUT NAME="__R3159.0_BOOL_i" VALUE="0" />
<INPUT NAME="__R79564.6_BOOL_i" VALUE="0" />
<INPUT NAME="__R79564.7_BOOL_i" VALUE="0" />
<INPUT NAME="__R37958.1_BOOL_i" VALUE="1" />
<INPUT NAME="__R24863.0_BOOL_i" VALUE="0" />
<INPUT NAME="__R5439.1_BOOL_i" VALUE="1" />
<INPUT NAME="__R2357.1_BOOL_i" VALUE="1" />
<INPUT NAME="__R22657_REAL_.1f" VALUE="27.2" />
<INPUT NAME="__R22661_REAL_.1f" VALUE="0.0" />
<INPUT NAME="__R2357.0_BOOL_i" VALUE="1" />
<INPUT NAME="__R22655.1_BOOL_i" VALUE="0" />
<INPUT NAME="__R40551_REAL_.1f" VALUE="22.0" />
<INPUT NAME="__R79561.0_BOOL_i" VALUE="0" />
<INPUT NAME="__R79561.1_BOOL_i" VALUE="0" />
<INPUT NAME="__R2398_STRING[15]_s" VALUE="UFH Down" />
</PAGE>

bouwew · July 17, 2022, 5:44pm

Have you Googled “home assistant scrape XML”? This give you various sources of topics on this community on the same subject.

Roughlee · July 17, 2022, 8:39pm

I have and none of them work.

bouwew · July 18, 2022, 7:37am

I can imagine, the xml-data you show is not real xml.
I don’t know how the Scrape integration works but I think you need to first look for the complete content of 1 line: INPUT NAME="__R22657_REAL_.1f" VALUE="27.2" and then via a second step (value_template?) obtain the number 27.2.

Roughlee · July 18, 2022, 10:30am

Thank you but, not sure how to do that.

bouwew · July 18, 2022, 1:18pm

Please look at this topic for inspiration: RESTful and XML Integration

The xml-data in that topic looks like your xml-data.

Roughlee · July 18, 2022, 5:24pm

I have seen this page and I did try some of it, there is allot. I will try again and reply. Thank you

Roughlee · July 21, 2022, 8:30am

Not having much luck. I have this code but every time I make a change it kills HomeAssitant and I have to restore from a snapshot.
sensor:

platform: rest
resource: http://192.168.14.232/login.xml
username: user
password: Password
name: _R22657_REAL.1f

Any ideas?

bouwew · July 21, 2022, 11:15am

Please look it this post: RESTful and XML Integration - #22 by Reijer and the post above. This looks a bit like what you have/want.

I think you need not to write name: but value_template: '{{ value_json[...][...] }}' like in the example I link above.

And the trick is to find the correct strings to put between the [ ]'s.

Roughlee · July 22, 2022, 6:57am

OK i have tried this
sensor:

platform: rest
name: Steam System Data
resource: http://192.168.14.232/login.xml
username: user
password: 1Desmonddo
authentication: basic
value_json[“INPUT”]["-NAME"]["-NAME"][?(@["-VALUE"]=="_R22657_REAL.1f")]["@Value"]

But I keep getting this

 can not read an implicit mapping pair; a colon is missed at line 21, column 89:
     ... ="__R22657_REAL_.1f")]["@Value"]
                                         ^

Arrow inditcating the underscore after “REAL”
any ideas?

Daniel76 · July 22, 2022, 7:05am

I think it should look like this:
`

value_template: '{{ value_json [“INPUT”]["-NAME"]["-NAME"][?(@["-VALUE"]=="_*R22657_REAL* .1f")]["@Value"] }}'

bouwew · July 22, 2022, 12:37pm

I’ve used the XML to JSON converter that is in the first post of the example thread, after some rearranging, the result is this:

{"PAGE":
	{
		"@TITLE":"                                  ",
		"ACCESS": 
			{
				"@PAGE_LEVEL":"0",
                "@UCID":"1588753443",
				"@USER_LEVEL":"0"
			},
		"INPUT":
			[
				{"@NAME":"__R3189_TIME_Thh:mm:ss","@VALUE":"02:00:00"},
				{"@NAME":"__R37954_TIME_Thh:mm:ss","@VALUE":"00:10:00"},
				{"@NAME":"__R4008.0_BOOL_i","@VALUE":"1"},
				{"@NAME":"__R79565.0_BOOL_i","@VALUE":"0"},
				{"@NAME":"__R79565.1_BOOL_i","@VALUE":"0"},
				{"@NAME":"__R3159.0_BOOL_i","@VALUE":"0"},
				{"@NAME":"__R79564.6_BOOL_i","@VALUE":"0"},
				{"@NAME":"__R79564.7_BOOL_i","@VALUE":"0"},
				{"@NAME":"__R37958.1_BOOL_i","@VALUE":"1"},
				{"@NAME":"__R24863.0_BOOL_i","@VALUE":"0"},
				{"@NAME":"__R5439.1_BOOL_i","@VALUE":"1"},
				{"@NAME":"__R2357.1_BOOL_i","@VALUE":"1"},
				{"@NAME":"__R22657_REAL_.1f","@VALUE":"27.2"},
				{"@NAME":"__R22661_REAL_.1f","@VALUE":"0.0"},
				{"@NAME":"__R2357.0_BOOL_i","@VALUE":"1"},
				{"@NAME":"__R22655.1_BOOL_i","@VALUE":"0"},
				{"@NAME":"__R40551_REAL_.1f","@VALUE":"22.0"},
				{"@NAME":"__R79561.0_BOOL_i","@VALUE":"0"},
				{"@NAME":"__R79561.1_BOOL_i","@VALUE":"0"},
				{"@NAME":"__R2398_STRING[15]_s","@VALUE":"UFH Down"}
			]
	}
}

bouwew · July 22, 2022, 12:48pm

Based on this I think it should be something like: {{ value_json ["PAGE"]["INPUT"][?(@["NAME"]=="_*R22657_REAL* .1f")]["@Value"] }}

martinmariknet · December 22, 2024, 10:19pm

Hello, may I ask for help? I have very similar input data:

<?xml version="1.0" encoding="windows-1250" ?>
<?xml-stylesheet type="text/xsl" href="TOPENI.XSL?U=915" version="1.0" ?>
<PAGE TITLE="                                  ">
<ACCESS PAGE_LEVEL="0" USER_LEVEL="0" UCID="915" />
<INPUT NAME="__R11078_REAL_.1f" VALUE="20.6" />
<INPUT NAME="__R11086_REAL_.1f" VALUE="14.6" />
<INPUT NAME="__R11094_REAL_.1f" VALUE="20.5" />
<INPUT NAME="__R11074_REAL_.1f" VALUE="19.2" />
<INPUT NAME="__R11098_REAL_.1f" VALUE="18.1" />
<INPUT NAME="__R11090_REAL_.1f" VALUE="20.7" />
<INPUT NAME="__R11082_REAL_.1f" VALUE="17.7" />
<INPUT NAME="__R11102_REAL_.1f" VALUE="22.7" />
<INPUT NAME="__R11106_REAL_.1f" VALUE="19.5" />
<INPUT NAME="__R11122_REAL_.1f" VALUE="18.0" />
<INPUT NAME="__R11130_REAL_.1f" VALUE="10.0" />
<INPUT NAME="__R11138_REAL_.1f" VALUE="20.0" />
<INPUT NAME="__R11118_REAL_.1f" VALUE="18.0" />
<INPUT NAME="__R11142_REAL_.1f" VALUE="17.0" />
<INPUT NAME="__R11134_REAL_.1f" VALUE="20.0" />
<INPUT NAME="__R11126_REAL_.1f" VALUE="18.0" />
<INPUT NAME="__R11146_REAL_.1f" VALUE="23.0" />
<INPUT NAME="__R11150_REAL_.1f" VALUE="20.0" />
<INPUT NAME="__R11226.0_BOOL_i" VALUE="0" />
<INPUT NAME="__R11285.0_BOOL_i" VALUE="0" />
<INPUT NAME="__R11344.0_BOOL_i" VALUE="1" />
<INPUT NAME="__R11403.0_BOOL_i" VALUE="0" />
<INPUT NAME="__R11462.0_BOOL_i" VALUE="0" />
<INPUT NAME="__R11521.0_BOOL_i" VALUE="0" />
<INPUT NAME="__R11580.0_BOOL_i" VALUE="0" />
<INPUT NAME="__R11639.0_BOOL_i" VALUE="1" />
<INPUT NAME="__R11698.0_BOOL_i" VALUE="1" />
</PAGE>

And I’m trying to get multiple values for different variables e.g. _R11078_REAL.1f, _R11086_REAL.1f, …

I’m using following statement:

rest:
  - authentication: basic
    username: "xxx"
    password: "xxx"
    resource: http://192.168.1.12/TOPENI.XML
    scan_interval: 120
    sensor:
     - name: "Temp-living"
       value_template: '{{ value_json ["PAGE"]["INPUT"][(["NAME"]=="__R11082_REAL_.1f")]["@VALUE"] }}'

but as a result I always get first value - 20.6. Do you know where could be a problem?

Thanks a lot.

bouwew · December 23, 2024, 10:33am

Please try: {{ value_json ["PAGE"]["INPUT"][?(@["NAME"]=="__R11082_REAL_.1f")]["@VALUE"] }}

Looks like you missed the “?” and “@” characters.

martinmariknet · December 23, 2024, 5:29pm

Thanks lot, unfortunately I’m getting following errors:

2024-12-23 18:26:54.472 ERROR (MainThread) [homeassistant.config] Invalid config for 'rest' at configuration.yaml, line 31: invalid template (TemplateSyntaxError: unexpected char '?' at 32) for dictionary value 'rest->0->sensor->0->value_template', got '{{ value_json ["PAGE"]["INPUT"][?(@["NAME"]=="__R11082_REAL_.1f")]["@VALUE"] }}', please check the docs at https://www.home-assistant.io/integrations/rest
2024-12-23 18:26:54.473 ERROR (MainThread) [homeassistant.setup] Setup failed for 'rest': Invalid config.

Here’s my current part of configuration.yaml:

rest:
  - authentication: basic
    username: "rdstatadm"
    password: "AO1KuR5i"
    resource: http://192.168.3.12/TOPENI.XML
    scan_interval: 120
    sensor:
     - name: "Teplota obyvak"
       value_template: '{{ value_json ["PAGE"]["INPUT"][?(@["NAME"]=="__R11082_REAL_.1f")]["@VALUE"] }}'

bouwew · December 23, 2024, 7:36pm

I went here: https://xmltodict-debugger.glitch.me/ and copied your data in the XML-field, then pressed submit. This gave me the converted json.
Next, I pressed Copy and copied the data in https://jsonpath.com/, in the Inputs window, replacing what was there as an example. I removed the converted: part at the beginning and the extracted null at the end.

After some digging I came up with this expression:
PAGE[INPUT][?(@[‘\x40NAME’]==‘_R11086_REAL.1f’)]

The x40 is a hex representation of @, as described here: json - How do you escape the @ symbol in jsonpath? - Stack Overflow

You can see the result above on the right.
I put [@VALUE] behind the above expression but that doesn’t work. While PAGE[INPUT][0][@VALUE] does work.

So I’m thinking you need to do this in two steps:
First create something like: ` result = {{ value_lson PAGE[INPUT][?(@[‘\x40NAME’]==‘_R11086_REAL.1f’)] }}
Next: result_2 = {{ value_json result[@VALUE] }}

Unfortunately, I have no idea yet how to make this work in HA

martinmariknet · December 24, 2024, 9:46am

Thx for reply. Using this statement I’m getting following configuration validation error:

Configuration errors
Error loading /config/configuration.yaml: while parsing a block mapping
  in "/config/configuration.yaml", line 30, column 8
expected <block end>, but found '<block mapping start>'
  in "/config/configuration.yaml", line 31, column 9

Troon · December 24, 2024, 10:01am

This should work:

value_template: "{{ value_json['PAGE']['INPUT']|selectattr('@NAME','eq','__R11082_REAL_.1f')|map(attribute='@VALUE')|first }}"

Tested, using this tool to convert your XML to JSON in the same way HA does, removing its converted wrapper element:

martinmariknet · December 24, 2024, 3:04pm

Thx for reply. I’m definitively not that advance to figure that out.

Awesome, great, thx a lot, works perfectly.