Value_template for XML Reading

Configuration
1

Hello HA community,
please help with sensor for reading values from XML file via rest platform
I would like to read the value <description> but on the “geocode” condition example <value>5202</value>
There is example of XML file (edited):

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<alert xmlns="urn:oasis:names:tc:emergency:cap:1.2">
  <identifier>2.49.0.0.203.0.CZ.210511104547.XOCZ50_OKPR_000135</identifier>
  <sender>[email protected]</sender>
  <sent>2021-05-11T12:45:47+02:00</sent>
  <status>Actual</status>
  <msgType>Update</msgType>
  <scope>Public</scope>
  <code>SIVS:CHMI/111/89/2018</code>
  <code>HPPS</code>
  <references>[email protected],2.49.0.0.203.0.CZ.210510091947.XOCZ50_OKPR_000134,2021-05-10T11:19:47+02:00</references>
  <incidents></incidents>
  <info>
    <language>cs</language>
    <category>Met</category>
    <event>Silný vítr</event>
    <responseType>Execute</responseType>
    <responseType>Avoid</responseType>
    <urgency>Immediate</urgency>
    <severity>Moderate</severity>
    <certainty>Likely</certainty>
    <audience>veřejnost, HZS, web, Meteoalarm</audience>
    <eventCode>
      <valueName>SIVS</valueName>
      <value>III.1</value>
    </eventCode>
    <onset>2021-05-11T12:30:30+02:00</onset>
    <expires>2021-05-11T20:00:00+02:00</expires>
    <senderName>ČHMÚ, Racko</senderName>
    <description>Silný jihovýchodní až jižní vítr s nárazy nad 65 km/h.</description>
    <instruction>Možná poškození stromů a lesních porostů, možné menší škody na budovách. Nebezpečí úrazu uvolněnými předměty a zlomenými větvemi, možné komplikace v dopravě. Doporučuje se zajistit okna, dveře, odstranit nebo upevnit volně uložené předměty, zabezpečit skleníky apod. Dbát zvýšené opatrnosti při pohybu venku a při řízení vozidel. Na horách omezit túry a nevydávat se zejména do hřebenových partií.</instruction>
    <web>http://www.chmi.cz/files/portal/docs/meteo/om/zpravy/index.html</web>
    <parameter>
      <valueName>situation</valueName>
      <value>Zvlněná studená fronta postupuje přes Německo zvolna k východu. Před ní k nám proudí velmi teplý vzduch od jihu.</value>
    </parameter>
    <parameter>
      <valueName>eventEndingTime</valueName>
      <value>2021-05-11T20:00:00+02:00</value>
    </parameter>
    <parameter>
      <valueName>awareness_level</valueName>
      <value>2; yellow; Moderate</value>
    </parameter>
    <parameter>
      <valueName>awareness_type</valueName>
      <value>1; Wind</value>
    </parameter>
    <area>
      <areaDesc>Královéhradecký kraj (Dobruška, Náchod, Nové Město nad Metují, Rychnov nad Kněžnou)</areaDesc>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5202</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5209</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5211</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5213</value>
      </geocode>
    </area>
    <area>
      <areaDesc>Pardubický kraj (Česká Třebová, Hlinsko, Chrudim, Králíky, Lanškroun, Litomyšl, Moravská Třebová, Polička, Přelouč, Svitavy, Ústí nad Orlicí, Žamberk)</areaDesc>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5301</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5302</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5304</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5305</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5306</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5307</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5308</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5310</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5311</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5312</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5313</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>5315</value>
      </geocode>
    </area>
    <area>
      <areaDesc>Kraj Vysočina (Bystřice nad Pernštejnem, Havlíčkův Brod, Chotěboř, Jihlava, Moravské Budějovice, Náměšť nad Oslavou, Nové Město na Moravě, Telč, Třebíč, Velké Meziříčí, Žďár nad Sázavou)</areaDesc>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6101</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6102</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6104</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6105</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6106</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6107</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6108</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6112</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6113</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6114</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6115</value>
      </geocode>
    </area>
    <area>
      <areaDesc>Jihomoravský kraj</areaDesc>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6201</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6202</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6203</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6204</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6205</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6206</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6207</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6208</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6209</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6210</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6211</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6212</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6213</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6214</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6215</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6216</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6217</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6218</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6219</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6220</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>6221</value>
      </geocode>
    </area>
    <area>
      <areaDesc>Olomoucký kraj</areaDesc>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7101</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7102</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7103</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7104</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7105</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7106</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7107</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7108</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7109</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7110</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7111</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7112</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7113</value>
      </geocode>
    </area>
    <area>
      <areaDesc>Zlínský kraj</areaDesc>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7201</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7202</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7203</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7204</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7205</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7206</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7207</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7208</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7209</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7210</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7211</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7212</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>7213</value>
      </geocode>
    </area>
    <area>
      <areaDesc>Moravskoslezský kraj</areaDesc>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8101</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8102</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8103</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8104</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8105</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8106</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8107</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8108</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8109</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8110</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8111</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8112</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8113</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8114</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8115</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8116</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8117</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8118</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8119</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8120</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8121</value>
      </geocode>
      <geocode>
        <valueName>CISORP</valueName>
        <value>8122</value>
      </geocode>
    </area>
  </info>
</alert>

Maybe Picture of XML:

xml
xml929×676 43.4 KB

Here’s my attempt at the code:

sensor:
  - platform: rest
    name: "Výstrahy počasí"
    resource: http://192.168.10.10:8123/local/bourky.xml
    value_template: >
      {% set hodnota = value_json["alert"]["info"]["area"]["geocode"] | selectattr('value','eq','5202')| list %}
      {{ hodnota["description"] }}
    scan_interval: 3600

I’m sure it’s some stupid thing, but after 3 hours I’m out of ideas.
Thank you very much in advance :pray:

2

The automatic XML to JSON conversion in Home Assistant puts an @ symbol in front of keys.

So <valueName>awareness_level</valueName> becomes "@valueName": "awareness_level"

Use this utility: Free Online XML to JSON Converter - FreeFormatter.com to see a similar output.

I tried but the XML you pasted does not appear to be complete.

3

This isn’t much of a solution, because it’s data from Czech Hydrometeorogical Institute and in case of alerts, a new XML file is generated.The actual file is at this link, only if there is no warning, the description is empty.
https://www.chmi.cz/files/portal/docs/meteo/om/bulletiny/XOCZ50_OKPR.xml
So I saved the above sample file and am trying to play. Thank you.

4

I meant use the converter tool to see what it does to your xml when it is converted to json so you know how to construct the template.

Pasting the data from your xml resource into the converter actually reveals it does not use attributes so there will be no @ symbols in the json. The result is too big to paste here though.

However it does use lists so you need something like this

value_json.info[0]["area"]["area"][7]["geocode"][1]["value"]

That gives you 5202. There are no keys under it for you to list though.

Basically the data structure is not set up for the search you are trying to do.

Paste the json from the xml converter tool into this tool to inspect it:

See if you can work out your template from the structure revealed there.

5

Thank you for the kick me in the pants :grinning:
Tools like xml->Json converter and JSON Path Finder are really great.
So I found a way to get my geocode from xml with value 5202

value_template: '{{ value_json["alert"]["info"][8]["area"][0]["geocode"][0]["value"] }}

Now I just need some magic to list all the events (“info” → “event or description”) that contains this “geocode” with the value “5202”

6

Hallo Venca,
I guess a solution can be made with multiscrape integration (can be installed via HACS), without the need to convert anything from XML to Json or backwards…

below a sample for similar case - WattRouter integration, when XML structure looks like this:

 <meas>
- <I1>
  <P>-0.11</P> 
  </I1>
- <I2>
  <P>-0.04</P> 
  </I2>
- <I3>
  <P>-0.32</P> 
  </I3>

and then yaml code for a sensor is quite simple (somewhere at the upper part of configuration.yaml):

multiscrape:
# Integrace WattRouter - základní entity (okamžité měření)
# ========================================================
  - resource: !secret Watt_IP_meas
    scan_interval: 300
    sensor:
      - unique_id: wr_vykon_f1
        name: "Výkon/Odběr L1"
        select: "meas > I1 > P"
        unit_of_measurement: "kW"
      - unique_id: wr_vykon_f2
        name: "Výkon/Odběr L2"
        select: "meas > I2 > P"
        unit_of_measurement: "kW"
      - unique_id: wr_vykon_f3
        name: "Výkon/Odběr L3"
        select: "meas > I3 > P"
        unit_of_measurement: "kW"

plz let us know if you succeeded with convertion for CZ meteo info…
cheers!
:slight_smile:

1 Like
7

Hello Venca,

I have tried to write code in python, which can read xml from url directly and convert it to json. From what I know, there should not be issue to get it working in home assistant (with pyscript or appdaemon).

Here is my example code:

import requests
import re
import xmltodict
import json

link = "https://www.chmi.cz/files/portal/docs/meteo/om/bulletiny/XOCZ50_OKPR.xml"
f = requests.get(link)
f.encoding
a = f.text

c = re.sub(r'\n', '', a)
b = c.find("<alert")

if b > 0:
    d = c[b:]

    e = json.dumps(xmltodict.parse(d))
    x = json.loads(e)

    event_teplota = x["alert"]["info"][0]["event"]  ## read first event
    oblast_teplota = x["alert"]["info"][0]["area"][10]["geocode"][2]["value"]  ## read geocode

    print(event_teplota)
    print(oblast_teplota)

elif b == -1:
    print("Webpage does not response")
8

Hi Radek,
thank you for Reply. I tried HACS - Pyscript
in Configuration.yaml i have

pyscript:
  allow_all_imports: true

but Logs HA reports an error:

Exception in </config/pyscript/meteo.py> line 7: f = requests.get(link) ^ RuntimeError: Blocking calls must be done in the executor or a separate thread; Use `await hass.async_add_executor_job()`; at custom_components/pyscript/eval.py, line 1906: return func(*args, **kwargs)

Unfortunately I don’t know python syntax. And from what I understand, so it reads “only” the main first event? Or can it only find events for the selected geocode (example 4104)? Anyway, thank you very much for moving :slightly_smiling_face:

9

Hi Venca,

I work on this python script in IDLE. So we have to rewrite this code to be able to work with pyscript.
For now it reads only the first event:
event_teplota = x["alert"]["info"][0]["event"]
If you would like to read second event (in czech language) you have to change 0 to 2:
event_teplota = x["alert"]["info"][2]["event"]

From documentation of pyscript I see that we have to use task.executor and rewrite few things:

resp = requests.get(url) ## this will not work with pyscript
resp = task.executor(requests.get, url) ## we have to use task.executor
10

Hi,

here is my configuration.yaml

pyscript:
  allow_all_imports: true

logger:
  default: info 
  logs: 
    custom_components.pyscript: info

And here is code which works well for me:

@service
def nacti_data():
    import requests
    import re
    import xmltodict
    import json

    link = "https://www.chmi.cz/files/portal/docs/meteo/om/bulletiny/XOCZ50_OKPR.xml"
    f = task.executor(requests.get, link)
    f.encoding
    a = f.text
    
    c = task.executor(re.sub,r'\n', '', a)
    b = task.executor(c.find,"<alert")

    if b > 0:
        d = c[b:]
        e = task.executor(json.dumps,task.executor(xmltodict.parse,d))
        x = task.executor(json.loads,e)
        temperature_warning = x['alert']['info'][0]['event']
        temperature_description = x['alert']['info'][0]['description']
        log.info(f"First alert is: {temperature_warning}")
        log.info(f"First alert description: {temperature_description}")
        sensor.chmi_test = temperature_warning
    elif b == -1:
        log.info(f"Webpage does not response!")

Then you can use the Service tab in the Developer Tools page to call the service pyscript.nacti_data, then you should see new sensor called sensor.chmi_test

1 Like
11

Radek, you are “Python Devil” :slight_smile: It’s on a very good way and this is a workable solution.I know I’m very demanding, but I’d still like to push it higher. Petr Brouzda on his GitHub can pull all events by city ID (CISORP)
ChmiWarnings - GitHub
Example CZ city “Kraslice” has CISORP 4104, and if I use his PHP scripts on an external web server, I get all events for the selected area:
https://vakr.cloud/ChmiWarnings/chmi/vystrahy/4104
Here I found his PHP source code, where he probably parses the whole XML file according to the CISORP code:
https://github.com/petrbrouzda/ChmiWarnings/blob/535c4b146b0165b83438f5638b46479d8421b2c9/aplikace/app/Services/ChmiParser.php
I would like this solution in Python (where I can set the code CISORP and get all the events), because then I don’t have to use some external web server to do the parsing.
Anyway, you are really champion and thank you for your time and willingness :slight_smile:

12

Trying to replicate this for my script, below, but nothing happens (including nothing in the Core logs) when pyscript reloads the script or I call the function as a service. Did you define the sensor in the config.yaml? If so, may I see how you did so?

@service
def goatrock_tides():
  import aiohttp
  import asyncio
  from bs4 import BeautifulSoup
  import json

  async def fetch_data(url, headers=None):
      async with aiohttp.ClientSession() as session:
          async with session.get(url, headers=headers) as response:
              return await response.text()

  async def main():
      # URL of the web page containing the table
      url = 'https://www.tideschart.com/United-States/California/Sonoma-County/Goat-Rock-Beach/Weekly/'
    
      # Define headers (if needed)
      headers = {
          'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36',
          'Accept-Language': 'en-US,en;q=0.9',
    }

      # Fetch the web page content asynchronously with headers
      html_content = await fetch_data(url, headers=headers)

      # Parse the HTML content using BeautifulSoup
      soup = BeautifulSoup(html_content, 'html.parser')

      # Find the table element with the specified class
      table = soup.find('table', class_='table table-hover tidechart mb-4')

      # Initialize an empty list to store the table data
      table_data = []

      # Find all rows within the table body
      rows = table.tbody.find_all('tr')

      # Loop through each row and extract the data
      for row in rows:
          columns = row.find_all('td')
          day = columns[0].text.strip()
          first_tide = columns[1].text.strip().replace('\u25BC', ',').replace('\u25B2', ',')
          second_tide = columns[2].text.strip().replace('\u25BC', ',').replace('\u25B2', ',')
          third_tide = columns[3].text.strip().replace('\u25BC', ',').replace('\u25B2', ',')
          fourth_tide = columns[4].text.strip().replace('\u25BC', ',').replace('\u25B2', ',')
          sunrise = columns[5].text.strip().replace('\u25b2', '').replace('\u25bC', '').replace(' ', '')
          sunset = columns[6].text.strip().replace('\u25b2', '').replace('\u25bC', '').replace(' ', '')

          # Create a dictionary for the current row
          row_data = {
              "Day": day,
              "1st Tide": first_tide,
              "2nd Tide": second_tide,
              "3rd Tide": third_tide,
              "4th Tide": fourth_tide,
              "Sunrise": sunrise,
              "Sunset": sunset
      }

          # Append the row data to the list
          table_data.append(row_data)

      # Convert the table data list to JSON
      json_output = json.dumps(table_data, indent=4)

      # Print or save the JSON data as needed
      log.info(json_output)
      sensor.goat_tides = json_output

  if __name__ == '__main__':
      asyncio.run(main())
13

hey, I’m a noob here, but I suppose using task.executor makes the difference

14

You’re right, I’m basically doing it all wrong :slight_smile: Starting over again with better examples and different libraries.