Problem extracting data in custom component Python

Now, looking at this XPath: From the two XML snippets you posted, I cannot see how this XPath would match with type='location'. There are only two areas - one with type “region” and one with type “metropolitan”.

        <area aac="NSW_FA001" description="New South Wales" type="region">
        <area aac="NSW_ME004" description="Central Coast" type="metropolitan" parent-aac="NSW_FA001">

The difference between the two XML snippets appears to be that “day_1” has the uv_alert under index="1" and “day_0” has the uv_alert under index="0".
My understanding from the XML structure is that the index indicates for which day that forecast is, i.e. 0=today, 1=tomorrow, etc.

So, maybe you need to rethink what your sensors represent? Is sensor _0 always today regardless of the time? So, maybe if it’s after 4:40pm, and there is no uv_alert available anymore for index="0", you need to set the sensor’s state to None/Unknown?

Thing is other areas of Australia will have that key there… the forecast one works.

If I take the text off the end I get pages of errors continually… the test is not working at all…

Sorry, can’t really help here. I did take a quick look at the code (at least the diffs between sensor.py & sensor.py.work), but this seems to have more to do with parsing XML than Python, and that’s not really “in my wheelhouse.”

Alright, hang in there. I got this running on my dev machine. I’ll add a few more fixes and share the code tonight.

1 Like

@DavidFW1960: I just sent you the code directly. Please let us know if that’s getting you closer to what you want to achieve, and please share the final code/component, maybe others find that useful, too.

1 Like

Will do. It will be on HACS as soon as I get a chance to test it. There’s a few people who want it.

Thanks a lot for helping sort this out!

1 Like

Got someone who is in Perth and he’s not getting the fire_danger. Looking at the XML file, it looks like this:

    <forecast>
        <area aac="WA_FA001" description="Western Australia" type="region">
            <forecast-period start-time-local="2019-11-23T04:40:07+08:00" end-time-local="2019-11-23T04:40:07+08:00" start-time-utc="2019-11-22T20:40:07Z" end-time-utc="2019-11-22T20:40:07Z">
                <text type="warning_summary_footer">For latest warnings go to www.bom.gov.au, subscribe to RSS feeds, call 1300 659 210* or listen for warnings on relevant TV and radio broadcasts.</text>
                <text type="product_footer">* Calls to 1300 numbers cost around 27.5c incl. GST, higher from mobiles or public phones.</text>
            </forecast-period>
        </area>
        <area aac="WA_MW009" description="Perth Coast: Two Rocks to Dawesville" type="coast" parent-aac="WA_FA001">
            <warning-summary type="marine_forecast" start-time-local="2019-11-23T05:00:00+08:00" end-time-local="2019-11-25T00:00:00+08:00" start-time-utc="2019-11-22T21:00:00Z" end-time-utc="2019-11-24T16:00:00Z">Strong Wind Warning for Sunday for Perth Coast</warning-summary>
        </area>
        <area aac="WA_ME001" description="Perth" type="metropolitan" parent-aac="WA_FA001">
            <forecast-period index="0" start-time-local="2019-11-23T00:00:00+08:00" end-time-local="2019-11-24T00:00:00+08:00" start-time-utc="2019-11-22T16:00:00Z" end-time-utc="2019-11-23T16:00:00Z">
                <text type="forecast">Sunny. Winds easterly 15 to 25 km/h shifting south to southwesterly 20 to 30 km/h in the middle of the day. Winds gusting up to 60 km/h about the hills this morning.</text>
                <text type="fire_danger">
                    <p>Perth Coastal Plain: High</p>
                    <p>Perth Hills: High</p>
                </text>
                <text type="uv_alert">Sun protection 7:50am to 4:10pm, UV Index predicted to reach 12 [Extreme]</text>
            </forecast-period>
            <forecast-period index="1" start-time-local="2019-11-24T00:00:00+08:00" end-time-local="2019-11-25T00:00:00+08:00" start-time-utc="2019-11-23T16:00:00Z" end-time-utc="2019-11-24T16:00:00Z">
                <text type="forecast">Sunny. Winds south to southeasterly 15 to 20 km/h tending south to southwesterly 25 to 35 km/h in the morning then tending south to southeasterly 15 to 25 km/h in the late evening.</text>
            </forecast-period>

So there’s 2 regions for the fire_danger. It is coming into HA as blank. Is there a way to read the 2 values in? Then the user can use a Template to split and extract what he wants. It’s IDW12300

TIA.

Ha! can’t post it. Doh!

OK, so the XML either provides plain text inside the text tag, or there could be multiple p tags inside the text tag.

In the following, I’ve added .strip() to the the fire danger reading to ensure all the whitespace is removed. Then I’m checking if the remaining string is empty, and if so use an XPath expression to find all the p sub-tags, and then concatenate the into a comma-separated list.

        if condition == 'fire_danger':
            if PRODUCT_ID_LAT_LON_LOCATION[self._product_id][3] == 'City':
                _LOGGER.debug("City")
                fire_danger_data = self._data.find(_FIND_QUERY_4.format(index))
                _LOGGER.debug("fire_danger_data = %s", fire_danger_data)
                if fire_danger_data is not None:
                    fire_danger = fire_danger_data.text.strip()
                    if fire_danger == '':
                        # Check if there are sub-tags.
                        fire_danger_data_paragraphs = fire_danger_data.findall("./p")
                        if fire_danger_data_paragraphs is not None:
                            paragraphs = [paragraph.text for paragraph in fire_danger_data_paragraphs]
                            return ", ".join(paragraphs)
                    return fire_danger
            else:

That seems to work fine. Thanks again!

Ha! Me again… lol…
A user has reported an error here: Australian Weather Forecast using BOM Public FTP
He is using IDV10703 which has a slightly different format.
He says he sees the error for uv_alert and fire_danger and icon but I suspect icon is ok… (Although having said that, the original dev had avoided documenting the icon monitored condition so maybe there is a problem with that and he knew it…)
Are you able to take a look? It seems it has different area assignments to mine (IDN11052)
I did try to add an extra condition and search for ‘Location’ but caused more trouble than I solved lol.
Anyway if you can take a look I’d appreciate it.

The code apparently already caters for distinguishing between ‘City’ and ‘Town’. You just had to apply the same error handling for the ‘Town’ part.

OLD (lines 422-431)

            else:
                _LOGGER.debug("not City")
                uv_alert_data = self._data.find(
                    _FIND_QUERY.format(index, 'uv_alert')).text
                _LOGGER.debug("uv_alert_data = %s", uv_alert_data)
                if uv_alert_data is not None:
                    uv_alert = self._data.find(
                        _FIND_QUERY.format(index, 'uv_alert')).text
                    _LOGGER.debug("uv_alert = %s", uv_alert)
                    return uv_alert

NEW

            else:
                _LOGGER.debug("not City")
                uv_alert_data = self._data.find(_FIND_QUERY.format(index, 'uv_alert'))
                _LOGGER.debug("uv_alert_data = %s", uv_alert_data)
                if uv_alert_data is not None:
                    uv_alert = uv_alert_data.text
                    _LOGGER.debug("uv_alert = %s", uv_alert)
                    return uv_alert

OLD (lines 447-456):

            else:
                _LOGGER.debug("not City")
                fire_danger_data = self._data.find(
                    _FIND_QUERY.format(index, 'fire_danger')).text
                _LOGGER.debug("fire_danger_data = %s", fire_danger_data)
                if fire_danger_data is not None:
                    fire_danger = self._data.find(
                        _FIND_QUERY.format(index, 'fire_danger')).text
                    _LOGGER.debug("fire_danger = %s", fire_danger)
                    return fire_danger

NEW

            else:
                _LOGGER.debug("not City")
                fire_danger_data = self._data.find(
                    _FIND_QUERY.format(index, 'fire_danger'))
                _LOGGER.debug("fire_danger_data = %s", fire_danger_data)
                if fire_danger_data is not None:
                    fire_danger = fire_danger_data.text
                    _LOGGER.debug("fire_danger = %s", fire_danger)
                    return fire_danger
1 Like

Yeah that was what I tried but couldn’t make that work… I’ll try the change. Thanks again.

1 Like