Another take on Australian weather forecasts using BoM feeds

I’ve been using the BoM sensor for current weather data for a little while now, but was inspired by this thread to start fetching forecast data as well: Weather forecast for Australia using BOM website

But, while that thread uses the scrape sensor to scrape an HTML page, I wanted to use one of the data feeds. Doing this is probably pretty advanced, and definitely a lot more involved, because the files are in XML format and HA doesn’t seem to have the tools to easily parse these built in. But it also exposes a lot more useful information, and once you get a feel for it I think it’s a lot more flexible.

These sensors and automation are set up to send me notification of the weather forecast and some current conditions in the morning.

To start, I had to install the libxml2-utils package on my Raspberry Pi, just using the regular package manager.

After that I had to find the data file I was interested in. I went to the forecast page (for Sydney, http://www.bom.gov.au/nsw/forecasts/sydney.shtml ) and took a note of the “Product ID” at the bottom of the page (for the Sydney forecast, IDN10064). Data files are available from an FTP service at ftp://ftp.bom.gov.au/anon/gen/fwo/ , look for a file named after your product ID with a .xml extension. For Sydney, the full URL is ftp://ftp.bom.gov.au/anon/gen/fwo/IDN10064.xml .

I tried using the downloader component for this, but currently the downloader doesn’t seem to work for FTP (I’ve opened #10968 which will hopefully fix this). So I set up a shell command to download a file, then an automation to run the command once in the evening:

### configuration.yaml
# Substitute the URL for your forecast here
shell_command:
  download_forecast: curl -s --retry 3 --output $HOME/.homeassistant/downloads/forecast/sydney-metro.xml ftp://ftp.bom.gov.au/anon/gen/fwo/IDN10064.xml

### automations.yaml
- id: Download BoM Sydney forecast
  alias: Download BoM Sydney forecast
  trigger:
    platform: time
    hours: 20
    minutes: 12
    seconds: 0
  action:
    service: shell_command.download_forecast

Then I set up some command_line sensors to extract the data I was interested in. These sensors use the xmllint command line tool, and this is where things get hairy. I’ve written queries that work for my use case in the Sydney forecast, but if you’re trying to adapt this you’ll probably need to tweak the queries. For that, you’d need to read the XML file to get an idea of the structure, have a look at how to construct basic XPath expressions, and do some trial and error.

Anyway, I wanted to get the full long forecast, the min and max forecast temperatures, chance and amount of rain, and finally the forecast icon code. These sensors run on match on the forecast-period element with index of 1. That is, they match on tomorrow’s forecast. I only download the forecast once, late in the evening, then these sensors are valid for the next day. In future I’d like to experiment with updating the forecast during the day, but this works for me now.

### configuration.yaml
sensor:
  - platform: command_line
    name: weather forecast
    command: "xmllint --xpath 'string(//product/forecast[1]/area[@aac=\"NSW_ME00
1\"]/forecast-period[@index=\"1\"]/text)' $HOME/.homeassistant/downloads/forecas
t/sydney-metro.xml"
    value_template: '{{ value | truncate(255) }}'

  - platform: command_line
    name: weather forecast icon
    command: "xmllint --xpath 'string(//product/forecast[1]/area[@aac=\"NSW_PT13
1\"]/forecast-period[@index=\"1\"]/element[@type=\"forecast_icon_code\"])' $HOME
/.homeassistant/downloads/forecast/sydney-metro.xml"

  - platform: command_line
    name: weather forecast rain chance
    command: "xmllint --xpath 'string(//product/forecast[1]/area[@aac=\"NSW_PT13
1\"]/forecast-period[@index=\"1\"]/text[@type=\"probability_of_precipitation\"])
' $HOME/.homeassistant/downloads/forecast/sydney-metro.xml"

  - platform: command_line
    name: weather forecast rain amount
    command: "xmllint --xpath 'string(//product/forecast[1]/area[@aac=\"NSW_PT13
1\"]/forecast-period[@index=\"1\"]/element[@type=\"precipitation_range\"])' $HOM
E/.homeassistant/downloads/forecast/sydney-metro.xml"

  - platform: command_line
    name: weather forecast minimum temp
    command: "xmllint --xpath 'string(//product/forecast[1]/area[@aac=\"NSW_PT13
1\"]/forecast-period[@index=\"1\"]/element[@type=\"air_temperature_minimum\"])'
$HOME/.homeassistant/downloads/forecast/sydney-metro.xml"

  - platform: command_line
    name: weather forecast maximum temp
    command: "xmllint --xpath 'string(//product/forecast[1]/area[@aac=\"NSW_PT131\"]/forecast-period[@index=\"1\"]/element[@type=\"air_temperature_maximum\"])' $HOME/.homeassistant/downloads/forecast/sydney-metro.xml"

The final part I needed was the existing BoM sensor. All I’m using so far is current temperature, so:

### configuration.yaml
sensor:
  - platform: bom
    station: IDN60901.94768
    name: Sydney
    monitored_conditions:
      - air_temp

Finally I have an automation to send a push notification with the forecast every morning. This notification is one of a few places I’m using the forecast icon code. The notification template defines an array of emoji representing the different weather icons (sunny, cloudy, rain, storm, etc). Then the title template uses the icon code as an index of this array to create a dynamic title.

### automations.yaml
- id: weather forecast notification
  alias: weather forecast notification
  trigger:
    platform: time
    at: 06:15
  action:
    service: notify.html5_notifier
    data_template:
      title: >
        {%- set icons = ["", "\u2600\uFE0F", "\uD83C\uDF19", "\u26C5", "\u2601\uFE0F", "", "\uD83C\uDF01", "", "\uD83C\uDF27\uFE0F", "\uD83C\uDF2C\uFE0F", "\uD83C\uDF01", "\uD83C\uDF26\uFE0F", "\uD83C\uDF27\uFE0F", "\uD83D\uDE37", "‽", "\u2603\uFE0F", "\uD83C\uDF29\uFE0F", "\uD83C\uDF27\uFE0F", "\uD83C\uDF27\uFE0F", "\uD83C\uDF00"] -%}
        {%- set index = states.sensor.weather_forecast_icon.state | int -%}
        {{ icons[index] }} Weather forecast
      message: |
        Good morning!
        {{ states('sensor.weather_forecast') }}
        {{ states.sensor.weather_forecast_minimum_temp.state }}°-{{ states.sensor.weather_forecast_maximum_temp.state }}°, currently {{ states.sensor.bom_sydney_air_temp_c.state }}°. {{ states.sensor.weather_forecast_rain_chance.state }} chance of {{ states.sensor.weather_forecast_rain_amount.state }} rain.
2 Likes

I love this idea as an alternative to the scrape sensors… I didn’t like the idea of pulling down the shtml page several times a minute.

I am currently trying to implement this on hass.io but i’m having no luck with the xmllint command. It runs inside my own SSH session but will not run from the command line sensor.

Does the command fail and produce an error in your log, or is there nothing logged but you’re getting an unknown value for the sensor? If there’s an error, the only thing I can think of is making sure you specify a full path for xmllint (on my system, /usr/bin/xmllint, might be different for hass.io?) in your configuration.

If it’s running but not finding any data, then the only thing I can think of is differences in xml format. I’ve recently been learning that the BoM forecast is frequently spread across a couple of different XML files. You may have to do some fishing and adjust the xmllint path accordingly. :confused:

My home assistant system was offline for a few weeks while I moved from close to the centre of Sydney out west. Now that things are settling again I’m rebuilding my setup, and have found that outside the metro area there’s a little bit more work involved recreating this setup. Hopefully explaining how I found the info I wanted will help others in the same situation.

To start with, I went to my local forecast page at http://www.bom.gov.au/nsw/forecasts/blacktown.shtml , and at the bottom of the page it says “Product derived from IDN11060 and IDN10064”. So I looked up both of those IDs on the FTP site and what I’ve found is this:

  • ftp://ftp.bom.gov.au/anon/gen/fwo/IDN10064.xml contains the long detailed forecast precis for metropolitan Sydney. This is contained in the <area aac="NSW_ME001" description="Sydney" type="metropolitan" parent-aac="NSW_FA001"> area tags.
  • ftp://ftp.bom.gov.au/anon/gen/fwo/IDN11060.xml contains a lot of the less major areas in NSW. The forecast page says it’s for Blacktown, so I searched the file for that and found this area tag: <area aac="NSW_PT255" description="Blacktown" type="location" parent-aac="NSW_PW005">. The forecasts enclosed by this tag contain localised temperature and rainfall forecasts, but only a very short precise.

So my approach is now to grab the long precis from the NSW_ME001 area in the first file, and the rest of the forecast data from the NSW_PT255 area in the second file.

To start with, I need two shell commands to download my two files:

### configuration.yaml
shell_command:
  download_sydney_forecast: curl -s --retry 3 --output $HOME/.homeassistant/downloads/forecast/sydney-metro.xml ftp://ftp.bom.gov.au/anon/gen/fwo/IDN10064.xml
  download_nsw_forecast: curl -s --retry 3 --output $HOME/.homeassistant/downloads/forecast/nsw-regional.xml ftp://ftp.bom.gov.au/anon/gen/fwo/IDN11060.xml

### automations.yaml
- id: Download BoM forecast
  alias: Download BoM forecast
  trigger:
    platform: time
    hours: 20
    minutes: 12
    seconds: 0
  action:
    - service: shell_command.download_sydney_forecast
    - service: shell_command.download_nsw_forecast

The sensor for the long precis is using the NSW_ME001 area, so this works for me:

sensor:
  - platform: command_line
    name: weather forecast precis
    command: "xmllint --xpath 'string(//product/forecast[1]/area[@aac=\"NSW_ME001\"]/forecast-period[@index=\"1\"]/text)' $HOME/.homeassistant/downloads/forecast/sydney-metro.xml | awk 'length>255{$0=substr($0,0,252)\"…\"}1'"
    value_template: '{{ value | truncate(255) }}'

But the rest of the data is in another file and uses the NSW_PT255 area, so has a slightly different path. I’m using these sensors:

sensor:
  - platform: command_line
    name: weather forecast icon
    command: "xmllint --xpath 'string(//product/forecast[1]/area[@aac=\"NSW_PT255\"]/forecast-period[@index=\"1\"]/element[@type=\"forecast_icon_code\"])' $HOME/.homeassistant/downloads/forecast/nsw-regional.xml"
  - platform: command_line
    name: weather forecast rain chance
    command: "xmllint --xpath 'string(//product/forecast[1]/area[@aac=\"NSW_PT255\"]/forecast-period[@index=\"1\"]/text[@type=\"probability_of_precipitation\"])' $HOME/.homeassistant/downloads/forecast/nsw-regional.xml"
  - platform: command_line
    name: weather forecast rain amount
    command: "xmllint --xpath 'string(//product/forecast[1]/area[@aac=\"NSW_PT255\"]/forecast-period[@index=\"1\"]/element[@type=\"precipitation_range\"])' $HOME/.homeassistant/downloads/forecast/nsw-regional.xml"
  - platform: command_line
    name: weather forecast minimum temp
    command: "xmllint --xpath 'string(//product/forecast[1]/area[@aac=\"NSW_PT255\"]/forecast-period[@index=\"1\"]/element[@type=\"air_temperature_minimum\"])' $HOME/.homeassistant/downloads/forecast/nsw-regional.xml"
  - platform: command_line
    name: weather forecast maximum temp
    command: "xmllint --xpath 'string(//product/forecast[1]/area[@aac=\"NSW_PT255\"]/forecast-period[@index=\"1\"]/element[@type=\"air_temperature_maximum\"])' $HOME/.homeassistant/downloads/forecast/nsw-regional.xml"

This mix populates my sensors with info identical to the original forecast page, which is close enough for me.