Scrape the fire risk 🔥

Configuration
1

Hello fellow home Assitors!

I having a hard time to successful scrape the fire risk for portugal from the “The Portuguese Institute for Sea and Atmosphere” in short IPMA (www.ipma.pt/en).

They offer a Wildfire Risk indicator on which base some things a allowed/prohibited by law (for example bbq :cut_of_meat:, the use of firecrackers :firecracker: or smoking the bees :honeybee: out to get their honey :honey_pot:). The indicator itself has 5 different stages like this:

image
Found on this domains:

The site showing the actual risk itself is dynamic and presented inside a map:

[Sorry new user can only put one images in a post :slightly_frowning_face:]
https://www.ipma.pt/en/riscoincendio/rcm.pt/#0#0105

I tried already a lot’s of combinations but where never able to get the actual risk level (1 - 5) scraped, here is my last try:

  - platform: scrape
    name: Risco
    resource: http://www.ipma.pt/en/riscoincendio/rcm.pt/index.jsp#0#0105
    select: 'img'
    index: 14
    select: 'img:nth-of-type(14)'
    select: ".maincol-full .leaflet-container .leaflet-map-pane .leaflet-objects-pane .leaflet-popup-pane .leaflet-popup .leaflet-popup-content-wrapper .leaflet-popup-content table"
    attribute: class
    attribute: title
    headers:
      User-Agent: Mozilla/5.0

Some one got an idea how to do this? :fire_engine:

2

Actually I don’t need to get the image itself, the title of the image is also indicating the level:

image

But how could I do this? :face_with_monocle:

3

I just looked at the HTML code of the page itself and it appears that the data is actually stored in a lot of JavaScript code. That screenshot of the HTML structure is just what’s rendered in the web browser when you select a location.
The scrape sensor basically just downloads the HTML code and then tries to find values based on your select statements, but it doesn’t execute JavaScript to render the HTML.

Long story short: I don’t think you can use the scrape sensor on that page. And I don’t think there is any other standard integration that would help you.

If you are up to some Python development, you could have a look at a custom sensor I built a while ago that does find things on a HTML page using a regular expression, and adopt that for your needs.

1 Like
4

Maybe you can find an API for the service?
It’s usually easier to find when you know the language.

But it seems the mobile page is also JavaScript driven (at least it took forever to load so I assume…)

@exxamalte will that actually make any difference with regex?
Usually it does not, you need a headless browser to get the data after JavaScript.

1 Like
5

All the JavaScript payload is embedded in that HTML page, so regex could be used to find the relevant data and extract it.

Absolutely agree, that would be the preference. Another option to look for is some standard format such as XML/RSS or JSON - much easier to extract data from such formats.

1 Like
6

Good point. I actually already contacted IPMA last year and asked for a API for this data but (at this point) they didn’t have it for this particular information. The IPMA Weather Integration for Home Assistant from @dgomes makes use of this API.

Now I looked again and found this here:

Invocação :
https://api.ipma.pt/open-data/forecast/meteorology/rcm/rcm-d{idDay}.json
Nota: Apenas estão disponíveis dados diários. {idDay} varia entre 0 e 1, sendo que:

  • 0 - é dia equivalente ao hoje
  • 1 - amanhã

Códigos de rcm:

  • 1 - Risco reduzido
  • 2 - Risco moderado
  • 3 - Risco Elevado
  • 4 - Risco muito elevado
  • 5 - Risco máximo

Resultado (formato json): { “dataPrev”: “2020-02-03”, “dataRun”: “2020-02-02”, “fileDate”: “2020-02-03 09:05:01”, “local”: { “{DICO}”: { “data”: { “rcm”: 1 }, “DICO”: “1001”, “latitude”: 39.55, “longitude”: -8.97 },…

  • dataPrev: data da previsão
  • dataRun: data da corrida do modelo
  • fileDate: data de atualização do ficheiro (taxa de atualização horária)
  • DICO: Identificador único de concelho (de acordo com a CAOP - DGT)
  • rcm: código relativo ao tipo de risco de incêndio
  • latitude: latitude
  • longitude: longitude

Still could need some little help how I could “dump” this into home assistant the best way :relaxed:

7

Look here:

Here’s an example from somewhere else:

  - platform: rest
    resource: https://api.ambientweather.net/v1/devices?applicationKey=KEY_HERE&apiKey=OTHER_KEY_HERE
    name: weather_station_report
    json_attributes:
    scan_interval: 300
    value_template: '{{ value_json[0].info.location }}'
1 Like
8

thank’s @NK553 very valuable!

I just went through all json examples but couldn’t find one fitting my use case.

If I’m looking at the json it represents all locations in whole portugal. They are unique by the identifier called “DICO”. I some how need to limit the return of the “rcm” value based by the definided “dico”.

{"dataPrev": "2020-06-14", "dataRun": "2020-06-13", "fileDate": "2020-06-14 09:05:01", "local": {"0101": {"data": {"rcm": 1}, "dico": "0101", "latitude": 40.58, "longitude": -8.44}, "0102": {"data": {"rcm": 1}, "dico": "0102", "latitude": 40.6909, "longitude": -8.4784}, "0103": {"data": {"rcm": 1}, "dico": "0103", "latitude": 40.44, "longitude": -8.44},

For someone who knows json/rest/etc. it’s probably a no-brainer - for me it’s completely new :grimacing:

9

I could do it with my hands tied behind my back in PHP.
If you have any way of hosting a PHP file then I can do it.
But I’m not yet good with yaml and all that

10

This for examples represents all the data for the location “dico” 0102 in the json file for today.

I’m not quite sure how this is cascaded but can it be that it needs to be queried somewhat like this:

  - platform: rest
    resource: http://api.ipma.pt/open-data/forecast/meteorology/rcm/rcm-d0.json
    name: fire_risk_today
    json_attributes:
    scan_interval: 24h
    value_template: '{{ value_json[0102].rcm }}'

?

12

Ok had some more time to work with it now that the kids are in bed.


sensor:
  - platform: rest
    resource: http://api.ipma.pt/open-data/forecast/meteorology/rcm/rcm-d0.json
    name: fire_risk_today
    json_attributes:
    scan_interval: 86400 # 24 hours in seconds
    value_template: '{{ value_json["local"]["0102"]["data"].rcm }}'

image
image1105×73 6.38 KB

1 Like
13

Wow! Great @Hellis81 ! Thank you so much! :partying_face:

From here it looks like the update for the json file is daily at 09:05. Is their a way to force the update shortly after - let’s say daily at 09:10 for example?

14

I don’t know.
There is a service call called homeassistant.update_entity but when I use it on the entity it doesn’t do anything.
Perhaps because my force update is default (false) or because it does not work. But that is one thing you could try.
Change the force update to true on the sensor and create an automation to do this service call when time is 09:10

1 Like
15

image

I got this now but it’s just the static picture. I tried with picture entity but it only allows two states. The conditonal card also can only show true and false.

Is there any lovelace card out they which can show picture 1 on value 1 - picture 2 on value 2 - picture 3 on value 3 and so on?

16

I believe that is a conditional card. But I have never used them.

17

You are right @Hellis81

I build this somewhat ugly card but it works (I hope :stuck_out_tongue: )

type: vertical-stack
title: Fire Risk Today
cards:
  - type: conditional
    conditions:
      - entity: sensor.fire_risk_today
        state: '1'
    card:
      type: picture-entity
      entity: sensor.fire_risk_today
      image: 'https://www.ipma.pt/bin/icons/svg/rcm/out-Reduzido.svg'
      aspect_ratio: 30%
      show_name: false
      show_state: false
  - type: conditional
    conditions:
      - entity: sensor.fire_risk_today
        state: '2'
    card:
      type: picture-entity
      entity: sensor.fire_risk_today
      image: 'https://www.ipma.pt/bin/icons/svg/rcm/out-Moderado.svg'
      aspect_ratio: 30%
      show_name: false
      show_state: false
  - type: conditional
    conditions:
      - entity: sensor.fire_risk_today
        state: '3'
    card:
      type: picture-entity
      entity: sensor.fire_risk_today
      image: 'https://www.ipma.pt/bin/icons/svg/rcm/out-Elevado.svg'
      aspect_ratio: 30%
      show_name: false
      show_state: false
  - type: conditional
    conditions:
      - entity: sensor.fire_risk_today
        state: '4'
    card:
      type: picture-entity
      entity: sensor.fire_risk_today
      image: 'https://www.ipma.pt/bin/icons/svg/rcm/out-MuitoElevado.svg'
      aspect_ratio: 30%
      show_name: false
      show_state: false
  - type: conditional
    conditions:
      - entity: sensor.fire_risk_today
        state: '5'
    card:
      type: picture-entity
      entity: sensor.fire_risk_today
      image: 'https://www.ipma.pt/bin/icons/svg/rcm/out-Maximo.svg'
      aspect_ratio: 30%
      show_name: false
      show_state: false
18

Only one downside is that it looks like lovelace is reserving the whole spaces for the five pictures even just one is shown

image
image1035×850 17.6 KB

:frowning_face:

19

Now go out and start a forest fire and see when it updates :laughing:

20

Been sleeping and had a thought.
Perhaps you can host those images locally and have all 5 numbered 1-5.
Then you have a sixth which is say ‘current.svg’.
In an automation when the level is changed you copy the ‘level.svg’ to ‘current.svg’.
That way you can have one picture card with static code and everything happens in the background.

I found this about copying files in HA,have never used it myself.

21

You should bring the discussion to https://forum.cpha.pt in Portuguese :slight_smile:

I think I can make this work as an extra sensor in IPMA integration, first needs extra code in https://github.com/dgomes/pyipma

1 Like