Scrape the fire risk 🔥

exxamalte · June 14, 2020, 1:41pm

I just looked at the HTML code of the page itself and it appears that the data is actually stored in a lot of JavaScript code. That screenshot of the HTML structure is just what’s rendered in the web browser when you select a location.
The scrape sensor basically just downloads the HTML code and then tries to find values based on your select statements, but it doesn’t execute JavaScript to render the HTML.

Long story short: I don’t think you can use the scrape sensor on that page. And I don’t think there is any other standard integration that would help you.

If you are up to some Python development, you could have a look at a custom sensor I built a while ago that does find things on a HTML page using a regular expression, and adopt that for your needs.

Hellis81 · June 14, 2020, 1:55pm

Maybe you can find an API for the service?
It’s usually easier to find when you know the language.

But it seems the mobile page is also JavaScript driven (at least it took forever to load so I assume…)

@exxamalte will that actually make any difference with regex?
Usually it does not, you need a headless browser to get the data after JavaScript.

exxamalte · June 14, 2020, 2:14pm

All the JavaScript payload is embedded in that HTML page, so regex could be used to find the relevant data and extract it.

Absolutely agree, that would be the preference. Another option to look for is some standard format such as XML/RSS or JSON - much easier to extract data from such formats.

FireScraper · June 14, 2020, 2:45pm

Good point. I actually already contacted IPMA last year and asked for a API for this data but (at this point) they didn’t have it for this particular information. The IPMA Weather Integration for Home Assistant from @dgomes makes use of this API.

Now I looked again and found this here:

Invocação :
https://api.ipma.pt/open-data/forecast/meteorology/rcm/rcm-d{idDay}.json
Nota: Apenas estão disponíveis dados diários. {idDay} varia entre 0 e 1, sendo que:

0 - é dia equivalente ao hoje

1 - amanhã

Códigos de rcm:

1 - Risco reduzido

2 - Risco moderado

3 - Risco Elevado

4 - Risco muito elevado

5 - Risco máximo

Resultado (formato json): { “dataPrev”: “2020-02-03”, “dataRun”: “2020-02-02”, “fileDate”: “2020-02-03 09:05:01”, “local”: { “{DICO}”: { “data”: { “rcm”: 1 }, “DICO”: “1001”, “latitude”: 39.55, “longitude”: -8.97 },…

dataPrev: data da previsão

dataRun: data da corrida do modelo

fileDate: data de atualização do ficheiro (taxa de atualização horária)

DICO: Identificador único de concelho (de acordo com a CAOP - DGT)

rcm: código relativo ao tipo de risco de incêndio

latitude: latitude

longitude: longitude

Still could need some little help how I could “dump” this into home assistant the best way

NK553 · June 14, 2020, 5:00pm

Look here:

Here’s an example from somewhere else:

  - platform: rest
    resource: https://api.ambientweather.net/v1/devices?applicationKey=KEY_HERE&apiKey=OTHER_KEY_HERE
    name: weather_station_report
    json_attributes:
    scan_interval: 300
    value_template: '{{ value_json[0].info.location }}'

FireScraper · June 14, 2020, 5:25pm

thank’s @NK553 very valuable!

I just went through all json examples but couldn’t find one fitting my use case.

If I’m looking at the json it represents all locations in whole portugal. They are unique by the identifier called “DICO”. I some how need to limit the return of the “rcm” value based by the definided “dico”.

{"dataPrev": "2020-06-14", "dataRun": "2020-06-13", "fileDate": "2020-06-14 09:05:01", "local": {"0101": {"data": {"rcm": 1}, "dico": "0101", "latitude": 40.58, "longitude": -8.44}, "0102": {"data": {"rcm": 1}, "dico": "0102", "latitude": 40.6909, "longitude": -8.4784}, "0103": {"data": {"rcm": 1}, "dico": "0103", "latitude": 40.44, "longitude": -8.44},

For someone who knows json/rest/etc. it’s probably a no-brainer - for me it’s completely new

Hellis81 · June 14, 2020, 5:30pm

I could do it with my hands tied behind my back in PHP.
If you have any way of hosting a PHP file then I can do it.
But I’m not yet good with yaml and all that

FireScraper · June 14, 2020, 5:57pm

This for examples represents all the data for the location “dico” 0102 in the json file for today.

I’m not quite sure how this is cascaded but can it be that it needs to be queried somewhat like this:

  - platform: rest
    resource: http://api.ipma.pt/open-data/forecast/meteorology/rcm/rcm-d0.json
    name: fire_risk_today
    json_attributes:
    scan_interval: 24h
    value_template: '{{ value_json[0102].rcm }}'

?

Hellis81 · June 14, 2020, 7:11pm

Ok had some more time to work with it now that the kids are in bed.


sensor:
  - platform: rest
    resource: http://api.ipma.pt/open-data/forecast/meteorology/rcm/rcm-d0.json
    name: fire_risk_today
    json_attributes:
    scan_interval: 86400 # 24 hours in seconds
    value_template: '{{ value_json["local"]["0102"]["data"].rcm }}'

FireScraper · June 14, 2020, 7:35pm

Wow! Great @Hellis81 ! Thank you so much!

From here it looks like the update for the json file is daily at 09:05. Is their a way to force the update shortly after - let’s say daily at 09:10 for example?

Hellis81 · June 14, 2020, 7:47pm

I don’t know.
There is a service call called homeassistant.update_entity but when I use it on the entity it doesn’t do anything.
Perhaps because my force update is default (false) or because it does not work. But that is one thing you could try.
Change the force update to true on the sensor and create an automation to do this service call when time is 09:10

FireScraper · June 14, 2020, 9:20pm

I got this now but it’s just the static picture. I tried with picture entity but it only allows two states. The conditonal card also can only show true and false.

Is there any lovelace card out they which can show picture 1 on value 1 - picture 2 on value 2 - picture 3 on value 3 and so on?

Hellis81 · June 14, 2020, 9:46pm

I believe that is a conditional card. But I have never used them.

FireScraper · June 14, 2020, 10:36pm

You are right @Hellis81

I build this somewhat ugly card but it works (I hope )

type: vertical-stack
title: Fire Risk Today
cards:
  - type: conditional
    conditions:
      - entity: sensor.fire_risk_today
        state: '1'
    card:
      type: picture-entity
      entity: sensor.fire_risk_today
      image: 'https://www.ipma.pt/bin/icons/svg/rcm/out-Reduzido.svg'
      aspect_ratio: 30%
      show_name: false
      show_state: false
  - type: conditional
    conditions:
      - entity: sensor.fire_risk_today
        state: '2'
    card:
      type: picture-entity
      entity: sensor.fire_risk_today
      image: 'https://www.ipma.pt/bin/icons/svg/rcm/out-Moderado.svg'
      aspect_ratio: 30%
      show_name: false
      show_state: false
  - type: conditional
    conditions:
      - entity: sensor.fire_risk_today
        state: '3'
    card:
      type: picture-entity
      entity: sensor.fire_risk_today
      image: 'https://www.ipma.pt/bin/icons/svg/rcm/out-Elevado.svg'
      aspect_ratio: 30%
      show_name: false
      show_state: false
  - type: conditional
    conditions:
      - entity: sensor.fire_risk_today
        state: '4'
    card:
      type: picture-entity
      entity: sensor.fire_risk_today
      image: 'https://www.ipma.pt/bin/icons/svg/rcm/out-MuitoElevado.svg'
      aspect_ratio: 30%
      show_name: false
      show_state: false
  - type: conditional
    conditions:
      - entity: sensor.fire_risk_today
        state: '5'
    card:
      type: picture-entity
      entity: sensor.fire_risk_today
      image: 'https://www.ipma.pt/bin/icons/svg/rcm/out-Maximo.svg'
      aspect_ratio: 30%
      show_name: false
      show_state: false

FireScraper · June 14, 2020, 10:43pm

Only one downside is that it looks like lovelace is reserving the whole spaces for the five pictures even just one is shown

Hellis81 · June 14, 2020, 10:44pm

Now go out and start a forest fire and see when it updates

Hellis81 · June 15, 2020, 4:59am

Been sleeping and had a thought.
Perhaps you can host those images locally and have all 5 numbered 1-5.
Then you have a sixth which is say ‘current.svg’.
In an automation when the level is changed you copy the ‘level.svg’ to ‘current.svg’.
That way you can have one picture card with static code and everything happens in the background.

I found this about copying files in HA,have never used it myself.

dgomes · June 15, 2020, 12:51pm

You should bring the discussion to https://forum.cpha.pt in Portuguese

I think I can make this work as an extra sensor in IPMA integration, first needs extra code in https://github.com/dgomes/pyipma

FireScraper · June 15, 2020, 3:41pm

Fixed it! With the custom state-switch card

dgomes · January 7, 2024, 2:32pm

For future reference, the IPMA integration already provides this information