[SOLVED] Need help to scrape data from website

sfbuch · October 29, 2020, 10:32pm

hi,

i would like to scrape data from this website - the table where you can see the name of the lake and the corrsponding temperature:

http://www.wassertemperatur.org/oesterreich/

could you please give me a hint how to define the scrape function to achieve this?

i tried it with a couple of html tags. with td it shows at least the name of the first lake, when trying tbody the entity is no longer there to integrate in lovelace.

nickrout · October 29, 2020, 11:22pm

Does that site have an api?

sender · October 30, 2020, 6:26am

sfbuch · October 30, 2020, 9:13am

no, it doesn’t.

if it had one and they would provide it in json, how could i use the data with a custom integration? scrape would not work for it?

sender · October 30, 2020, 9:25am

Do this.

go to the page and right click on the temp you want as a sensor:

Inspect element.

Then select:
CSS Selector:

Make a sensor in home assistant and put the CSS data in (in confguration.yaml under
Sensor:

  - platform: scrape
    name: Ausee Temperature Test
    resource: http://www.wassertemperatur.org/oesterreich/
    select: ".entry-content > table:nth-child(4) > tbody:nth-child(1) > tr:nth-child(2) > td:nth-child(2) > span:nth-child(1)"

Result:

more info on the link I gave before:

sfbuch · October 30, 2020, 9:53am

thanks, it worked perfectly!

i read the scrape documentation before, but i just could not figure out, how to define “select”.

thanks!

sender · October 30, 2020, 9:57am

Mark the “topic as solution” please

Makis · October 30, 2020, 10:50am

Hi, I am trying to do the same for education but my chrome or edge browser doesn’t have the copy CSS option. Which browser are you using?

sfbuch · October 30, 2020, 11:36am

you have these options in firefox.

sender · October 30, 2020, 12:03pm

Edge also (Inspect)

Makis · October 30, 2020, 2:22pm

I tried edge also but I didnt found the css option there

sfbuch · October 31, 2020, 1:19am

@sender @Makis

when i tried to apply this procedure on other sites, unfortunately i was not able to scrape the data although the fields where properly identified when choosing inspect element.

on

http://laendris.donaustationen.at/index.php?module_id=6
http://laendris.donaustationen.at/index.php?module_id=6&action=details&pegelstelle_id=13&Tage=1&l_laende_id=33

i could not load the values on the overview as well as on the details page.

on

https://www.noel.gv.at/wasserstand/#/de/Messstellen/Details/207373/Wasserstand/3Tage

i also could not load the values. see corresponding screenshots 1-3.

could you imagine was the problem is and could you tell me, when the above procedure works and when not? i tried this now on a couple sites, it works perfectly on the one mentioned above but does not others.

thanks in advance!

aerodolphin · August 10, 2021, 9:12am

Hi all, I would like to take some information of the next website:

http://www.climaat.angra.uac.pt/boias/index2.htm

For example, how I can get the temperature shown in the last row of the table?
“Temperatura da água à superficie: 23 ºC” (Water surface temperature)

kongjudas · April 14, 2022, 6:00pm

Hi folks. Can someone please help me get the KP-index from this site? I’ve tried and tried, but I can’t get my head around the formula. If it’s at all possible, that is. I would really appreciate it, thank you!
https://www.spaceweatherlive.com/en/auroral-activity.html

nickrout · April 14, 2022, 9:46pm

If you read the site faq you are advised not to do thst.

SmartLiving.Rocks · August 23, 2022, 8:12am

Really cool - thanks for your detailed explanation! I am now tracking the water level of my river nearby

I am also interested in getting the current gas level for Europe from this site: https://agsi.gie.eu/
I have copied the sector path as you described, but the sensor in Home Assistant stays unknown

div.tabulator-row:nth-child(1) > div:nth-child(3)

Maybe someone could help me to get the data into Home Assistant - that would be great!

alekseyn · September 11, 2022, 6:49am

This scrape method will not work. All of the data on that webpage is generated by javascript calling API. Check for page source to see if the data is actually there. So conventional ‘scrape’ will not work here. I did look at that page and found an API call. BUT that call without headers was not working well. Seems like API needed exact headers to work. With headers it was returning a JSON output that we still need to parse so this works for me:

sensor:
  #Aggregated Gas Storage Inventory in the EU
  - platform: rest
    name: AGSI EU Gas Storage Full
    value_template: "{{ value_json['data'][0]['full'] }}"
    unit_of_measurement: "%"
    scan_interval: 3600
    resource: https://agsi.gie.eu/api?date=today
    headers:
      Content-Type: application/json
      User-Agent: 'Mozilla/5.0 (iPad; CPU OS 15_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/104.0.5112.99 Mobile/15E148 Safari/604.1'
      Upgrade-Insecure-Requests: 1
      Referer: 'https://agsi.gie.eu/'

Also a command line sensor can be built from this python script

import requests; import json; 
headers = { 
    'Upgrade-Insecure-Requests': '1', 
    'User-Agent': 'Mozilla/5.0 (iPad; CPU OS 15_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/104.0.5112.99 Mobile/15E148 Safari/604.1',
   'referer': 'https://agsi.gie.eu/'
};

resp = requests.get('https://agsi.gie.eu/api?date=today', headers=headers);
data = json.loads(resp.content);
print(data['data'][0]['full']);

SmartLiving.Rocks · September 11, 2022, 9:22am

Thanks for your help. Someone on facebook helped me with the rest sensor.

sensor rest:
  - platform: rest
    scan_interval: 3600
    name: Gas Speicher De
    resource: https://agsi.gie.eu/api?country=DE
    headers:
      content-type: "application/json"
      x-key: !secret gas_token #Hier den API Key eintragen! 
    json_attributes_path: "$.data[0].['.']"
    json_attributes:
      - name
      - code
      - url
      - gasDayStart
      - gasInStorage
      - consumption
      - consumptionFull
      - injection
      - withdrawl
      - workingGasVolume
      - injectionCapacity
      - status
      - trend
      - full
      - info
    value_template: >-
      {{ value_json.message }}

SmartLiving.Rocks · September 11, 2022, 9:24am

Here is the yaml for the template sensor.

template agsi:
  - sensor:
      - name: Füllstand Deutschland Total
        icon: mdi:gas-burner
        unit_of_measurement: "%"
        state: "{{ state_attr('sensor.gas_speicher_de', 'full') }}"
        device_class: "gas"
      - name: Gas im Speicher Deutschland
        icon: mdi:storage-tank
        unit_of_measurement: "TWh"
        state: "{{ state_attr('sensor.gas_speicher_de', 'gasInStorage') }}"
        #device_class: "gas"
      - name: Trend Gas Speicher Deutschland
        icon: mdi:storage-tank
        unit_of_measurement: "%"
        state: "{{ state_attr('sensor.gas_speicher_de', 'trend') }}"
        #device_class: "gas"
      - name: Gasverbrauch Deutschland
        icon: mdi:gas-burner
        unit_of_measurement: "TWh"
        state: "{{ state_attr('sensor.gas_speicher_de', 'consumption') }}"
      - name: Gasverbrauch Total Deutschland
        icon: mdi:gas-burner
        unit_of_measurement: "%"
        state: "{{ state_attr('sensor.gas_speicher_de', 'consumptionFull') }}"

alekseyn · September 12, 2022, 5:35pm

Awesome! Thanks for sharing.

Just in case someone will run into this thread: my solution does not require API registration. It mimics a website visit and therefore no need for an API key.