How can I scrape data from website?

Configuration
1

Hi. I’m trying to make a scrape sensor for a value from a website.

The address of website is Stake | PartyParrot

What I want to get is the value in red box.

e
e1103×434 19.8 KB

Using developer tool, I found that CSS selector address is like belows.

#__next > div.bg-scaffold.min-h-screen.flex.flex-col.justify-end.sm:justify-between > main > section:nth-child(1) > div > div.border-secondary.border.rounded-lg.p-3.sm:p-4 > div > div:nth-child(5) > span

But using that one in “select” in configuration was unsuccessful.

Is there anyone to help me with scraping? Thanks in advance.

2

It’s also worth checking developer tools in the browser first to see if there is a better way to get the data. And there is.

The Webpage makes a request to https://parrot.fi/api/v1/stakepool/parrot which returns JSON data so now you can simply use a rest sensor.

The value_template will be:

value_json[0]['amountFloat']|float(0)

(The reason the scrape doesn’t work - is because the data isn’t part of the page and is loaded dynamically after the page has finished loading)

So the code will be something like this:

  - platform: rest
    resource: https://parrot.fi/api/v1/stakepool/parrot
    name: Parrot Stakepool
    value_template: "{{ value_json[0]['amountFloat']|float(0) }}"

Hope that helps.

4

Thanks for the answer! Actually, I didn’t know there is API for this website, and tried to use scrape as an alternative. May I ask how did you find the address of API or find the existence of API for this?

5

I’m using Chrome but Firefox has similar tools.

Open the webpage. Right click on the webpage and choose “inspect”. In the window that opens switch to the network tab, and then reload the tab with the page in that you want to monitor. In the network tab - click on “Fetch/XHR” to limit the results of the network requests to only ones that are made by the webpage after it loaded.

In this case the page makes regular requests to 2 endpoints. sentry_key - which I assume is some sort of analytics, and an API.

6

Found it in Chrome. Thank you so much for your kind answer!

1 Like