Delay scrape to wait for page to load

Configuration
1

While a page is loading a certain thing I want to scrape is given as n/a. And that’s the only thing scrape is grabbing before leaving of course.

Is it somehow possible to delay the scraping to wait for a page to complete its inline loading?

2

what gets loaded? If the page loads then a javascript (or other script) loads data, you won’t be able to get it I’m afraid. It’s one of the component’s limitations…

3

I am not very familiar with it, but I think AJAX? Takes about 5 seconds. But as expected scrape is not waiting for ‘n/a’ to change to a number.

4

check this page to see if the method applies to what you’re trying to do:
http://toddhayton.com/2015/03/11/scraping-ajax-pages-with-python/

5

Thanks, that might be something for the future.

On a more general note, some weeks ago I tested scrape and it seemed to be doing well on a static page. Just now tried various more pages and the result is just nothing. What an absolutely horrible component it is.

6

it takes a bit of learning to get the data out, but once you get the idea it’s pretty cool.
Also depending on what you want, you may be better off installing beautifulsoup and running your own script then send the data to HA via MQTT (that’s what I did)…

1 Like
7

I have another example that needs a delay before scraping!

I am trying to scrape Energy Production from EMA website (apsystems):
https://www.apsystemsema.com/ema/intoDemoUser.action?id=0b28481b73ec0fd90173f4ee8ab51ffd&locale=pt_BR

The code works fine:

  - platform: scrape
    resource: https://www.apsystemsema.com/ema/intoDemoUser.action?id=0b28481b73ec0fd90173f4ee8ab51ffd&locale=pt_BR
    name: Energia Hoje
    select: "#today"
    unit_of_measurement: "kWh"
    scan_interval: 10

But it gives me “0 kWh” because it takes 500ms to load the value!!!

Did you get success @Saturnus?

8

No support for delay with the build in stuff.

9

Any update on delay function?

2 Likes