I am trying to scrape my provider’s account/subscription status page, but failing big time.
I am pretty sure this is because it must be the slowest site in the universe. Although it only provides me with a rather simple page, it takes very long to show completely - easily up to half a minute. It seems to be generated in at least three stages, automatically refreshing after every stage to show some additional info.
I think the problem is that right after getting to the first stage being shown, Multiscrape goes to work but totally fails because the information I’m after is not yet available.
So, the question is : can Multiscrape be told to WAIT for the full page to be shown, by monitoring the availability of some selector or other token ? Even a simple wait-time parameter would help, because the total scraping speed is of no importance to me (one scrape per day is all I need).
Thanks for any pointers or suggestions !