Help request for Web scraper

Hello all,

I am looking to scrape data for my (Household/Recycling)bin collection schedule. The website I’m trying to use can be found here (with a random address)

After a couple of days trying to retrieve data without luck by following the guides for BeautifulSoup, please could you give me a hint how to define the scrape function ‘select:’ to achieve this?

I have tried the method listed to use Firefox:
highlight “data”, and then select Inspect element.
Then select: CSS Selector
and inserted the output into the ‘select:’ of the sensor

any pointers for where I’m going wrong would be greatly appreciated

FWIW, that url give you you a plain, simple, HTML, probably easier to scrape.

Wow, thank you.
I don’t quite know how you found that link, but it fits the bill perfectly.
I will get the integration setup tomorrow :slight_smile:

With the new link, the scraper platform works without issue using

  - platform: scrape
    name: household-waste-collection
    resource: https://www.cornwall.gov.uk/umbraco/Surface/Waste/MyCollectionDays?uprn=010001203494&url=https%3A%2F%2Fwww.cornwall.gov.uk%2Fmy-area%2F%3FPostcode%3DTR3%206AA%26Uprn%3D010001203494&subscribe=False
    select: "#household > span:nth-child(3)"
    value_template: '{{ (strptime(value, "%d %b").replace(year=now().year)) }}'
    scan_interval: 86400

not sure why it couldn’t scrape that data from the full/original link.
Next to try using the multi scraper module to pull data for the 3 Bins at once

As you pointed out, scraping from a JS enabled site (as they all are), is hit-and-miss.

Often, though, those site actually calls APIs from the client JS, and that’s what I checked via the dev tool of the browser.
There wasn’t an json API per se, here, but plain HTML is easy to manage as well :wink:

1 Like

Sorry for jumping on the end of the thread, but looks very similar to what wigan councuil have done.

I’ve been trying to every way to get access to the data for scraping without success, wondered if anyone else had any ideas ?

My Neighbourhood - Search Example postcode = Wa3 3BE
2 school street has a UPRN of UPRN100011743901 so was hoping to getd irectly to it via My Neighbourhood - Search but to no avail!

Do you have any suggestions @koying on the off chance that you can spot some secret api that i cant see ?

Cheers