If you send me the url - I’ll do you a python beautiful soup version
One scrape to produce a json file then use the rest sensor to hit your local file.
Similar to : Bin / Waste Collection
Are you practicing your python? I totally need to get into that a lot more.
Here’s a sample house: https://mychelmsford.secure.force.com/WasteServices/WM_WasteViewProperty?id=a02240000054H2rAAE
Current command_line for parseBinCollection.sh below, if any of it is recyclable:
cat /config/scripts/binCollection.json | jq ‘…|.“text”? | select (. !=null)’ | tr -d ‘"’ | grep Bin -A3
Thanks
How about a physical indicator, take a look at https://hackaday.com/2020/01/23/bindaycator-lets-you-know-when-to-take-out-the-trash/
2 Likes
#!/usr/bin/env python3
#This script pulls (in one hit) the data from Chelmsford Council Bins Data
#import the wonderful Beautiful Soup and the URL grabber
from urllib.request import Request, urlopen
import json
from bs4 import BeautifulSoup
#Set a user agent so we look like a browser ;-)
user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)'
headers = {'User-Agent': user_agent}
#Make the Request
req = Request('https://mychelmsford.secure.force.com/WasteServices/WM_WasteViewProperty?id=a02240000054H2rAAE')
req.add_header('User-Agent', user_agent)
fp = urlopen(req).read()
#decode the page
page = fp.decode("utf8")
#Make a BS4 object
soup = BeautifulSoup(page, features="html.parser")
soup.prettify()
#Form a JSON wrapper
data = {"bins":[]}
#Search for the specific table using BS4
rows = soup.find("tbody", {"id" : lambda L: L and L.startswith('Registration:')}).find_all("tr")
#Loops the Rows
for row in rows:
#set the vars per bin and date for each row
cells = row.find_all("td")
binType = cells[1].get_text()
lcDate = cells[2].get_text()
ncDate = cells[3].get_text()
fcDate = cells[4].get_text()
#Make each Bin element in the JSON
dict_data = {
"BinType": binType,
"Last Collection Date": lcDate,
"Next Collection Date": ncDate,
"Following Collection Date": fcDate,
}
#Add data to the main JSON Wrapper
data["bins"].append(dict_data)
#Make the JSON
json_data = json.dumps(data,sort_keys=True, indent=4)
#Output the data - run this script with a redirect to send the output to a file
#Suggest Crontab
print(json_data)
Outputs
{
"bins": [
{
"BinType": "Black Bin",
"Following Collection Date": "Friday 14/02/2020",
"Last Collection Date": "Friday 17/01/2020",
"Next Collection Date": "Friday 31/01/2020"
},
{
"BinType": "Brown Bin",
"Following Collection Date": "Friday 07/02/2020",
"Last Collection Date": "Saturday 11/01/2020",
"Next Collection Date": "Friday 24/01/2020"
},
{
"BinType": "Card Sack",
"Following Collection Date": "Friday 14/02/2020",
"Last Collection Date": "Friday 17/01/2020",
"Next Collection Date": "Friday 31/01/2020"
},
{
"BinType": "Food Waste External Bin",
"Following Collection Date": "Friday 31/01/2020",
"Last Collection Date": "Friday 17/01/2020",
"Next Collection Date": "Friday 24/01/2020"
},
{
"BinType": "Green Box",
"Following Collection Date": "Friday 14/02/2020",
"Last Collection Date": "Friday 17/01/2020",
"Next Collection Date": "Friday 31/01/2020"
},
{
"BinType": "Paper Sack",
"Following Collection Date": "Friday 07/02/2020",
"Last Collection Date": "Saturday 11/01/2020",
"Next Collection Date": "Friday 24/01/2020"
},
{
"BinType": "Plastic Sacks",
"Following Collection Date": "Friday 07/02/2020",
"Last Collection Date": "Saturday 11/01/2020",
"Next Collection Date": "Friday 24/01/2020"
}
]
}
Created a Repo for this on GitHub https://github.com/robbrad/UKBinCollectionData
Raise an issue to get your council added - or do it yourself with a pull request.
2 Likes
i wrote a other verson
that read a CSV file via node-red
1 Like
Wow. Thanks heaps @Robbrad - my council will be happy I’m not DDoSing their website 
Now to do a HA template for countdown days
1 Like
Thanks to @Robbrad I have got my data from here https://www.peterborough.gov.uk/api/jobs/2020-1-01/2020-12-31/100090181402
{"recordCount":2,"jobs_FeatureScheduleDates":[{"jobID":9573337,"jobName":"Empty Bin 240L Black","jobDescription":"Empty Bin 240L Black","jobStatus":"Not Started","previousDate":"2020-01-14T09:23:27.173","nextDate":"2020-01-28T07:00:00","rescheduledDate":"0001-01-01T00:00:00","rescheduledDateSpecified":false,"rescheduledReason":"","uprn":100090181402,"usrn":30101686,"address1":"","address2":"12","street":"CANTERBURY ROAD","locality":"WERRINGTON","town":"PETERBOROUGH","postCode":"PE4 6PE","easting":516645,"northing":303330,"longitude":-0.278563,"latitude":52.615194},{"jobID":9584045,"jobName":"Empty Bin 240L Green","jobDescription":"Empty Bin 240L Green","jobStatus":"Not Started","previousDate":"2020-01-21T11:53:24.833","nextDate":"2020-02-04T07:00:00","rescheduledDate":"0001-01-01T00:00:00","rescheduledDateSpecified":false,"rescheduledReason":"","uprn":100090181402,"usrn":30101686,"address1":"","address2":"12","street":"CANTERBURY ROAD","locality":"WERRINGTON","town":"PETERBOROUGH","postCode":"PE4 6PE","easting":516645,"northing":303330,"longitude":-0.278563,"latitude":52.615194}]}
I would like the ‘jobName’ and ‘nextDate’ collected - one for the Black bin (jobID: 9573337) and one for the Green bin (jobID: 9584045).
jobs_FeatureScheduleDates:
- jobID: 9573337
jobName: Empty Bin 240L Black
jobDescription: Empty Bin 240L Black
jobStatus: Not Started
previousDate: '2020-01-28T09:52:17.837'
nextDate: '2020-02-11T07:00:00'
rescheduledDate: '0001-01-01T00:00:00'
rescheduledDateSpecified: false
rescheduledReason: ''
uprn: 100090181402
usrn: 30101686
address1: ''
address2: '12'
street: CANTERBURY ROAD
locality: WERRINGTON
town: PETERBOROUGH
postCode: PE4 6PE
easting: xx
northing: xx
longitude: xx
latitude: xx
- jobID: 9584045
jobName: Empty Bin 240L Green
jobDescription: Empty Bin 240L Green
jobStatus: Not Started
previousDate: '2020-01-21T11:53:24.833'
nextDate: '2020-02-04T07:00:00'
rescheduledDate: '0001-01-01T00:00:00'
rescheduledDateSpecified: false
rescheduledReason: ''
uprn: 100090181402
usrn: 30101686
address1: ''
address2: '12'
street: CANTERBURY ROAD
locality: WERRINGTON
town: PETERBOROUGH
postCode: xx
easting: xx
northing: xx
longitude: xx
latitude: xx
friendly_name: bindates
The template sensor does not work. Can anyone help with what am I doing wrong? Is it something to do with the fact everything is a sub attribute of jobs_FeatureScheduleDates?
- platform: rest
name: bindates
json_attributes:
- jobs_FeatureScheduleDates
resource: https://www.peterborough.gov.uk/api/jobs/2020-1-01/2020-12-31/10008050889
value_template: '{{ value_json.bindates }}'
- platform: template
sensors:
bin:
value_template: '{{ states.sensor.bindates.attributes["jobs_FeatureScheduleDates"]["jobName"] }}'
nextcollection:
value_template: '{{ states.sensor.bindates.attributes["jobs_FeatureScheduleDates"]["nextDate"] }}'
device_class: timestamp
- platform: unifiprotect
yeah it worked well, all i had to also do was have HA update the sensor value, im sure there is a better way but all i did was added it to one of my daily automations
Nice idea, thanks. My council have a rather tricky way of getting data, they send a token which needs to be added to a 2nd Curl command, I’ll see if I can add an easy way of doing it.
Hi All, different post but on similar subject. I have an automation to advise me at 1800hrs on garbage night to put the bin out. Its a TTS but it continually fails. I tried another test automation with tts at a time trigger and it worked. Can anyone please advise why this might be not working.
- id: ‘1xxxxxxxxxx’
alias: Garbage Reminder
description: TTS warning for putting Garbage Out
trigger:- at: ‘18:00:00’
platform: time
condition: - condition: state
entity_id: sensor.garbage
state: ‘0’
action: - data:
entity_id: media_player.all_speakers
message: Hi. Dont forget to take the bin out.
service: tts.google_say
- at: ‘18:00:00’
I tried this for my local council but I’m struggling to figure out the URL to scrape.
https://www.sefton.gov.uk/bins-recycling/bin-collections/collection-days.aspx
Any ideas? Thanks.
Lichfield (UK) is a beautiful place to live, with one huge caveat that sends shudders through the thousands of households: the council’s bin collection URL isn’t particularly machine-readable.
EDIT: prior config removed as it no longer applies, as the council restructured the web calendar. If you were looking for this, the HACS Waste Collection Schedule now supports our leafy city.
1 Like
Bravo! We will beat this scourge !
Has anyyone done anything with Cardiff Council - would love to achieve this but API’s and JSON scraping is over my head unfortunatley.
Anyone fancy trying to deocde it please? https://www.cardiff.gov.uk/ENG/resident/Rubbish-and-recycling/When-are-my-bins-collected/Pages/default.aspx
You could use CF5 3SX as a test postcode
That’s going to be tricky, I’m afraid. The page is even less scrapable than Lichfield’s. There is an API request going on behind the scenes: it’s a POST request to https://api.cardiff.gov.uk/WasteManagement/api/WasteCollection with a payload like {"systemReference":"web","language":"eng","uprn":100100085369} (uprn is unique property reference number at a guess: that one is from the postcode you suggested).
That call returns something useful like this (including typo right at the start!):
{
"lanugage": "eng",
"collectionWeeks": [
{
"week": 1,
"date": "2020-06-29T00:00:00",
"day": "Monday",
"bins": [
{
"binId": "25 Food Caddy",
"imageUrl": "https://www.cardiff.gov.uk/ENG/resident/Rubbish-and-recycling/When-are-my-bins-collected/PublishingImages/20.png",
"type": "Food",
"collectionType": "Standard"
},
{
"binId": "Recycling Bag",
"imageUrl": "https://www.cardiff.gov.uk/ENG/resident/Rubbish-and-recycling/When-are-my-bins-collected/PublishingImages/19.png",
"type": "Recycling",
"collectionType": "Standard"
}
]
},
{
"week": 2,
"date": "2020-07-06T00:00:00",
"day": "Monday",
"bins": [
{
"binId": "140 Residual Bin",
"imageUrl": "https://www.cardiff.gov.uk/ENG/resident/Rubbish-and-recycling/When-are-my-bins-collected/PublishingImages/3.png",
"type": "General",
"collectionType": "Standard"
},
{
"binId": "25 Food Caddy",
"imageUrl": "https://www.cardiff.gov.uk/ENG/resident/Rubbish-and-recycling/When-are-my-bins-collected/PublishingImages/20.png",
"type": "Food",
"collectionType": "Standard"
},
{
"binId": "Recycling Bag",
"imageUrl": "https://www.cardiff.gov.uk/ENG/resident/Rubbish-and-recycling/When-are-my-bins-collected/PublishingImages/19.png",
"type": "Recycling",
"collectionType": "Standard"
}
]
},
{
"week": 3,
"date": "2020-07-13T00:00:00",
"day": "Monday",
"bins": [
{
"binId": "240 Green Bin",
"imageUrl": "https://www.cardiff.gov.uk/ENG/resident/Rubbish-and-recycling/When-are-my-bins-collected/PublishingImages/5.png",
"type": "Garden",
"collectionType": "Standard"
},
{
"binId": "25 Food Caddy",
"imageUrl": "https://www.cardiff.gov.uk/ENG/resident/Rubbish-and-recycling/When-are-my-bins-collected/PublishingImages/20.png",
"type": "Food",
"collectionType": "Standard"
},
{
"binId": "Recycling Bag",
"imageUrl": "https://www.cardiff.gov.uk/ENG/resident/Rubbish-and-recycling/When-are-my-bins-collected/PublishingImages/19.png",
"type": "Recycling",
"collectionType": "Standard"
}
]
},
{
"week": 4,
"date": "2020-07-20T00:00:00",
"day": "Monday",
"bins": [
{
"binId": "140 Residual Bin",
"imageUrl": "https://www.cardiff.gov.uk/ENG/resident/Rubbish-and-recycling/When-are-my-bins-collected/PublishingImages/3.png",
"type": "General",
"collectionType": "Standard"
},
{
"binId": "25 Food Caddy",
"imageUrl": "https://www.cardiff.gov.uk/ENG/resident/Rubbish-and-recycling/When-are-my-bins-collected/PublishingImages/20.png",
"type": "Food",
"collectionType": "Standard"
},
{
"binId": "Recycling Bag",
"imageUrl": "https://www.cardiff.gov.uk/ENG/resident/Rubbish-and-recycling/When-are-my-bins-collected/PublishingImages/19.png",
"type": "Recycling",
"collectionType": "Standard"
}
]
}
],
"hasGlassBin": false
}
but the API is set up with security to prevent calls other than from that webpage (which generates an Authorization: Bearer cookie as a “password”), which is a real shame. Perhaps it might be worth trying to contact someone techie in the council to get this opened up? Failing that, perhaps move to a different area 
1 Like
Thanks for taking a look. Trust my council to be the arkward one - but hey at least I know nobody will steal my bin collection days - classified information
You know I struggled with getting my bin collection from my council’s api. Then I gave up and went with this custom component. Unless you have some really weird random collection cycle, it copes really well. https://github.com/bruxy70/Garbage-Collection
Hi Everybody.
Previously I have managed to cobble things together and get a working output. I have since moved and am now battling with a new Council website. There doesn’t appear to be an easily accessible UTRN or other identifier… I can see. looked at URL, but then you have to put postcode into page, it then does an HTTP post & download, then you enter your address from the list and it then downloads the correct data. I cant see a way to do this as before. Any assistance would be greatly appreciated.
Test Postcode : MK43 9JT
Address: pick any from the drop down list.
then how to get that specific data?
Thanks in Advance.
If you use the web-console (F12 in chrome / edge-chromium) and go to the network view, then follow through to where you want to go and you’ll be able to pick out the URL to your address, i.e.
Is for number 10 in the postcode area you gave.
2 Likes