Bin / Waste Collection

JAN 2020 Update: Created a Repo for this on GitHub https://github.com/robbrad/UKBinCollectionData Raise an issue to get your council added - or do it yourself with a pull request.


It’s quite annoying not knowing which bin colour you need to take out and our council doesn’t distribute the fridge magnet calendars anymore. There are three bins, General waste, Recycling and Garden (Black/Gray and Green)

More often than not we are the last to take our bins out to the street because we want to see what colour people have taken out - A flawed system, because the day the first person takes their bin out gets it wrong, will be the day everyone gets it wrong and the end of the world. :crazy_face:

Home Assistant to the rescue. WARNING the following is not pretty.

The council in the UK (Cheshire East) are kind enough to provide a calendar, but no api or fancy way to get the raw data. To make things worse one week it’s one bin and the next week its two bins!

Website to parse: http://online.cheshireeast.gov.uk/MyCollectionDay/

I thought could I capture the table from this page and somehow parse it. I had to work out the post URL from the result page, find the class of the table and export the data.

Software required: curl, pup and jq

Using my Ubuntu Home Assistant Server I created a script to curl the url out to a JSON - I run a cron job each Thursday night after collection to refresh the JSON data

#!/bin/sh
curl -s "http://online.cheshireeast.gov.uk/MyCollectionDay/?##################" | pup '.wasteType td json{}' > binCollection.json

Perfect - I now had the following result

[
 {
  "children": [
   {
    "alt": "bin image",
    "src": "Images/bin_grey_32.gif",
    "tag": "img"
   }
  ],
  "class": "imgCell",
  "tag": "td"
 },
 {
  "tag": "td",
  "text": "Thursday"
 },
 {
  "tag": "td",
  "text": "31-May-2018"
 },
 {
  "tag": "td",
  "text": "Recycling wheeled bin"
 },
 {
  "tag": "td"
 },
 {
  "children": [
   {
    "alt": "bin image",
    "src": "Images/bin_brown_32.gif",
    "tag": "img"
   }
  ],
  "class": "imgCell",
  "tag": "td"
 },
 {
  "tag": "td",
  "text": "Thursday"
 },
 {
  "tag": "td",
  "text": "31-May-2018"
 },
 {
  "tag": "td",
  "text": "Garden Waste wheeled bin"
 },
 {
  "tag": "td"
 },
 {
  "children": [
   {
    "alt": "bin image",
    "src": "Images/bin_black_32.gif",
    "tag": "img"
   }
  ],
  "class": "imgCell",
  "tag": "td"
 },
 {
  "tag": "td",
  "text": "Thursday"
 },
 {
  "tag": "td",
  "text": "07-Jun-2018"
 },
 {
  "tag": "td",
  "text": "Other Waste wheeled bin"
 },
 {
  "tag": "td"
 },
 {
  "children": [
   {
    "alt": "bin image",
    "src": "Images/bin_grey_32.gif",
    "tag": "img"
   }
  ],
  "class": "imgCell",
  "tag": "td"
 },
 {
  "tag": "td",
  "text": "Thursday"
 },
 {
  "tag": "td",
  "text": "14-Jun-2018"
 },
 {
  "tag": "td",
  "text": "Recycling wheeled bin"
 },
 {
  "tag": "td"
 },
 {
  "children": [
   {
    "alt": "bin image",
    "src": "Images/bin_brown_32.gif",
    "tag": "img"
   }
  ],
  "class": "imgCell",
  "tag": "td"
 },
 {
  "tag": "td",
  "text": "Thursday"
 },
 {
  "tag": "td",
  "text": "14-Jun-2018"
 },
 {
  "tag": "td",
  "text": "Garden Waste wheeled bin"
 },
 {
  "tag": "td"
 },
 {
  "children": [
   {
    "alt": "bin image",
    "src": "Images/bin_black_32.gif",
    "tag": "img"
   }
  ],
  "class": "imgCell",
  "tag": "td"
 },
 {
  "tag": "td",
  "text": "Thursday"
 },
 {
  "tag": "td",
  "text": "21-Jun-2018"
 },
 {
  "tag": "td",
  "text": "Other Waste wheeled bin"
 },
 {
  "tag": "td"
 }
]

I created another script to parse that data which my sensor in Home Assistant could run

#!/bin/sh
cat /home/user/scripts/binCollection.json |  jq '.[].text'| grep ` date -d 'this Thursday' '+%d-%b-%Y'` -A1| grep bin | tr -d '"' | head -n 1

Now I have my parse data I have the following in my sensors.yaml

  - platform: command_line
    command: '/home/user/scripts/parseBinCollection.sh'
    name: BinCollectionType

To make this look nicer I colour my sensor icon based on the bin to be collected in my customize_sensors.yaml - kudos to @Mikeinnc

sensor.BinCollectionType:
  friendly_name: Bin Collection Type
  icon: mdi:delete
  templates:
    icon_color: >
      if (state === 'Other Waste wheeled bin') return 'rgb(0,0,0)';
      if (state === 'Recycling wheeled bin') return 'rgb(128,128,128)';
      if (state === 'Garden Waste wheeled bin') return 'rgb(139,69,19)';
      return 'rgb(68,115,158)';
    hidden: if (state === '') return true;;

Result!
2

The second bin type is in the second sensor (couldn’t think of a better way to represent this - for the day when there are two bins - might be nice to hide this sensor in some why when the result is unknown) - UPDATE using hidden in the template when the state is nothing worked - there was an issue in my script, but now the sensor hides when there is no value - e.g. every other week

I’m quite happy with this result and it saves me having to check the website or risk the wrong bin being taken out - it’s a total hack and will probably fail, eventually.

21 Likes

Brill. This is already on my to do list. Need to see if my council publish a useable schedule.

1 Like

Well done! Very nice job. :+1:

Did something similar with my city. This is a great example of how generic and custom components provide value: all of our cities/towns/etc. publish this info in different ways, so rather than filling the HASS core with one-off integrations, let us do the work ourselves!

1 Like

And it’s much more fun this way :stuck_out_tongue_winking_eye:

Great work. Will try with my local council. Thought about doing this from a calendar but the bank holidays make it difficult with random collection dates around holidays. Cheers!

Awesome work, I would love to try and do something like this, but I think my programming skill’s aren’t up to scratch at all for it. Shame you aren’t a bit further east then we would have shared the same council :smiley:

Does your council publish this info in a calendar of some sort? If you post here, someone may be able to help you out. :smile:

1 Like

They do, kinda, https://www.highpeak.gov.uk/hpbc-cluster its looks the same as @Robbrad council site, where you have to enter a postcode(zip code) and then it gives you the information. This is where I would get stuck, as would not have a clue how to even work out what the link is to parse the correct information for my address.

Do you have a sample address? Msg me yours if you like

It won’t be elegant but Ill put something together for you :+1:

1 Like

Thank you very much sir, that would be amazing. Nothing in my home assistant is pretty at the moment :smiley: so it will fit right in :smiley: I’ll message you now :+1:

I started taking a look – don’t know if it can be done as easily as yours, but here’s what I’ve gleaned.

Query 1: Get Location

If you make a call to https://maps.staffsmoorlands.gov.uk/map2014/Cluster.svc/findLocation with these query parameters:

  1. callback: getAddressesCallback<CURRENT EPOCH IN MILLISECONDS>
  2. script: \Cluster\HP\HPBins.AuroraScript$
  3. address: <QUERY STRING>

…you’ll get a response that looks like this:

getAddressesCallback1528319110177({
	"Error": null,
	"InProgress": false,
	"OperationId": "00000000-0000-0000-0000-000000000000",
	"ProgressMessage": null,
	"ProgressRatio": null,
	"StartTime": null,
	"SubmissionTime": null,
	"Locations": [{
		"Description": "Horse Riding Stables <b>Laneside<\/b> Farm <b>Laneside<\/b> Road New Mills SK22 4LU",
		"Details": null,
		"Id": "53039",
		"X": 401040.6,
		"Y": 385360.8
	}, {
		"Description": "<b>Laneside<\/b> Cottage <b>Laneside<\/b> Road New Mills SK22 4LU",
		"Details": null,
		"Id": "31927",
		"X": 401030.6,
		"Y": 385365.8
	}, {
		"Description": "73 <b>Laneside<\/b> Road New Mills SK22 4LT",
		"Details": null,
		"Id": "54698",
		"X": 401014.6,
		"Y": 385432.8
	}],
	"TotalHits": 3
});

Query 2: Get Collection Info

Once you pick the correct location from the previous query, take note of the Id, X, and Y properties. Then, If you make a call to https://maps.staffsmoorlands.gov.uk/map2014/Cluster.svc/getpage with these query parameters:

  1. script: \Cluster\HP\HPBins.AuroraScript$
  2. taskId: HPbins
  3. format: js
  4. updateOnly: true
  5. query: x=<X VALUE>;y=<Y VALUE>;id=<ID VALUE>

This returns some rather unfortunate JavaScript that, nevertheless, contains the pickup info:

(function injectCss() {
	var head = document.getElementsByTagName('head')[0];
	var style = document.createElement('style');
	var rules = document.createTextNode(".auroraAddressBox { width: 60%; }\n\n.auroraAddressBoxPopup {\n    background-color: #FFF; /* White background color */\n    border: 1px solid #317082; /* Dark green border */\n    font-size: 0.9em;\n    height: 250px; /* Height of box */\n    overflow: auto; /* Scrolling features */\n    position: absolute; /* Never change this one */\n    text-align: left;\n    width: 300px; /* Width of box */\n    z-index: 100;\n}\n\n.auroraAddressBoxPopup div {\n    cursor: pointer;\n    font-size: 0.9em;\n    /* General rule for both .optionDiv and .optionDivSelected */\n    margin: 1px;\n    padding: 1px;\n}\n\n.auroraAddressBoxPopup .optionDiv {\n    /* Div for each item in list */\n    font-family: Arial;\n    font-size: 14px;\n}\n\n.auroraAddressBoxPopup .optionDivSelected {\n    background-color: #317082;\n    color: #FFF;\n    /* Selected item in the list */\n    font-family: Arial;\n    font-size: 14px;\n    font-weight: bold;\n}\n\n.auroraAddressBoxPopupIFrame {\n    background-color: #F00;\n    position: absolute;\n    z-index: 5;\n}\n\n#myarea img {\n    width: 120px;\n}\n\n.auroraItemImage {\n    border: 1px #CCC solid;\n    float: left;\n    margin: 0 12px 17px 0;\n    padding: 4px;\n}\n\n.auroraSection { display: none; }\n\n.auroraHeader {\n    clear: both;\n    font-family: Arial, sans-serif;\n    font-size: 1.231em;\n    font-weight: bold;\n    margin-bottom: 0 !important;\n    margin-top: 0px !important;\n}\n\n.auroraFooter {\n    font: 0.8em/1.5em \"Lucida Grande\", \"Lucida Sans Unicode\", Helvetica, Arial, sans-serif;\n    font-size: 70%;\n}\n\n.auroraList { \n    list-style-type: disc; \n    margin-left: 20px;\n}\n\n.auroraListItem {\n    background: none;\n    clear: both;\n    color: black;\n    display: list-item;\n    margin: 0;\n    padding: 0;\n}\n\n.auroraListItem img {\n    width: auto;\n}\n\n.auroraListItem {\n    background: none;\n    clear: both;\n    color: black;\n    display: list-item;\n    margin: 0;\n    padding: 0;\n}\n\n.auroraList li a {\n    border-bottom: 1px dotted #111;\n    font-weight: bold;\n}\n\n#DR1 .auroraList li.auroraListItem { background: none; }\n\n.auroraListItemWithImage { list-style-type: none; }\n\n.auroraListItem li { display: list-item; }\n\n.wasterBLACK {\n    color: black;\n    font-weight: bold;\n}\n\n.wasterBLUE {\n    color: blue;\n    font-weight: bold;\n}\n\n.wasterGREEN {\n    color: green;\n    font-weight: bold;\n}\n\n.wasterBROWN {\n    color: saddlebrown;\n    font-weight: bold;\n}\n \n.wasterRED{\n    color:red;\n  font-weight: bold;\n}");

	style.type = 'text/css';
	if (style.styleSheet) style.styleSheet.cssText = rules.nodeValue;
	else style.appendChild(rules);
	head.appendChild(style);
})();
document.getElementById("DR1").innerHTML = "\u003cp class=\"auroraSection\"\u003eREFUSE AND RECYCLING\u003c/p\u003e\r\n\u003cp class=\"auroraHeader\"\u003eYour Bin Collection Day\u003c/p\u003e\r\n\u003cul class=\"auroraList\"\u003e\r\n\u003cli class=\"auroraListItem\"\u003eYour next collection day for: \u003cul\u003e\u003cli\u003e\u003cspan class=\u0027wasterBROWN\u0027\u003eBROWN\u003c/span\u003e bin and \u003cspan class=\u0027wasterRED\u0027\u003ered\u003c/span\u003e bag is Wednesday 13/06 \u003cbr /\u003e and then 27/06, 11/07. Your usual collection day for \u003cspan class=\u0027wasterBROWN\u0027\u003eBROWN\u003c/span\u003e bin and \u003cspan class=\u0027wasterRED\u0027\u003ered\u003c/span\u003e bag is Wednesday.\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\r\n\u003c/ul\u003e\r\n";

Obviously, that needs to be scrubbed/decoded/etc. to be used in HASS.

2 Likes

Ok @Cee this is as far as I got - Wife says I need to get the washing up done…

  1. Get the data
    curl -s "https://maps.staffsmoorlands.gov.uk/map2014/Cluster.svc/getpage?script=\Cluster\HP\HPBins.AuroraScript$&taskId=HPbins&format=js&updateOnly=true&query=x%3D405866%3By%3D373349%3Bid%3D10958" > rawData.txt

  2. Extract and convert the unicode to html - (THIS IS A RIGHT HACK !!! and welcome any Code Golfers out there)
    echo -en "$(cat rawData.txt | grep document.getElementById | awk -F'="' '{print $2}' | rev | cut -c 4- | rev)" > decodedHtml.txt

  3. Get the data into a parse-able format (JSON)
    cat decodedHtml.txt |pup 'body json{}' > binCollection.json

Now you need to get creative with a JSON parser or just grep for values you need personnaly id use JQ to find the level you want and go from there

One crappy way would be
cat binCollection.json | jq '.[]' | grep text

results in

"text": "REFUSE AND RECYCLING"
"text": "Your Bin Collection Day"
"text": "BLACK"
"text": "BLACK"
"text": "bin is Wednesday 6/06 and then 20/06, 4/07. Your usual collection day for bin is Wednesday."
"text": "BROWN"
"text": "red"
"text": "BROWN"
"text": "red"
"text": "bin and bag is Wednesday 13/06 and then 27/06, 11/07. Your usual collection day for bin and bag is Wednesday."
"text": "GREEN"
"text": "GREEN"
"text": "bin is Wednesday 13/06 and then 27/06, 11/07. Your usual collection day for bin is Wednesday."
"text": "Your next collection day for:"

Good Luck! :slight_smile:

1 Like

Interesting, those responses look exactly like my (different) council. Looking into mine now…

That’s amazing, I would have got stuck evening trying to work out how to get the correct link. Thank you, thank you, again, super awesome, I’ll see if I can make head or tail of it :smiley:

1 Like

so it seems Hassbian doesn’t have pub installed,

pi@hassbian:~ $ cat decodedHtml.txt |pup 'body json{}' > binCollection.json
-bash: pup: command not found

Let me see if i can work out a way to install it, unless you know another way around it.

sudo apt-get update
Then
sudo apt-get install pup

pi@hassbian:/home/homeassistant/mopidy $ sudo apt-get install pup
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package pup

I did the update first, but no joy.

Guessing I would have to do the direct install from https://github.com/ericchiang/pup ?

Yes sorry I forgot I did it like this

go get github.com/ericchiang/pup

Ok i prodded and probed around linux for a bit, threw lots of random alphabet letters at a command prompt, and I now finally have this output :smiley:

pi@hassbian:~ $ cat binCollection.json | jq '.[]' | grep text
      "text": "REFUSE AND RECYCLING"
      "text": "Your Bin Collection Day"
                      "text": "BLACK"
                      "text": "BLACK"
                  "text": "bin is Wednesday 20/06 and then 4/07, 18/07. Your usual collection day for bin is Wednesday."
                      "text": "BROWN"
                      "text": "red"
                      "text": "BROWN"
                      "text": "red"
                  "text": "bin and bag is Wednesday 13/06 and then 27/06, 11/07. Your usual collection day for bin and bag is Wednesday."
                      "text": "GREEN"
                      "text": "GREEN"
                  "text": "bin is Wednesday 13/06 and then 27/06, 11/07. Your usual collection day for bin is Wednesday."
          "text": "Your next collection day for:"

We are getting somewhere now :smiley:

1 Like

thanks. Unfortunately my council publishes a PDF version with coloured text to reflect the type of bin :frowning:
http://www.crawley.gov.uk/pw/web/PUB272632
Ended up creating a new GCAL and making manual adjustments to cope with bank holidays, I just need to update it once a year…