OK. I’ve just checked my system and it appears to be working as expected.
Can you try the following (random) address and see if it works. It works well with the sample code I gave above.
https://www.wakefield.gov.uk/site/Where-I-Live-Results?uprn=63065603
If that doesn’t work then there is an issue somewhere else.
Keep trying @nwetp !
that JSON has an issue in that it has import( at the start and ); at the end - so its not stricktly a json - howevcer if I strip them off I can format the json string
{
“Results”: {
“bin_collections_combined”: [
{
““: 100010877913,
“_”: “<div class="clearfix 2"><img class="isharefeed" src="https://maps.bolton.gov.uk/residents/custom/mybolton_files/beigebin.jpg\”/>Your next beige bin collection will be on
Wednesday 11 September 2019
”
},
{
"”: 100010877913,
““: “<div class="clearfix 2"><img class="isharefeed" src="https://maps.bolton.gov.uk/residents/custom/mybolton_files/burgundybin.jpg\”/>Your next burgundy bin collection will be on
Wednesday 11 September 2019
”
},
{
“__”: 100010877913,
"”: “<div class="clearfix 2"><img class="isharefeed" src="https://maps.bolton.gov.uk/residents/custom/mybolton_files/greenbin.jpg\”/>Your next green bin collection will be on
Wednesday 11 September 2019
"
},
{
“__”: 100010877913,
“_”: “<div class="clearfix 2"><img class="isharefeed" src="https://maps.bolton.gov.uk/residents/custom/mybolton_files/greybin.jpg\”/>Your next grey bin collection will be on
Wednesday 04 September 2019
"
}
]
}
}
I’d refer to my origional post on this as its a bit more complex than just a rest sensor - You will need to scrape the JSON and use a Home assistant template to get out the values - obviously after stripping the import( and ); bit
Hi
Just tried and still an error.
The only thing that has changed is my router as im now on 4g sim router as no room in exchange.
Everything else works since the move, so i wonder what it is as i can access the link fine to the website via a browser.
It doesnt use a random port or anything apart from port 80 or 443 ?
Thanks for your help
Martyn
It does sound like your router could be part of the issue, though that is a bit beyound my area of understanding. I don’t believe there are any non-standard ports being used.
Are you able to copy over your flow (using the random address below) to see if it works on my set up?
Hi, sorry for the delay in reply but got no notification.
I tried the random address and it was the same.
I think its the router too but everything else works fine.
Cheers for your help.
Hi, what version of HA are you running as i just seen that scraping has been removed in the 0.98 version.
This might explain it.
I’m using 0.97.2
However as the scrape is happening outside of ha (in node red), this should not be an issue.
Still worth a try with an older version?
Will be interesting to see when you upgrade to 0.98.
I have a mate who has it, so he is going to try the flow but i dont know what version he is has currently.
My mate is on 0.98.2 and he is getting the header error too. So looks like it is down to 0.98.
Thanks for your help but i guess there is no fix
Cheers
Martyn
I got that same error when using the Hassio node-red addon. I resolved by installing node red standalone on a spare raspberry pi running raspbian and then running the bin scrape from there.
I guess alternatively you could invoke a wget / curl command from a nodered exec node and get the content that way. I suspect its something to do with the version of the http-parser in node, and the server you’re calling returning duff headers that cause the parser to choke.
What colour would you display for a black bin?
I guess whatever color you want to substitute?
I have yellow/red/green lidded bins so it works fine for me.
It’s a bugger for us we have green and blue, no problem.
But then we’ve got black and grey too …
I wonder where I can find an old multi-selector ??? Or maybe I should build my own… Hmmm.! I feel another project coming on.
Hi, That is an idea, I have a webserver running node red too. I could try on that. How did you get the data to show on the HA though ?
For Wakefield (UK) Bin Collections (copied mostly from previous posts)
In node red I used the following flow sending it via mqtt to the mosquitto broker in Home Assistant:-
Replace the URL in the http request with a valid URL:
[{"id":"b3b9112.7186e7","type":"tab","label":"Flow 4","disabled":false,"info":""},{"id":"48ddbf6c.583e58","type":"inject","z":"b3b9112.7186e7","name":"","topic":"","payload":"","payloadType":"date","repeat":"21600","crontab":"","once":false,"onceDelay":0.1,"x":110,"y":40,"wires":[["4425f225.f6690c"]]},{"id":"c2f5a643.386ba","type":"debug","z":"b3b9112.7186e7","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":450,"y":120,"wires":[]},{"id":"4425f225.f6690c","type":"http request","z":"b3b9112.7186e7","name":"","method":"GET","ret":"txt","url":"https://www.wakefield.gov.uk/site/Where-I-Live-Results?uprn=1234567","tls":"","x":170,"y":120,"wires":[["c2f5a643.386ba","671d937b.284aec","5c77a43b.21cc9c"]]},{"id":"671d937b.284aec","type":"html","z":"b3b9112.7186e7","name":"","property":"payload","outproperty":"payload","tag":"table[class=\"mb10 wilWasteContent gardenFutureData\"]","ret":"html","as":"single","x":490,"y":80,"wires":[["13fcb302.8626e5"]]},{"id":"13fcb302.8626e5","type":"debug","z":"b3b9112.7186e7","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":830,"y":80,"wires":[]},{"id":"5c77a43b.21cc9c","type":"html","z":"b3b9112.7186e7","name":"","property":"payload","outproperty":"payload","tag":"div[class=\"mb10 ind-waste-wrapper\"]","ret":"html","as":"single","x":410,"y":240,"wires":[["c9ec1ce5.516198","4f09f215.41d8fc","6ef6538e.bb042c","f3b60799.89885"]]},{"id":"c9ec1ce5.516198","type":"debug","z":"b3b9112.7186e7","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","x":650,"y":420,"wires":[]},{"id":"4f09f215.41d8fc","type":"html","z":"b3b9112.7186e7","name":"","property":"payload[2]","outproperty":"payload","tag":"div","ret":"html","as":"single","x":650,"y":360,"wires":[["b70c77b8.826be","73e044.a322bfbc"]]},{"id":"b70c77b8.826be","type":"debug","z":"b3b9112.7186e7","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload[6]","x":800,"y":400,"wires":[]},{"id":"6ef6538e.bb042c","type":"html","z":"b3b9112.7186e7","name":"","property":"payload[1]","outproperty":"payload","tag":"div","ret":"html","as":"single","x":650,"y":260,"wires":[["9c45239f.285f18","4cfeb625.c0efc"]]},{"id":"9c45239f.285f18","type":"debug","z":"b3b9112.7186e7","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload[6]","x":820,"y":260,"wires":[]},{"id":"f3b60799.89885","type":"html","z":"b3b9112.7186e7","name":"","property":"payload[0]","outproperty":"payload","tag":"div","ret":"html","as":"single","x":650,"y":220,"wires":[["a63a00a1.759e1","f95e2b63.b5eb9"]]},{"id":"a63a00a1.759e1","type":"debug","z":"b3b9112.7186e7","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload[6]","x":820,"y":160,"wires":[]},{"id":"e695a1c5.9861","type":"mqtt out","z":"b3b9112.7186e7","name":"","topic":"home/outside/recycling","qos":"2","retain":"true","broker":"6ef3cbf0.00da34","x":970,"y":300,"wires":[]},{"id":"6ee6ef90.bfeac","type":"mqtt out","z":"b3b9112.7186e7","name":"","topic":"home/outside/garden","qos":"2","retain":"true","broker":"6ef3cbf0.00da34","x":940,"y":360,"wires":[]},{"id":"a27dec0b.c5ee8","type":"mqtt out","z":"b3b9112.7186e7","name":"","topic":"home/outside/waste","qos":"2","retain":"true","broker":"6ef3cbf0.00da34","x":960,"y":200,"wires":[]},{"id":"f95e2b63.b5eb9","type":"function","z":"b3b9112.7186e7","name":"","func":"msg.payload = msg.payload[6];\nreturn msg;","outputs":1,"noerr":0,"x":790,"y":200,"wires":[["a27dec0b.c5ee8"]]},{"id":"4cfeb625.c0efc","type":"function","z":"b3b9112.7186e7","name":"","func":"msg.payload = msg.payload[6];\nreturn msg;","outputs":1,"noerr":0,"x":790,"y":300,"wires":[["e695a1c5.9861"]]},{"id":"73e044.a322bfbc","type":"function","z":"b3b9112.7186e7","name":"","func":"msg.payload = msg.payload[6];\nreturn msg;","outputs":1,"noerr":0,"x":770,"y":360,"wires":[["6ee6ef90.bfeac"]]},{"id":"6ef3cbf0.00da34","type":"mqtt-broker","z":"","name":"Hassio","broker":"192.168.0.100","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]
Then in home assistant in the sensor.yaml file I added the following:
- platform: mqtt
state_topic: 'home/outside/waste'
name: 'Waste'
- platform: mqtt
state_topic: 'home/outside/garden'
name: 'Garden'
- platform: mqtt
state_topic: 'home/outside/recycling'
name: 'Recycling'
Got Bolton working thanks to some help here: Help getting JSON into Restful / Template Sensor
Thanks guys!
Okay, I have been inspired by some of the efforts on this thread and it appeals to the lazy B4574rd in me (okay doing this is more work but it’s fun)
Given that the colour light wouldn’t work for me I was thinking of building a rotary indicator.
This way whenever I’m in the kitchen I could just look up and see what the next collection was.
I could have it ‘between’ colours when not due and have it select the colour on the day before and on the day of collection.
I was looking for a mains powered solenoid to operate a ratchet that I’d cut into an MDF disc.
This seemed like a lot of time and effort so …
I was walking round ikea and they have some very cheap, very naff clocks
Hows about me buying one of these clocks, getting rid of the hour hand, painting the face into sections, getting a cheap phone charger (lots lying around) a buck converter, use a HA compatible power switch, and sticking them together.
we have a four week cycle (but you could adapt for 3 week or whatever)
I’d run the clock 7.5 min after the collection to take it to a border, then 7.5 mins the day before etc. You could run it at say 03:00 or something (no one should ever see it move !)
The switch brings the psu on, feeds the buck converter to supply 3v (2 off AAA’s equivalent)
And you have a steady state coloured indicator, no need to look at a screen. Wife friendly 
I could stick a more ‘visible’ pointer over the hand if it was too thin
What do you think ?
Any alternate suggestions ?
Love the creativity! Why not take the clock idea but setup a stepper motor? You could control the steps from a nodemcu
I just get my amazon echo to announce every hour from 6pm the day before what bin needs to be put out