Just a note, not sure if you have changed it for the screenshot, but there is a full address in the top of the website screenshot.
1 Like
Thanks.
Yes, I did change the address.
1 Like
This is the un-developed version - note the cookie or session might expire and this needs to be addressed
You can see the address 1 Hedgerow Close at the end of the url
curl 'http://www.stevenage.gov.uk/find/' -H 'Connection: keep-alive' -H 'Cache-Control: max-age=0' -H 'Origin: http://www.stevenage.gov.uk' -H 'Upgrade-Insecure-Requests: 1' -H 'Content-Type: application/x-www-form-urlencoded' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'Referer: http://www.stevenage.gov.uk/find/' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.9' -H 'Cookie: _ga=GA1.3.418271238.1531777870; _gid=GA1.3.2006517484.1531777870; ASP.NET_SessionId=xizub0v52p2bwlbh3rxb5jjc; _gat=1; _hjIncludedInSample=1' --data '__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUKMTM5NDg2MzcxMQ9kFgJmD2QWAmYPZBYCAgEPZBYEAgEPEGRkFgBkAgMPZBYEAgMPZBYCZg8UKwACDxYEHgtfIURhdGFCb3VuZGceC18hSXRlbUNvdW50AidkZBYCZg9kFk4CAQ9kFgJmDxUDABIvY291bmNpbC1zZXJ2aWNlcy8QQ291bmNpbCBTZXJ2aWNlc2QCAg9kFgJmDxUDABMvYWJvdXQtdGhlLWNvdW5jaWwvEUFib3V0IHRoZSBDb3VuY2lsZAIDD2QWAmYPFQMAES9hYm91dC1zdGV2ZW5hZ2UvD0Fib3V0IFN0ZXZlbmFnZWQCBA9kFgJmDxUDABEvbmV3cy1hbmQtZXZlbnRzLw9OZXdzIGFuZCBFdmVudHNkAgUPZBYCZg8VAwAHL2xpbmtzLwVMaW5rc2QCBg9kFgJmDxUDAA8vaGF2ZS15b3VyLXNheS8NSGF2ZSBZb3VyIFNheWQCBw9kFgJmDxUDABMvY29tbXVuaXR5LWNlbnRyZXMvEUNvbW11bml0eSBDZW50cmVzZAIID2QWAmYPFQMADi9wZXN0LWNvbnRyb2wvDFBlc3QgY29udHJvbGQCCQ9kFgJmDxUDAAovYmVuZWZpdHMvCEJlbmVmaXRzZAIKD2QWAmYPFQMADS9jb3VuY2lsLXRheC8LQ291bmNpbCBUYXhkAgsPZBYCZg8VAwAJL2hvdXNpbmcvB0hvdXNpbmdkAgwPZBYCZg8VAwALL2xpY2Vuc2luZy8cTGljZW5zaW5nIGFuZCBTdHJlZXQgVHJhZGluZ2QCDQ9kFgJmDxUDAAkvZ2FyYWdlcy8HR2FyYWdlc2QCDg9kFgJmDxUDABEvYnVzaW5lc3MtYWR2aWNlLwhCdXNpbmVzc2QCDw9kFgJmDxUDABAvYW5pbWFsLXdlbGZhcmUvDkFuaW1hbCBXZWxmYXJlZAIQD2QWAmYPFQMACy9wb2xsdXRpb24vCVBvbGx1dGlvbmQCEQ9kFgJmDxUDABUvbGFuZHNjYXBlLXdvb2RsYW5kcy8XTGFuZHNjYXBlIGFuZCBXb29kbGFuZHNkAhIPZBYCZg8VAwAVL2xvY2FsLWxhbmQtc2VhcmNoZXMvE0xvY2FsIExhbmQgU2VhcmNoZXNkAhMPZBYCZg8VAwAVL3JlY3ljbGluZy1hbmQtd2FzdGUvE1dhc3RlIGFuZCBSZWN5Y2xpbmdkAhQPZBYCZg8VAwAWL3N0cmVldC1jYXJlLWNsZWFuaW5nLxhTdHJlZXQgQ2FyZSBhbmQgQ2xlYW5pbmdkAhUPZBYCZg8VAwATL2hlYWx0aC1hbmQtc2FmZXR5LxFIZWFsdGggYW5kIFNhZmV0eWQCFg9kFgJmDxUDAA0vZm9vZC1zYWZldHkvC0Zvb2QgU2FmZXR5ZAIXD2QWAmYPFQMAFS9wYXNzZW5nZXItdHJhbnNwb3J0LxNQYXNzZW5nZXIgVHJhbnNwb3J0ZAIYD2QWAmYPFQMAIC9yb2Fkcy1hbmQtcGF0aHdheXMtbWFpbnRlbmFuY2UvHlJvYWRzIGFuZCBQYXRod2F5cyBNYWludGVuYW5jZWQCGQ9kFgJmDxUDAAkvcGFya2luZy8HUGFya2luZ2QCGg9kFgJmDxUDAAcvNDg3MDgvE0NvbW11bml0eSBUcmFuc3BvcnRkAhsPZBYCZg8VAwASL2NvbW1lcmNpYWwtd2FzdGUvHkNvbW1lcmNpYWwgV2FzdGUgYW5kIFJlY3ljbGluZ2QCHA9kFgJmDxUDABAvdGF4aS1saWNlbnNpbmcvDlRheGkgTGljZW5zaW5nZAIdD2QWAmYPFQMACC8xMzU3NjgvFlB1YmxpYyBIZWFsdGggRnVuZXJhbHNkAh4PZBYCZg8VAwAILzE0OTY5MC8rUGxhbm5pbmcsIEJ1aWxkaW5nIENvbnRyb2wgYW5kIFJlZ2VuZXJhdGlvbmQCHw9kFgJmDxUDABAvYnVzaW5lc3MtcmF0ZXMvDkJ1c2luZXNzIFJhdGVzZAIgD2QWAmYPFQMADi9yZWdlbmVyYXRpb24vDFJlZ2VuZXJhdGlvbmQCIQ9kFgJmDxUDAAgvMTYzMzEyLxNMZWlzdXJlIGFuZCBDdWx0dXJlZAIiD2QWAmYPFQMAGi9zcG9ydHMtY2x1YnMtYW5kLWNlbnRyZXMvGFNwb3J0cyBDbHVicyBhbmQgQ2VudHJlc2QCIw9kFgJmDxUDAAcvNTI3MTAvB0N5Y2xpbmdkAiQPZBYCZg8VAwAXL3BhcmtzLWFuZC1vcGVuLXNwYWNlcy8VUGFya3MgYW5kIE9wZW4gU3BhY2VzZAIlD2QWAmYPFQMABi9wbGF5LwRQbGF5ZAImD2QWAmYPFQMABy8zMTAyOC8WQXJ0cyBhbmQgRW50ZXJ0YWlubWVudGQCJw9kFgJmDxUDAAovdGhlLXRvdXIvETIwMTggVG91ciBzZXJpZXMgZAIHD2QWCAIDDxYCHgRUZXh0ZWQCCw8WAh8BAv%2F%2F%2F%2F8PZAINDxYCHwECBBYKZg9kFgJmDxUEJDEgSGVkZ2Vyb3cgQ2xvc2UsIFN0ZXZlbmFnZSwgU0cyIDdFQhRUdWVzZGF5IDI0IEp1bHkgMjAxOAlSZWN5Y2xpbmcdMUhlZGdlcm93Q2xvc2VTdGV2ZW5hZ2VTRzI3RUJkAgEPZBYCZg8VAhRUdWVzZGF5IDMxIEp1bHkgMjAxOAZSZWZ1c2VkAgIPZBYCZg8VAhVUdWVzZGF5IDcgQXVndXN0IDIwMTgJUmVjeWNsaW5nZAIDD2QWAmYPFQIWVHVlc2RheSAxNCBBdWd1c3QgMjAxOAZSZWZ1c2VkAgQPZBYCZg8VAhZUdWVzZGF5IDIxIEF1Z3VzdCAyMDE4CVJlY3ljbGluZ2QCDw8WAh8BAgMWCGYPZBYCZg8VAQVNYW5vcmQCAQ9kFgJmDxUGDEdyYWhhbSBTbmVsbBBMaWJlcmFsIERlbW9jcmF0DzE3NyBEdXJoYW0gUm9hZAwwMTQzOCAyMzgzOTkdZ3JhaGFtLnNuZWxsQHN0ZXZlbmFnZS5nb3YudWsdZ3JhaGFtLnNuZWxsQHN0ZXZlbmFnZS5nb3YudWtkAgIPZBYCZg8VBg9BbmR5IE1jR3Vpbm5lc3MQTGliZXJhbCBEZW1vY3JhdA80MDMgVmFyZG9uIFJvYWQMMDE0MzggMjE5NTY5IEFuZHkuTWNHdWlubmVzc0BzdGV2ZW5hZ2UuZ292LnVrIEFuZHkuTWNHdWlubmVzc0BzdGV2ZW5hZ2UuZ292LnVrZAIDD2QWAmYPFQYMUm9iaW4gUGFya2VyEExpYmVyYWwgRGVtb2NyYXQPOCBDYW1lcm9uIENsb3NlDDAxNDM4IDcyNDc0Nh1yb2Jpbi5wYXJrZXJAc3RldmVuYWdlLmdvdi51ax1yb2Jpbi5wYXJrZXJAc3RldmVuYWdlLmdvdi51a2QYAQUtY3RsMDAkY3RsMDAkQ29udGVudFBsYWNlSG9sZGVyMSRNZW51JFRvcExldmVsDxQrAA5kZGRkZGRkPCsAJwACJ2RkZGYC%2F%2F%2F%2F%2Fw9kKH1nQhBd4loa1ZSkb0JY%2FTs9kUAgjbCbiGfI6QUECxU%3D&__VIEWSTATEGENERATOR=B72F5FBC&__EVENTVALIDATION=%2FwEdAAfj2SGJ%2BYySK10dhT7Z9CxxBRnY16VB%2BawiyIVNhmj7d8EVmI3aJcQdmuXHemHELFCoJoTREh6yQXa%2BkswJLGCOHIAGA40MkPa9Woarpjy083LhTQZWmmHCfz3lo41MX02r%2BnwvbOKjWJAVNQqUlqTebs1TP6IKPXYJxFLrqyK0JfY%2Bj5JSU5pwUWOFAL%2BOlq4%3D&ctl00%24ctl00%24SearchBox=Search+for...&ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24txtNumber=1&ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24txtStreet=hedgerow+close&ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24txtPostcode=&ctl00%24ctl00%24ContentPlaceHolder1%24ContentPlaceHolder1%24BtnLookup=Street+Lookup' --compressed
holy moly 
2 Likes
I don’t know about others but cheshire east supports GET so you can use a scrape sensor
Select and value for recycling bin
select: "tr.wasteType_3"
value_template: '{{ as_timestamp(strptime(value, " %A %d-%b-%Y Recycling wheeled bin ")) }}'That’s what I originally did, but there was some issue with the way it worked when I was testing it.
Also I was unable to achieve a throttled way to get the information. While Cheshire East offers the page for general consumption via a web browser, it’s not an api, so I didn’t want to hammer the page. I couldn’t work that out using the scrape sensor.
Very true, Whilst a scrape sensor works I wanted a countdown, which I can’t make work with scrape so i’ve gone down the Node Red route.
It scrapes the data phrases it to number of days until its bin day and passes it back to HA via MQTT
then HA has a automation that kicks off the Node Red flow everyday just after midnight
This allowed me to have one sensor for each bin.
I may tweak it yet so it displays “Today” instead of 0 Days and “Tomorrow” instead of 1 Days
3 Likes
That’s amazing nice work
Quick question as I’m a bit stuck. I get the following json from my cities api.
{ "next_pickup_date_long": "Thursday, July 26th", "next_recycling_date": "7-26-2018", "next_recycling": "The next pickup is trash only."}
And I’m trying to get the next_recycling and next_pickup_date_long into some sensors:
- platform: rest
resource: !secret trashapi
name: Trash Next Recycling
value_template: '{{json.next_recycling}}'
- platform: rest
resource: !secret trashapi
name: Trash Next Pickup
value_template: '{{json.next_pickup_date_long}}'
And despite being able to use the template tool to get the correct output they give “unknown”.
I’ve tried quite a few things including:
value_template: '{{value_json.next_pickup_date_long}}'
Hoping someone might be able to tell me what a bonehead I am and point me in the right direction.
Thanks in advance.
What does the sensor say in the logs? (Info logging level)
I hadn’t noticed before, but it appears the json is wrapped in square brackets. Won’t be at home for a while to test, but I think I’ll have to find a way to strip them. Thanks, I’m going to blame lack of sleep for ignoring the logs.
1 Like
Just to follow up. I had tried many variations of this syntax, but never landed on the winning combo until now. So to help the next guy, here it is:
value_template: '{{value_json[0].next_pickup_date_long}}'
1 Like
Hey,
Trying to pull something similar from my local council’s site (fake address) https://mapping.dorsetforyou.gov.uk/mylocal/viewresults/10091017024
Doesn’t look like I can use your method as the resulting site looks much more complicated than yours. Don’t suppose you could take a look and see if it’s possible?
I did look at the scrape method for html-tag but don’t think there’s unique selectors for each bin collection.
Thanks
Lots of awesome work here, but I’ve gotta say I think everyone is missing a more elegant and universal solution - iCal - most of these councils provide ical feeds (because that’s actually the right thing for them to provide!)
I’m using a customised version of this: Integrate iCal-Events
and then I have the following configuration (I should ideally have a scan limit/refresh rate set in config, but it’s hardcoded in my custom component):
sensor:
- platform: ical
name: "Refuse Collection"
url: "councils-ical-url!"
prefix: xxxx
map:
- value: "Black box"
icon: mdi:glass-tulip
name: "Black - glass and electronics Bin"
- value: "Green refuse"
icon: mdi:delete
name: "Green - refuse Bin"
- value: "Brown garden bin"
icon: mdi:nature
name: "Brown - garden bin"
- value: "Blue recycling bin"
icon: mdi:recycle
name: "Blue - recycling Bin"
- value: "Food caddy"
icon: mdi:food
name: "Grey - food Bin"
automation:
- alias: Evening bins notification_018383
trigger:
- platform: event
event_type: early_eve
condition:
condition: or
conditions:
- condition: template
value_template: '{{ states.sensor.xxxx0.attributes.days == 1 }}'
- condition: template
value_template: '{{ states.sensor.xxxx1.attributes.days == 1 }}'
- condition: template
value_template: '{{ states.sensor.xxxx2.attributes.days == 1 }}'
- condition: template
value_template: '{{ states.sensor.xxxx3.attributes.days == 1 }}'
- condition: template
value_template: '{{ states.sensor.xxxx4.attributes.days == 1 }}'
action:
- service: notify.slk
data_template:
message: "{% set counter = 0 %} {% if states.sensor.xxxx0.attributes.days == 1 %}{{ states.sensor.xxxx0.attributes.friendly_name}}{% if counter > 0 %},{% endif %}{% endif %}{% if states.sensor.xxxx1.attributes.days == 1 %}{{ states.sensor.xxxx1.attributes.friendly_name }}{% if counter > 0 %},{% endif %} {% endif %}{% if states.sensor.xxxx2.attributes.days == 1 %}{{ states.sensor.xxxx2.attributes.friendly_name }}{% if counter > 0 %},{% endif %} {% endif %} {% if states.sensor.xxxx3.attributes.days == 1 %}{{ states.sensor.xxxx3.attributes.friendly_name }}{% if counter > 0 %},{% endif %} {% endif %}{% if states.sensor.xxxx4.attributes.days == 1 %}{{ states.sensor.xxxx4.attributes.friendly_name }} {% if counter > 0 %},{% endif %}{% endif %}needs to be taken out tommorrow"Infact - here you go ya filthy animals 
:
"""
Support for iCal-URLs
For more details about this platform, please refer to the documentation at
https://home-assistant.io/components/sensor.ical/
"""
import logging
from datetime import timedelta
#import voluptuous as vol
import datetime as dt
from datetime import *
from homeassistant.util import Throttle
from homeassistant.helpers.entity import Entity
from homeassistant.components.sensor import PLATFORM_SCHEMA
from homeassistant.const import STATE_UNKNOWN, CONF_NAME
import homeassistant.helpers.config_validation as cv
import voluptuous as vol
DOMAIN='group_avaerage'
DEFAULT_NAME="GROUPING"
PLATFORM_SCHEMA = PLATFORM_SCHEMA.extend({
vol.Optional(CONF_NAME, default=DEFAULT_NAME): cv.string,
})
import icalendar, requests, arrow
_LOGGER = logging.getLogger(__name__)
REQUIREMENTS = ['icalendar', 'requests', 'arrow>=0.10.0']
ICON = 'mdi:calendar'
DEFAULT_NAME = 'iCal Sensor'
DEFAULT_MAX_EVENTS = 5
DOMAIN = 'ical'
# Return cached results if last scan was less then this time ago.
MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=120)
def setup_platform(hass, config, add_devices, discovery_info=None):
"""Setup the iCal Sensor."""
url = config.get('url')
SNS_PREFIX = config.get("prefix", "sensor")
name = config.get('name', DEFAULT_NAME)
mp = config.get("map")
if mp is not None:
mp = list(mp)
maxevents = config.get('maxevents',DEFAULT_MAX_EVENTS)
if url is None:
_LOGGER.error('Missing required variable: "url"')
return False
data_object = ICalData(url)
data_object.update()
if data_object.data is None:
_LOGGER.error('Unable to fetch iCal')
return False
sensors = []
for eventnumber in range(maxevents):
sensors.append(ICalSensor(hass,data_object,eventnumber, SNS_PREFIX, mp))
add_devices(sensors)
# add_devices_callback([ICalSensor(hass, data_object,name)])
return True
def dateparser(calendar,date):
events = []
for event in calendar.walk('VEVENT'):
if type(event['DTSTART'].dt) is dt.date:
start = arrow.get(event['DTSTART'].dt)
start = start.replace(tzinfo='local')
else: start = event['DTSTART'].dt
if type(event['DTEND'].dt) is dt.date:
end = arrow.get(event['DTEND'].dt)
end = end.replace(tzinfo='local')
else: end = event['DTEND'].dt
if start.date() >= date.date():
events.append(dict(name=event['SUMMARY'],begin=start))
sorted_events = sorted(events, key=lambda k: k['begin'])
_LOGGER.info(sorted_events)
return sorted_events
# pylint: disable=too-few-public-methods
class ICalSensor(Entity):
"""Implementation of a iCal sensor."""
def __init__(self, hass, data_object, eventnumber,SNS_PREFIX, mp):
"""Initialize the sensor."""
self._eventno = eventnumber
self._hass = hass
self._days = 1000
self._pickup_type = data_object.data[eventnumber]['name']
self.data_object = data_object
self._name = SNS_PREFIX + str(eventnumber)
self._entity_id = 'sensor.'+SNS_PREFIX + str(eventnumber)
self.entity_id = 'sensor.'+SNS_PREFIX + str(eventnumber)
self.object_id = 'sensor.'+SNS_PREFIX + str(eventnumber)
self._icn = "help-circle-outline"
self.ma = mp
self.update()
@property
def name(self):
"""Return the name of the sensor."""
return self._friendly_name
@property
def icon(self):
"""Return the icon for the frontend."""
return ICON
@property
def state(self):
"""Return the date of the next event."""
return self._state
@property
def device_state_attributes(self):
"""Return the state attributes of the device."""
attr = {}
attr['friendly_name'] = self._friendly_name
attr['pickup_type'] = self._pickup_type
attr['icon'] = self._icn
attr['days'] = self._days
#attr[ATTR_ATTRIBUTION] = CONF_ATTRIBUTION
return attr
def update(self):
"""Get the latest update and set the state."""
self.data_object.update()
e = self.data_object.data
if self._eventno not in range(0,len(e)):
self._state = "No event"
else:
if not e:
self._state = "No event"
else:
val = e[self._eventno]
#self._name = val['name']
#self._friendlyname = val['name']
#self._name = val['name']
# glass = glass-tulip
# recycling = recycle
# food = food
# general = delete
# garden = nature
# default = help-circle-outline
#if ! map is none:
icon = "help-circle-outline"
t = val["name"]
mp = self.ma
if mp is not None:
for m in mp:
if m["value"] in val['name']:
icon = m['icon']
t = m['name']
# if "Food caddy" in val['name']:
# icon = "food"
# t = "Grey - food Bin"
# if "Blue recycling bin" in val['name']:
# icon = "recycle"
# t = "Blue - recycling Bin"
# if "Brown garden bin" in val['name']:
# icon = "nature"
# t = "Brown - garden bin"
# if "Green refuse" in val['name']:
# icon = "delete"
# t = "Green - refuse Bin"
# if "Black box" in val['name']:
# icon = "glass-tulip"
# t = "Black - glass and electronics Bin"
self._icn = "mdi:" + icon
self._pickup_type = t
self.friendly_name = t
self._friendly_name = t
today = date.today()
year = int(val['begin'].strftime("%Y"))
month = int(val['begin'].strftime("%-m"))
day = int(val['begin'].strftime("%-d"))
currentlyis = today.strftime("%a");
future = date(year,month,day)
v = int((future - today).days)
output = ''
currentdayindice = int(date.today().strftime("%w"))
if v == 0:
output = "Today"
if v == 1:
output = "Tommorrow"
w = currentdayindice + v
currentdayindice = int(date.today().strftime("%w"))
onday = future.strftime("%A")
if v >= 2:
output = ''
dayarr = ['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday']
dow = (7-currentdayindice)+v+2
while dow >= 7:
dow = dow - 7
output = "{}".format(onday)
#output = dow
if v >= 7:
s = ''
if v >= 14:
s = "s"
output = "{} week{} from {} ".format(int(v/7), s, onday)
self._state = output
self._days = v
#self._state = (future - today).days
#self._state = val['begin'].strftime("%-d %B %Y")
#self._state = "{} - {}".format( val['name'], val['begin'].strftime("%-d %B %Y"))
#pylint: disable=too-few-public-methods
class ICalData(object):
"""Class for handling the data retrieval."""
def __init__(self, resource):
self._request = requests.Request('GET', resource).prepare()
self.data = None
@Throttle(MIN_TIME_BETWEEN_UPDATES)
def update(self):
self.data = []
try:
with requests.Session() as sess:
response = sess.send(self._request, timeout=10)
cal = icalendar.Calendar.from_ical(response.text)
today = arrow.utcnow()
events = dateparser(cal,today)
self.data = events
except requests.exceptions.RequestException:
_LOGGER.error("Error fetching data: %s", self._request)
self.data = Nonei think there are more that provide gcal (and that would be the right thing, because more people use gcal then ical 
)
google does ical - I’m not referring to “icloud calendar” but rather to ics files (commonly called ical as it’s full name is icalendar) - which is a standard and supported by many companies (including google 
) https://tools.ietf.org/html/rfc5545
source: I too use google calendar, but I also use ical feeds as it’s the standardised way to share calendar information
then if thats the same thing, why use a custom component?
just add the google calendar to your HA and the ical file to your google calendar
- when I put that together, the google calendar thing was different
- I assume the google calendar thing names the sensors (IIRC that was how it did it back then) based on the name of the event - not good for this
- the custom component does a little extra
- the custom component negates the need to import the bin events in to your Google Calendar
- it’s simpler (less 3rd parties involved)