Trying to use CURL POST to find which bin goes out and when from Council website

Hi,

I’m new to HA and struggling a lot but also enjoying every sometimes tedious and frustrating moment as I learn bit by bit and the insane joy I get from solving a puzzle is just immense.

My current task is to query our local councils website and find out which bin goes out and when.

They have a two part form where you have to enter your Postcode and then select your address from the retrieved list of houses at that Postcode.

I have a working CURL POST command which does this and I can run this on the command line of my Pi and it gives me the full HTML page (expected) containing a table with the two bins and the next collection dates.

The command I am using is:

curl -X POST https://www.scotborders.gov.uk/bincollections -d "postcode=EH24 6XX&address=123456789"

I’ve added this as a shell_command to my configuration.yaml file and restarted.

shell_command:
bin_dates: curl -X POST https://www.scotborders.gov.uk/bincollections -d "postcode=EH24 6XX&address=123456789"

I’m trying to call this from an automation as the service shell_command.bin_dates and send the output as a message to my phone (via a script) but no alert is being sent. The message sends as soon as I remove the {{ bindates }}. I’m doing this as I don’t know of any other way of displaying the response within HA.

action:
  - service: shell_command.bin_dates
    response_variable: bindates
    data: {}
  - service: script.send_message_to_phone
    data:
      title: Your next bin is
      message: {{ bindates }}

I’m assuming one of two things are happening here. Either {{ bindates }} is empty in which case I would have expected a message to be sent with just a title. Or the full HTML output of CURL is too large for the message so nothing sends (but I can’t see any limits to the message size in the docs). Or something else, at this stage in my HA journey, it could be anything :slight_smile:

What I need to do and this is where I’m struggling, is to strip the response from CURL down to just the sections of the table I am after and I don’t know how to do this. Sadly the Council webpage doesn’t give me nice JSON I can run through to grab the values.

The section I am after is:

	<h2 id="dates">Bin collection days and dates for My ADDRESS</h2>
				<table border="1">
					<thead>
						<tr>
							<th>Bin</th>
							<th>Collection day</th>
							<th>Next collection</th>
						</tr>
					</thead>
					<tbody>
						<tr>
							<td>General</td>
							<td>Tuesday every 2 weeks</td>
							<td>12/12/2023</td>
						</tr>
						<tr>
							<td>Recycling</td>
							<td>Thursday every 2 weeks</td>
							<td>21/12/2023</td>
						</tr>
						<tr>
							<td>Food</td>
							<td>No collection</td>
							<td>No collection</td>
						</tr>
					</tbody>
				</table>

Specifically the first two containing General and Recycling and the dates.

Eg:
General 12/12/2023
Recycling: 21/12/2023

Ideally I would run this daily and it would send a message if it was the night before one of the bin dates and advise which bin goes out so we can be the streets binfluencer.

Any help on how to proceed would be most appreciated. And thank you.

Many thanks,

I am unable to answer your question, but you might like to use this custom integration. GitHub - mampfes/hacs_waste_collection_schedule: Home Assistant integration framework for (garbage collection) schedules

Thanks as I did not know this too and LOL… I just checked this for ‘France’ …which actually only covers Mamirolle with about 1700 inhabitants.(I do understand one can contribute)

Well it covers my council in little old New Zealand, which was a happy surprise.

Just to follow up.

None of the options available work with our Council and I also want to scrape other websites so I went the manual route.

Decided to use Node-Red so I could break out into a Python script then pass the results back in to HA using a Call Service notify.

I’m pasting my code here in case it helps anyone else. Feel free to suggest ways of improving it. I know it’s scrappy but it works :slight_smile:

#!/usr/bin/env python3

import requests
import datetime
from bs4 import BeautifulSoup

# Set up URL and payload
url="https://www.scotborders.gov.uk/bincollections"
payload = {"postcode": "EH99 9AA", "address": "123456789"}
# Capture response
response = requests.post(url, data=payload)
html = response.text
soup = BeautifulSoup(html, 'html.parser')
tbody = soup.find('tbody')
tr = tbody.findAll('tr')
# Create Bin List
binlist = []
for row in tr:
    td = row.findAll('td')
   # Insert rows as dictionaries into Bin List
    binlist.append({'Bin': td[0].text, 'Day': td[1].text, 'Date': td[2].text})
# Work out what tomorrow's date will be by taking today and adding a delta of 1 and formatting it correctly
tomorrowsdate = datetime.datetime.strftime((datetime.date.today() + datetime.timedelta(days=1)), '%d/%m/%Y')
# Convert the Council website date from a string into python datetime and format the same as the tomorrowsdate variable 
nextbin = datetime.datetime.strptime((binlist[0]['Date']), '%d/%m/%Y').date()
#Iterate through the list of dictionaries, first changing the words general to black and everything else to blue
# then compare the bindate with tomorrow and if they are the same, print out which bin it is
for bins in binlist. We ignore the Food waste as our Council removed this service from our town
        for key, value in bins.items():
                if (key == 'Bin'):
                        BinType = value
                        if (BinType == 'General'):
                                BinType = 'Black'
                        else:
                                BinType = 'Blue'
                if (key == 'Date'):
                        if (value != 'No collection'):
                                if (value == tomorrowsdate):
                                        print('The', BinType, 'bin goes out tomorrow')

And here is my Flow if it’s any use to you.

[
    {
        "id": "9837bc850425616d",
        "type": "tab",
        "label": "Flow 1",
        "disabled": false,
        "info": "",
        "env": []
    },
    {
        "id": "a22b375ebafd4a97",
        "type": "exec",
        "z": "9837bc850425616d",
        "command": "python /data/python/bins.py",
        "addpay": "",
        "append": "",
        "useSpawn": "false",
        "timer": "",
        "winHide": false,
        "oldrc": false,
        "name": "Python bins.py",
        "x": 400,
        "y": 320,
        "wires": [
            [
                "d911089a8750f07c",
                "fec9a5637b569eba"
            ],
            [
                "0bb96b1c6017c955"
            ],
            []
        ]
    },
    {
        "id": "1b1ebc547775ae8e",
        "type": "inject",
        "z": "9837bc850425616d",
        "name": "Run at specific time",
        "props": [],
        "repeat": "",
        "crontab": "30 18 * * 1,3",
        "once": false,
        "onceDelay": 0.1,
        "topic": "",
        "x": 180,
        "y": 320,
        "wires": [
            [
                "a22b375ebafd4a97"
            ]
        ]
    },
    {
        "id": "0bb96b1c6017c955",
        "type": "debug",
        "z": "9837bc850425616d",
        "name": "debug 1",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 600,
        "y": 360,
        "wires": []
    },
    {
        "id": "d911089a8750f07c",
        "type": "debug",
        "z": "9837bc850425616d",
        "name": "stdout",
        "active": true,
        "tosidebar": true,
        "console": true,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 600,
        "y": 300,
        "wires": []
    },
    {
        "id": "fec9a5637b569eba",
        "type": "api-call-service",
        "z": "9837bc850425616d",
        "name": "Bin Notification",
        "server": "1ccae86e51a00f24",
        "version": 5,
        "debugenabled": true,
        "domain": "notify",
        "service": "mobile_app_heds_iphone",
        "areaId": [],
        "deviceId": [],
        "entityId": [],
        "data": "{\"title\":\"Time to put the bins out\",\"message\": msg.payload}",
        "dataType": "jsonata",
        "mergeContext": "",
        "mustacheAltTags": false,
        "outputProperties": [],
        "queue": "none",
        "x": 620,
        "y": 240,
        "wires": [
            []
        ]
    },
    {
        "id": "1ccae86e51a00f24",
        "type": "server",
        "name": "Home Assistant",
        "version": 5,
        "addon": false,
        "rejectUnauthorizedCerts": false,
        "ha_boolean": "y|yes|true|on|home|open",
        "connectionDelay": true,
        "cacheJson": true,
        "heartbeat": true,
        "heartbeatInterval": "30",
        "areaSelector": "friendlyName",
        "deviceSelector": "friendlyName",
        "entitySelector": "friendlyName",
        "statusSeparator": ": ",
        "statusYear": "hidden",
        "statusMonth": "short",
        "statusDay": "numeric",
        "statusHourCycle": "default",
        "statusTimeFormat": "h:m",
        "enableGlobalContextStore": false
    }
]

Hello fellow Scottish Borders resident.
I also use Node Red.

Here it is in case it helps someone who doesn’t want to rely on external stuff.

[{"id":"bede72a39474db90","type":"tab","label":"BinDays","disabled":false,"info":"","env":[]},{"id":"6dfd4daded874656","type":"inject","z":"bede72a39474db90","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"21600","crontab":"","once":true,"onceDelay":"10","topic":"","payload":"","payloadType":"date","x":90,"y":40,"wires":[["76411310dd29e3ef"]]},{"id":"f7d817f9a7433071","type":"http request","z":"bede72a39474db90","name":"","method":"POST","ret":"txt","paytoqs":"ignore","url":"https://www.scotborders.gov.uk/bincollections","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"other","keyValue":"user-agent","valueType":"other","valueValue":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"},{"keyType":"other","keyValue":"origin","valueType":"other","valueValue":"https://www.scotborders.gov.uk"},{"keyType":"other","keyValue":"referer","valueType":"other","valueValue":"https://www.scotborders.gov.uk/bincollections"},{"keyType":"other","keyValue":"content-type","valueType":"other","valueValue":"application/x-www-form-urlencoded"}],"x":110,"y":120,"wires":[["8083d0859ef9d04d"]]},{"id":"1b164cb0e0d9ce5c","type":"debug","z":"bede72a39474db90","name":"debug 2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":340,"y":460,"wires":[]},{"id":"8083d0859ef9d04d","type":"file","z":"bede72a39474db90","name":"","filename":"/config/sbc_test_file.html","filenameType":"str","appendNewline":false,"createDir":false,"overwriteFile":"true","encoding":"none","x":150,"y":180,"wires":[["9f6f8e63da2013c5"]]},{"id":"76411310dd29e3ef","type":"function","z":"bede72a39474db90","name":"Set Post","func":"var p = {};\np.postcode = \"TD10+XXX\";\np.address = \"123456789\";\np.search = \"\";\nmsg.payload = p;\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":280,"y":40,"wires":[["f7d817f9a7433071"]]},{"id":"9f6f8e63da2013c5","type":"function","z":"bede72a39474db90","name":"Set Dates","func":"var p = msg.payload.split(\"<tr>\");\nmsg.payload = {};\n//msg.payload.data = p;\nmsg.payload.general = p[2].match(/([0-9]{2}\\/[0-9]{2}\\/20[0-9]{2})/g);\nmsg.payload.recycle = p[3].match(/([0-9]{2}\\/[0-9]{2}\\/20[0-9]{2})/g);\np = String(msg.payload.general[0]).split(\"/\");\nmsg.payload.general = p[2] + \"-\" + p[1] + \"-\" + p[0];\np = String(msg.payload.recycle[0]).split(\"/\");\nmsg.payload.recycle = p[2] + \"-\" + p[1] + \"-\" + p[0];\nvar d = new Date();\nvar d1 = Date.parse(msg.payload.general);\nd = new Date(d1);\nd.setHours(9);\nd.setMinutes(30);\nmsg.payload.general_date = d.toISOString();\nvar d2 = Date.parse(msg.payload.recycle);\nd = new Date(d2);\nd.setHours(9);\nd.setMinutes(30);\nmsg.payload.recycle_date = d.toISOString();\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":220,"y":240,"wires":[["1b164cb0e0d9ce5c","31aaad94e61a8666","486cdb30f80e3fe0"]]},{"id":"31aaad94e61a8666","type":"ha-sensor","z":"bede72a39474db90","name":"General Waste","entityConfig":"6c4ab0fdd17870b8","version":0,"state":"payload.general_date","stateType":"msg","attributes":[],"inputOverride":"allow","outputProperties":[],"x":140,"y":320,"wires":[[]]},{"id":"486cdb30f80e3fe0","type":"ha-sensor","z":"bede72a39474db90","name":"Recycling","entityConfig":"80f3b7e8ce4d9a2d","version":0,"state":"payload.recycle_date","stateType":"msg","attributes":[],"inputOverride":"allow","outputProperties":[],"x":120,"y":360,"wires":[[]]},{"id":"6c4ab0fdd17870b8","type":"ha-entity-config","server":"49949cfc.3312d4","deviceConfig":"","name":"sensor config for General Waste","version":6,"entityType":"sensor","haConfig":[{"property":"name","value":"Waste Collection General Waste"},{"property":"device_class","value":"timestamp"},{"property":"icon","value":""},{"property":"unit_of_measurement","value":""},{"property":"state_class","value":""},{"property":"last_reset","value":""}],"resend":true},{"id":"80f3b7e8ce4d9a2d","type":"ha-entity-config","server":"49949cfc.3312d4","deviceConfig":"","name":"sensor config for Recycling","version":6,"entityType":"sensor","haConfig":[{"property":"name","value":"Waste Collection Recycling"},{"property":"device_class","value":"timestamp"},{"property":"icon","value":""},{"property":"unit_of_measurement","value":""},{"property":"state_class","value":""},{"property":"last_reset","value":""}],"resend":true},{"id":"49949cfc.3312d4","type":"server","name":"Gaia","version":5,"addon":true,"rejectUnauthorizedCerts":true,"ha_boolean":"","connectionDelay":true,"cacheJson":false,"heartbeat":false,"heartbeatInterval":"60","areaSelector":"friendlyName","deviceSelector":"friendlyName","entitySelector":"friendlyName","statusSeparator":"at: ","statusYear":"hidden","statusMonth":"short","statusDay":"numeric","statusHourCycle":"h23","statusTimeFormat":"h:m","enableGlobalContextStore":true}]

I’m just using Regex stuff.
Specifically.

var p = msg.payload.split("<tr>");
msg.payload = {};
//msg.payload.data = p;
msg.payload.general = p[2].match(/([0-9]{2}\/[0-9]{2}\/20[0-9]{2})/g);
msg.payload.recycle = p[3].match(/([0-9]{2}\/[0-9]{2}\/20[0-9]{2})/g);
p = String(msg.payload.general[0]).split("/");
msg.payload.general = p[2] + "-" + p[1] + "-" + p[0];
p = String(msg.payload.recycle[0]).split("/");
msg.payload.recycle = p[2] + "-" + p[1] + "-" + p[0];
var d = new Date();
var d1 = Date.parse(msg.payload.general);
d = new Date(d1);
d.setHours(9);
d.setMinutes(30);
msg.payload.general_date = d.toISOString();
var d2 = Date.parse(msg.payload.recycle);
d = new Date(d2);
d.setHours(9);
d.setMinutes(30);
msg.payload.recycle_date = d.toISOString();
return msg;

Hello Borders resident!

Your script would take me a month to untangle and understand! Not enough big, easy words in it for me.

But I will try.

The only bit you need to change is the postcode and address. Everything else should work as is - and will update / create 2 sensors in your Home Assistant with the next date for each bin.

image

1 Like

Nice, I like that.

I’ve just realised my script sends a notify message whether the date matches or not so I need to test if there is a return from the Python script or not. Off to do some more Googling.

I see you are East of us. We are in Innerleithen.

1 Like

Yeah I’m in Greenlaw.

Here’s the automation that handles the sending the notification the DAY BEFORE the bin is due to be collected.

alias: Rubbish Reminder
description: ""
trigger:
  - platform: time
    at: "09:30:00"
condition: []
action:
  - choose:
      - conditions:
          - condition: template
            value_template: >-
              {%- set t =
              states('sensor.waste_collection_general_waste')|as_timestamp|float
              - 86400 -%}

              {%- set n = now() -%}

              {{ t|timestamp_custom("%Y%m%d") ==
              n|as_timestamp|timestamp_custom("%Y%m%d") }}
        sequence:
          - service: notify.tg_house_group
            data:
              message: The **General Waste** bin needs to be put out today.
              title: Rubbish Reminder
          - service: notify.mobile_app_pixel_6
            data:
              message: The Rubbish Bin needs to be put out today!
              title: Rubbish Reminder!
      - conditions:
          - condition: template
            value_template: >-
              {%- set t =
              states('sensor.waste_collection_recycling')|as_timestamp|float -
              86400 -%}

              {%- set n = now() -%}

              {{ t|timestamp_custom("%Y%m%d") ==
              n|as_timestamp|timestamp_custom("%Y%m%d") }}
        sequence:
          - service: notify.tg_house_group
            data:
              message: The **Recycling Bin** needs to be put out today
              title: Rubbish Reminder
          - service: notify.mobile_app_pixel_6
            data:
              message: The Recyling Bin needs to be put out today!
              title: Rubbish Reminder!
    default: []
mode: single