Bash scripting? Torquay refuse collection sensor

[ajoyce (Al)]

I’m trying to extract values from my council’s website to get my next bin collection date.

The website is here: http://echo2.ectrecycling.co.uk/EchoServiceLookup/lookup.aspx?cid=33

I’ve used the developer tools in Chrome to find all the details of the POST request that this input box generates, including the body and headers - the POST request then returns HTML to the page containing the results and upcoming collection dates.

So far, I’ve managed to recreate this POST request successfully in an online REST client, and now I’ve converted it into cURL and been able to run it successfully in Terminal.

I’m wondering what sort of workflow would be best to extract the dates? I don’t think using the REST sensor would work because the returned value would exceed the 255 character limit, and because it’s HTML and not JSON I couldn’t save it into attributes with json_attributes.

Should I be going down the bash script route? Maybe using grep? I don’t have a huge amount of experience with bash scripts so if anyone could spare a little knowledge I’d be hugely appreciative!

*edit: should add, running Hass.io on RPI 3 B+ from Hass.io installer

[ajoyce (Al)]

Right. Now I feel very stupid.

So far I have managed to make a shell_command to send the POST request and even do some grep-ing to being parsing it.

Shell_command works absolutely fine - it’s just a curl command with grep - but I think I need to run it as a separate script to parse the data further (just using grep isn’t going to isolate the data I need sufficiently).

As mentioned, I’m running Hass.io on RPI 3B+ using the installation instructions at https://www.home-assistant.io/hassio/installation/

I have tried making a basic test.sh in /config like this:

#!/bin/bash
curl -XPOST -H 'Host: echo2.ectrecycling.co.uk' -H 'Connection: keep-alive' -H 'Content-Length: 496' -H 'Cache-Control: max-age=0' -H 'Origin: http://echo2.ectrecycling.co.uk' -H 'Upgrade-Insecure-Requests: 1' -H 'DNT: 1' -H 'Content-Type: application/x-www-form-urlencoded' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'Referer: http://echo2.ectrecycling.co.uk/EchoServiceLookup/lookup.aspx?cid=33' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-GB,en;q=0.9' -H 'Cookie: ASP.NET_SessionId=vpjai445psbfaaqdan4bq245' -d '__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUJNDg4NzU1NDU0D2QWAgIDD2QWCgIDDw8WAh4EVGV4dAUMQWxsIFNlcnZpY2VzZGQCBw8PZBYCHgdvbmZvY3VzBQ10aGlzLnNlbGVjdCgpZAIPDw8WAh8AZWRkAhEPPCsACwEADxYCHgdWaXNpYmxlaGRkAhMPPCsACwEADxYCHwJoZGRk3yoZMnyDgUF0FVLcP9fq%2FlXhjxY%3D&__EVENTVALIDATION=%2FwEWBQK2p8XABQKTitn8DAL7nPiqBwLhlb2gAgKHyYPoA4Z6zV37ftrM65ck3S9F854K%2BH9N&txtHouseNum=HOUSENUM&txtStreet=STREET&txtPostcode=POSTCODE&btnGetServices=Find+Address&hiddenInputToUpdateATBuffer_CommonToolkitScripts=1' 'http://echo2.ectrecycling.co.uk/EchoServiceLookup/lookup.aspx?cid=33' | grep 'Household'

And in configuration.yaml:

sensor:
  - platform: command_line
    name: test
    command: "/config/test.sh"

But in the logger I’m getting:

2019-02-20 10:55:05 INFO (SyncWorker_6) [homeassistant.components.sensor.command_line] Running command: /config/test.sh
2019-02-20 10:55:05 ERROR (SyncWorker_6) [homeassistant.components.sensor.command_line] Command failed: /config/test.sh

Meanwhile, my shell_command:

shell_command:
  test: >-
    curl -XPOST -H 'Host: echo2.ectrecycling.co.uk' -H 'Connection: keep-alive' -H 'Content-Length: 496' -H 'Cache-Control: max-age=0' -H 'Origin: http://echo2.ectrecycling.co.uk' -H 'Upgrade-Insecure-Requests: 1' -H 'DNT: 1' -H 'Content-Type: application/x-www-form-urlencoded' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'Referer: http://echo2.ectrecycling.co.uk/EchoServiceLookup/lookup.aspx?cid=33' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-GB,en;q=0.9' -H 'Cookie: ASP.NET_SessionId=vpjai445psbfaaqdan4bq245' -d '__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUJNDg4NzU1NDU0D2QWAgIDD2QWCgIDDw8WAh4EVGV4dAUMQWxsIFNlcnZpY2VzZGQCBw8PZBYCHgdvbmZvY3VzBQ10aGlzLnNlbGVjdCgpZAIPDw8WAh8AZWRkAhEPPCsACwEADxYCHgdWaXNpYmxlaGRkAhMPPCsACwEADxYCHwJoZGRk3yoZMnyDgUF0FVLcP9fq%2FlXhjxY%3D&__EVENTVALIDATION=%2FwEWBQK2p8XABQKTitn8DAL7nPiqBwLhlb2gAgKHyYPoA4Z6zV37ftrM65ck3S9F854K%2BH9N&txtHouseNum=HOUSENUM&txtStreet=STREET&txtPostcode=POSTCODE&btnGetServices=Find+Address&hiddenInputToUpdateATBuffer_CommonToolkitScripts=1' 'http://echo2.ectrecycling.co.uk/EchoServiceLookup/lookup.aspx?cid=33' | grep 'Household'

…works absolutely fine.

Is this a permissions issue? Have I not stated the correct filepath? Is my scripting rubbish? The error logs aren’t particularly helpful.

FULL DISCLOSURE: Whilst I can do stuff like SSH-ing in and certain amounts of debugging, I’m not very knowledgeable when it comes to understanding how exactly Hass.io is installed and run, standard permissions, docker environments and syntax for commands - so I apologise if I’m being stupid.

(P.S. hoping @123 sees this as I know they can point me in the right direction!)

EDIT: have also tried using #!/usr/bin/env bash in the script and command: "~/config/test.sh" in the sensor but no dice

[ajoyce (Al)]

A workaround that seems to function well:

 - platform: command_line
    name: Recycling
    value_template: '{{ value.split("</td><td>")[2]|replace("</td>","") }}'
    scan_interval: 600
    command: >-
      curl -XPOST -H 'Host: echo2.ectrecycling.co.uk' -H 'Connection: keep-alive' -H 'Content-Length: 496' -H 'Cache-Control: max-age=0' -H 'Origin: http://echo2.ectrecycling.co.uk' -H 'Upgrade-Insecure-Requests: 1' -H 'DNT: 1' -H 'Content-Type: application/x-www-form-urlencoded' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'Referer: http://echo2.ectrecycling.co.uk/EchoServiceLookup/lookup.aspx?cid=33' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-GB,en;q=0.9' -H 'Cookie: ASP.NET_SessionId=vpjai445psbfaaqdan4bq245' -d '__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=%2FwEPDwUJNDg4NzU1NDU0D2QWAgIDD2QWCgIDDw8WAh4EVGV4dAUMQWxsIFNlcnZpY2VzZGQCBw8PZBYCHgdvbmZvY3VzBQ10aGlzLnNlbGVjdCgpZAIPDw8WAh8AZWRkAhEPPCsACwEADxYCHgdWaXNpYmxlaGRkAhMPPCsACwEADxYCHwJoZGRk3yoZMnyDgUF0FVLcP9fq%2FlXhjxY%3D&__EVENTVALIDATION=%2FwEWBQK2p8XABQKTitn8DAL7nPiqBwLhlb2gAgKHyYPoA4Z6zV37ftrM65ck3S9F854K%2BH9N&txtHouseNum=[YOUR HOUSE NUMBER]&txtStreet=[YOUR STREET]&txtPostcode=[YOUR POSTCODE]&btnGetServices=Find+Address&hiddenInputToUpdateATBuffer_CommonToolkitScripts=1' 'http://echo2.ectrecycling.co.uk/EchoServiceLookup/lookup.aspx?cid=33' | grep 'Recycling'

The command includes a grep to prevent the value being too big, then the value_template cuts the preserved line down to the right section with some split and replace functions.

Hope this helps others who are stuck with similar parsing issues, or live in the Torbay area and want to know when Tor2 should be collecting waste next!

I would still be indebted to anyone who would be able to help me with getting shell scripts to work on a standard Hass.io installation - I just can’t understand why I’m getting a ‘command failed’ error with no explanation and I expect I will want to use bash scripts in future. Many thanks if you can advise what I’m doing wrong!

[123 (Taras)]

You rang?

Sorry but I’m not well versed with Hass.io other than knowing there are certain things done differently (or not at all) compared to a vanilla ‘venv’ installation of Home Assistant.

I searched the forum for "command script hass.io" and got a few matches. Perhaps there’s something helpful here:

Run bash script hosted on HASSIO via shell command - Error

Hi, im trying to get a shell command to run which is hosten on HASSIO with HASSIO. I tried several different things that i found on this forum… none of them work. The command itself works perfectly fine if i SSH into my HASSIO manually. Does HASSIO use a different user than root on PI3? set_alexa: ‘ssh -i /config/.ssh/id_rsa root@HASSIO /share/alexa_remote_control.sh -d Wohnzimmer -e vol:10’ set_alexa: ‘/share/alexa_remote_control.sh -d Wohnzimmer -e vol:10’ set_alexa: /share/alexa_remote…

Hass/hassio shell command not working

I have no idea if this problem is hassio or hass related so i try to get help here. I use hassio on raspberry pi zero. I have tried different shell commands to call a batch file shell_command: server_oben_runterfahren: /bin/bash /data/shutdown_server.sh server_oben_runterfahren_2: '/bin/bash /data/shutdown_server.sh' server_oben_runterfahren_3: 'data/shutdown_server.sh' Here is the batch file content #!/bin/bash #ssh -i /data/.ssh/id_rsa -o StrictHostKeyChecking=no [email protected] …

[ajoyce (Al)]

I had seen the first link and exhausted options there but had not seen the second one you posted - I will give that a go this evening!