Bin / Waste Collection

johnyb · January 1, 2024, 1:11pm

Looks like the issues with Chelmsford has been fixed (by moving to selenium?) which is great. I now need to get the Selenium web driver URL bit working as per this…
“If your see a “URL of the remote Selenium web driver to use” field when setting up your council, you’ll need to provide the URL to a web driver you’ve set up seperately such as [standalone-chrome]”

My setup is Unraid, with HA Core running in a docker.
So I’ve added another docker for standalone-chrome (host network) and point to this http://IPaddress:4444 in the council details. But no entities are created.
Looking at the logs for the standalone-chrome docker looks like its getting some engagement. Any ideas what I’m doing wrong? Thanks

Starting Selenium Grid Standalone…
Tracing is disabled
13:00:16.581 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
13:00:16.600 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
13:00:17.209 INFO [NodeOptions.getSessionFactories] - Detected 4 available processors
13:00:17.252 INFO [NodeOptions.report] - Adding chrome for {“browserName”: “chrome”,“browserVersion”: “120.0”,“goog:chromeOptions”: {“binary”: “\u002fusr\u002fbin\u002fgoogle-chrome”},“platformName”: “linux”,“se:noVncPort”: 7900,“se:vncEnabled”: true} 1 times
13:00:17.278 INFO [Node.] - Binding additional locator mechanisms: relative
13:00:17.295 INFO [GridModel.setAvailability] - Switching Node ad76dcd5-4e17-4256-a53a-790c8e8c9fb4 (uri: http://172.17.0.1:4444) from DOWN to UP
13:00:17.296 INFO [LocalDistributor.add] - Added node ad76dcd5-4e17-4256-a53a-790c8e8c9fb4 at http://172.17.0.1:4444. Health check every 120s
13:00:17.414 INFO [Standalone.execute] - Started Selenium Standalone 4.16.1 (revision 9b4c83354e): http://172.17.0.1:4444
13:00:24.295 INFO [LocalDistributor.newSession] - Session request received by the Distributor:
[Capabilities {browserName: chrome, goog:chromeOptions: {args: [–headless, --no-sandbox, --disable-gpu, --start-maximized, --disable-dev-shm-usage], excludeSwitches: [enable-logging], extensions: }, pageLoadStrategy: normal}]
13:00:24.814 INFO [LocalNode.newSession] - Session created by the Node. Id: 6c00f1360e7e4136fcc1a03091c5d04c, Caps: Capabilities {acceptInsecureCerts: false, browserName: chrome-headless-shell, browserVersion: 120.0.6099.109, chrome: {chromedriverVersion: 120.0.6099.109 (3419140ab66…, userDataDir: /tmp/.org.chromium.Chromium…}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:45741}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://172.17.0.1:4444/sessio…, se:cdpVersion: 120.0.6099.109, se:vnc: ws://172.17.0.1:4444/sessio…, se:vncEnabled: true, se:vncLocalAddress: ws://172.17.0.1:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
13:00:24.821 INFO [LocalDistributor.newSession] - Session created by the Distributor. Id: 6c00f1360e7e4136fcc1a03091c5d04c
Caps: Capabilities {acceptInsecureCerts: false, browserName: chrome-headless-shell, browserVersion: 120.0.6099.109, chrome: {chromedriverVersion: 120.0.6099.109 (3419140ab66…, userDataDir: /tmp/.org.chromium.Chromium…}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:45741}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), se:bidiEnabled: false, se:cdp: ws://172.17.0.1:4444/sessio…, se:cdpVersion: 120.0.6099.109, se:vnc: ws://172.17.0.1:4444/sessio…, se:vncEnabled: true, se:vncLocalAddress: ws://172.17.0.1:7900, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
13:05:47.289 INFO [LocalNode.stopTimedOutSession] - Session id 6c00f1360e7e4136fcc1a03091c5d04c timed out, stopping…
13:05:47.366 INFO [LocalSessionMap.lambda$new$0] - Deleted session from local Session Map, Id: 6c00f1360e7e4136fcc1a03091c5d04c
13:05:47.367 INFO [GridModel.release] - Releasing slot for session id 6c00f1360e7e4136fcc1a03091c5d04c
13:05:47.367 INFO [SessionSlot.stop] - Stopping session 6c00f1360e7e4136fcc1a03091c5d04c

Robbrad · January 1, 2024, 1:50pm

Have you entered the full address like this

    "house_number": "1 Celeborn Street, South Woodham Ferrers, Chelmsford, CM3 7AE"

github.com

robbrad/UKBinCollectionData/blob/dee8fb81db84d8f4b1044bd63615af3930e24952/uk_bin_collection/tests/input.json#L159


      
              "wiki_command_url_override": "https://my.charnwood.gov.uk/location?put=cbcXXXXXXXX&rememberme=0&redirect=%2F",
              "wiki_name": "Charnwood Borough Council",
              "wiki_note": "Replace XXXXXXXX with UPRN keeping \"cbc\" before it."
          },
          "ChelmsfordCityCouncil": {
              "url": "https://www.chelmsford.gov.uk/myhome/",
              "wiki_name": "Chelmsford City Council",
              "wiki_note": "Follow the instructions [here](https://www.chelmsford.gov.uk/myhome/) until you get the page listing your \"Address\" then copy the entire address text and use that in the house number field.",
              "web_driver": "http://selenium:4444",
              "postcode": "CM3 7AE",
              "house_number": "1 Celeborn Street, South Woodham Ferrers, Chelmsford, CM3 7AE"
          },
          "CheshireEastCouncil": {
              "url": "https://online.cheshireeast.gov.uk/MyCollectionDay/SearchByAjax/GetBartecJobList?uprn=100012791226&onelineaddress=3%20COBBLERS%20YARD,%20SK9%207DZ&_=1689413260149",
              "wiki_command_url_override": "https://online.cheshireeast.gov.uk/MyCollectionDay/SearchByAjax/GetBartecJobList?uprn=XXXXXXXX&onelineaddress=XXXXXXXX&_=1689413260149",
              "wiki_name": "Cheshire East Council",
              "wiki_note": "Both the UPRN and a one-line address are passed in the URL, which needs to be wrapped in double quotes. The one-line address is made up of the house number, street name and postcode.\nUse the form [here](https://online.cheshireeast.gov.uk/mycollectionday/) to find them, then take the first line and post code and replace all spaces with `%20`."
          },
          "ConwyCountyBorough": {
              "postcode": "LL30 2DF",
              "uprn": "100100429249",

johnyb · January 1, 2024, 5:22pm

Initially yes sorry I had only used House Number in that field, but corrected this and still have same issue

Robbrad · January 1, 2024, 7:56pm

Looking at the logs can the Selenium instance access the internet?

Is there a simple test you can fire at it?

ChrisThomas · January 2, 2024, 11:06am

Thank you for adding South GLoucestershire to the list!

Definitely a more elegant solution to my rest sensor and basic dashboard card!

Happy New Year!

hazio · January 2, 2024, 12:35pm

Am I right in saying that this integration is a workaround for council boroughs that don’t have a proper api to extract waste collection data for public use?

I’m just trying to find the differences between this and waste_collection integration.

Robbrad · January 2, 2024, 12:48pm

Yes.

Most councils don’t have an API

johnyb · January 2, 2024, 4:18pm

Looks like for some reason I needed to run the standalone-chrome docker in Privileged mode, but that worked. I’ve pulled together a very short simple guide to adding a unraid docker for standlalone-chrome, might help somebody out including it in the wiki somewhere? Thanks for your help.

Robbrad · January 2, 2024, 10:07pm

FYI: @dp247 / @olliecullimore,

Thank you for bringing attention to another integration owned by @mampfes, detailed in this forum post: Waste Collection Schedule Framework. This integration appears to offer more global support, extending to 163 UK councils as of now, with York being the first one added on Jan 16, 2022.

This integration adopts a similar approach for several UK councils, utilizing requests and Beautiful Soup for scraping council data and resorting to APIs where accessible. You can find the GitHub repository here: mampfes/hacs_waste_collection_schedule.

It’s common to have multiple integrations performing identical tasks, and the choice of which to use is entirely yours.

Our project has become community-driven and published under the MIT license, benefiting from numerous contributions from individuals within this thread, its no easy task to stay on top of keeping the councils working and adding new ons.

It’s worth noting that any scraping method employed by this post’s integration or Steffen’s integration is susceptible to changes in council websites/layouts, etc.

There has been a recent development by Github user dt215git, facilitating the parsing of JSON files created by the Python part of this integration’s work. Details can be found here: JSON Parsing Development.

Robbrad · January 2, 2024, 10:54pm

Would you like to add a PR to the repo? or a wiki note? - thank you @johnyb

Robbrad · January 4, 2024, 12:27am

@srk23 Nice write-up - once this PR feat: #525 Adding API Server and Docker build by robbrad · Pull Request #527 · robbrad/UKBinCollectionData · GitHub is merged it may simplify your approach slightly

It’s basically an additional Docker server that provides an API instead of the need to save a file

johnyb · January 5, 2024, 10:18am

Unfortunately whilst this initially works (Chelmsford) after the selenium fix, the entities become unavailable after 12hrs. Would you be able to test on your setup as maybe its something to do with my setup (ie my standalone-chrome) rather than the plugin itself. Out of interest how often does the plugin poll the council server for changes / updates?

Robbrad · January 5, 2024, 10:56am

Every 12h : https://github.com/robbrad/UKBinCollectionData/blob/c9bcbe4cbea04e7c6dd7968b6d4e0c7e382f1f14/custom_components/uk_bin_collection/sensor.py#L112

I think it could have something to do with this : Performance issue

I just made a change fix: Update ChelmsfordCityCouncil.py by robbrad · Pull Request #537 · robbrad/UKBinCollectionData · GitHub

Which should bump the load time up to 120 seconds

Hmm the issue could be this

github.com

robbrad/UKBinCollectionData/blob/b6bd1308d51477dccc42f6254341a36362e3a965/custom_components/uk_bin_collection/sensor.py#L119


      
                  hass,
                  _LOGGER,
                  name="UK Bin Collection Data",
                  update_interval=timedelta(hours=12),
              )
              _LOGGER.info(f"{LOG_PREFIX} UKBinCollectionApp Init")
              self.ukbcd = ukbcd
              self.name = name
          
          async def _async_update_data(self):
              async with async_timeout.timeout(30) as cm:
                  _LOGGER.info(f"{LOG_PREFIX} UKBinCollectionApp Updating")
          
                  data = await self.hass.async_add_executor_job(self.ukbcd.run)
          
                  _LOGGER.info(f"{LOG_PREFIX} UKBinCollectionApp: {data}")
          
              if cm.expired:
                  _LOGGER.warning(f"{LOG_PREFIX} UKBinCollectionApp timeout expired during run")
                  
              return get_latest_collection_info(json.loads(data))

We have a 30s timeout on the custom component - let me bump that to 60s

Robbrad · January 5, 2024, 11:21am

@johnyb - ok 60s isnt enough

Let me investigate more and get back to you

johnyb · January 7, 2024, 10:45am

Looks like you’ve solved it. Many thanks. Out of interest what was the issue, the council site taking longer than the standard timeout you used?

Robbrad · January 7, 2024, 12:45pm

Have you updated to the latest version?

I released a version this morning that fixes the Selenium grid getting blocked by previous sessions running

Glad it’s working for you

johnyb · January 7, 2024, 1:39pm

I hadn’t no, but just have. (de80fd2 > 49c5879). Appreciated.

Stuart_Reid · January 7, 2024, 3:26pm

Hello, running HA on an Rpi, have added this from HACS repository but when I go to integrations and search for ‘UK Bin’ nothing comes up. I have other HACS repos that are working correctly, wonder if someone could point out what the issue might be? Thanks!

johnyb · January 8, 2024, 10:19am

Sorry to say this, but its stopped working again seemingly after updating to the latest version (de80fd2 > 49c5879). Let me know if there is any evidence or info I can help with.

Robbrad · January 8, 2024, 11:07am

Can you try the latest release : 9856ec9 [0.56.11]

Remove the device completely - upgrade via hacs, restart home assistant

In the intrgration turn on debugging, then open your selenium console at http://<ip_address>:4444

When you add the council untick headless mode (when you run normailly you want this ticketed) - add the council

Click sessions in the selenium UI - observe the session (password is “secret”)

This will show the live scrape of the council

In your HA log - check the values or errors - debug will show all the args used for the council

Let me know how that testing goes