Looking at the logs can the Selenium instance access the internet?
Is there a simple test you can fire at it?
Thank you for adding South GLoucestershire to the list!
Definitely a more elegant solution to my rest sensor and basic dashboard card!
Happy New Year!
1 Like
Am I right in saying that this integration is a workaround for council boroughs that don’t have a proper api to extract waste collection data for public use?
I’m just trying to find the differences between this and waste_collection integration.
Yes.
Most councils don’t have an API
1 Like
Looks like for some reason I needed to run the standalone-chrome docker in Privileged mode, but that worked. I’ve pulled together a very short simple guide to adding a unraid docker for standlalone-chrome, might help somebody out including it in the wiki somewhere? Thanks for your help.
1 Like
FYI: @dp247 / @olliecullimore,
Thank you for bringing attention to another integration owned by @mampfes, detailed in this forum post: Waste Collection Schedule Framework. This integration appears to offer more global support, extending to 163 UK councils as of now, with York being the first one added on Jan 16, 2022.
This integration adopts a similar approach for several UK councils, utilizing requests and Beautiful Soup for scraping council data and resorting to APIs where accessible. You can find the GitHub repository here: mampfes/hacs_waste_collection_schedule.
It’s common to have multiple integrations performing identical tasks, and the choice of which to use is entirely yours.
Our project has become community-driven and published under the MIT license, benefiting from numerous contributions from individuals within this thread, its no easy task to stay on top of keeping the councils working and adding new ons.
It’s worth noting that any scraping method employed by this post’s integration or Steffen’s integration is susceptible to changes in council websites/layouts, etc.
There has been a recent development by Github user dt215git, facilitating the parsing of JSON files created by the Python part of this integration’s work. Details can be found here: JSON Parsing Development.
@srk23 Nice write-up - once this PR feat: #525 Adding API Server and Docker build by robbrad · Pull Request #527 · robbrad/UKBinCollectionData · GitHub is merged it may simplify your approach slightly
It’s basically an additional Docker server that provides an API instead of the need to save a file
Unfortunately whilst this initially works (Chelmsford) after the selenium fix, the entities become unavailable after 12hrs. Would you be able to test on your setup as maybe its something to do with my setup (ie my standalone-chrome) rather than the plugin itself. Out of interest how often does the plugin poll the council server for changes / updates?
I think it could have something to do with this : Performance issue
I just made a change fix: Update ChelmsfordCityCouncil.py by robbrad · Pull Request #537 · robbrad/UKBinCollectionData · GitHub
Which should bump the load time up to 120 seconds
Hmm the issue could be this
We have a 30s timeout on the custom component - let me bump that to 60s
Looks like you’ve solved it. Many thanks. Out of interest what was the issue, the council site taking longer than the standard timeout you used?
Have you updated to the latest version?
I released a version this morning that fixes the Selenium grid getting blocked by previous sessions running
Glad it’s working for you
I hadn’t no, but just have. (de80fd2 > 49c5879). Appreciated.
Hello, running HA on an Rpi, have added this from HACS repository but when I go to integrations and search for ‘UK Bin’ nothing comes up. I have other HACS repos that are working correctly, wonder if someone could point out what the issue might be? Thanks!
Sorry to say this, but its stopped working again seemingly after updating to the latest version (de80fd2 > 49c5879). Let me know if there is any evidence or info I can help with.
Can you try the latest release : 9856ec9 [0.56.11]
Remove the device completely - upgrade via hacs, restart home assistant
In the intrgration turn on debugging, then open your selenium console at http://<ip_address>:4444
When you add the council untick headless mode (when you run normailly you want this ticketed) - add the council
Click sessions in the selenium UI - observe the session (password is “secret”)
This will show the live scrape of the council
In your HA log - check the values or errors - debug will show all the args used for the council
Let me know how that testing goes
Followed the guidance above. When I clicked on the selenium UI, observed the session - clicked on it used password, it opened a black window in middle of the page for couple of seconds (remained) blank, then the session closed. Back in HA the entities were added successfully.
Only entry in the HA logs was:
Setup of sensor platform uk_bin_collection is taking over 10 seconds.
6:44:41 PM – (WARNING) Sensor
Gather its now a matter of waiting 12/24hours to see if it refreshes (or is there a way to force this earlier?)
HA logs this morning:-
Timeout fetching Home data
6:45:42 AM – (ERROR) UK Bin Collection Data (custom integration)
Logger: custom_components.uk_bin_collection.sensor
Source: helpers/update_coordinator.py:306
Integration: UK Bin Collection Data (documentation, issues)
First occurred: 6:45:42 AM (1 occurrences)
Last logged: 6:45:42 AM
Full log extract:
2024-01-09 06:44:42.066 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] UKBinCollectionApp Updating
2024-01-09 06:45:42.067 ERROR (MainThread) [custom_components.uk_bin_collection.sensor] Timeout fetching Home data
2024-01-09 06:45:42.067 DEBUG (MainThread) [custom_components.uk_bin_collection.sensor] Finished fetching Home data in 60.001 seconds (success: False)
2024-01-09 06:45:42.067 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor Food waste, black bin, green box, card sack
2024-01-09 06:45:42.074 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-01-09
2024-01-09 06:45:42.074 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home Food waste, black bin, green box, card sack
2024-01-09 06:45:42.074 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: 0
2024-01-09 06:45:42.074 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: Today
2024-01-09 06:45:42.074 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Applying values for sensor Food waste, brown bin, paper sack, plastic and cartons bag
2024-01-09 06:45:42.075 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.next_collection: 2024-01-16
2024-01-09 06:45:42.075 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] Data Stored in self.name: Home Food waste, brown bin, paper sack, plastic and cartons bag
2024-01-09 06:45:42.075 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] _days: 7
2024-01-09 06:45:42.075 INFO (MainThread) [custom_components.uk_bin_collection.sensor] [UKBinCollection] State of the sensor: Next Week: Tuesday
i think mine has too, but only noticed today. not sure if its been triggerd by the HA update. or The project updates.
TypeError: create_webdriver() missing 1 required positional argument: 'user_agent'
2024-01-15 01:16:52.260 DEBUG (MainThread) [custom_components.uk_bin_collection.sensor] Finished fetching bin data in 0.070 seconds (success: False)
2024-01-15 01:16:52.260 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up uk_bin_collection platform for sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 300, in _async_refresh
self.data = await self._async_update_data()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/uk_bin_collection/sensor.py", line 133, in _async_update_data
data = await self.hass.async_add_executor_job(self.ukbcd.run)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 96, in run
return self.client_code(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 115, in client_code
return get_bin_data_class.template_method(address_url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 78, in template_method
bin_data_dict = self.parse_data(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uk_bin_collection/uk_bin_collection/councils/LeedsCityCouncil.py", line 40, in parse_data
driver = create_webdriver(web_driver, headless)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: create_webdriver() missing 1 required positional argument: 'user_agent'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 360, in _async_setup_platform
await asyncio.shield(task)
File "/config/custom_components/uk_bin_collection/sensor.py", line 75, in async_setup_entry
await coordinator.async_config_entry_first_refresh()
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 272, in async_config_entry_first_refresh
raise ex
homeassistant.exceptions.ConfigEntryNotReady: create_webdriver() missing 1 required positional argument: 'user_agent'