Garbage pickup date (mijnafvalwijzer.nl) custom_component

HI @xirixiz

Had a go creating a sensor for the day after tomorrow, sensor.trash_dat, which seems to work fine, and even better than my own template sensor, as it initialized immediately at startup.

I’ll post it here, in the full sensor.py, so you can see if it is alright. If you’re ok with it, feel free to add it to the CC.
Day after tomorrow is a steady sensor of use, since most of the time the trash is picked up in pairs per week. So one can see the full week of pickup in 1 glance.

"""
@ Authors     : Bram van Dartel
@ Date        : 10/12/2019
@ Description : Afvalwijzer Json/Scraper Sensor - It queries mijnafvalwijzer.nl or afvalstoffendienstkalender.nl.

sensor:
  - platform: afvalwijzer
    url: mijnafvalwijzer.nl (optional, default mijnafvalwijzer.nl)
    postcode: 1111AA
    huisnummer: 1
    toevoeging: A (optional)
    label_geen: 'Bla' (optional, default Geen)

23-02-2019 - Back to JSON release instead of scraper
23-02-2019 - Move scraper url, cleanup, and some minor doc fixes
24-02-2019 - Scraper debug log url fix
25-02-2019 - Update to new custom_sensor location
07-03-2019 - Make compatible for afvalstoffendienstkalender.nl as well
25-03-2019 - Remove Python 3.7.x f-strings, back to old format for beteer compatibility
26-04-2019 - Make compatible with hass 0.92
22-09-2019 - Add bs4 as a requirement in manifest.json (for hassio)
10-12-2019 - Fix whitespace bug

10-12-2019- Added a sensor for Day after tomorrow (sensor.trash_dat) @marijnthvdb
"""

VERSION = '3.0.11'

import logging
from datetime import date, datetime, timedelta

import bs4
import requests

import homeassistant.helpers.config_validation as cv
import voluptuous as vol
from homeassistant.components.sensor import PLATFORM_SCHEMA
from homeassistant.const import CONF_NAME
from homeassistant.helpers.entity import Entity
from homeassistant.util import Throttle

logger = logging.getLogger(__name__)

DEFAULT_NAME = 'afvalwijzer'
DOMAIN = 'afvalwijzer'
ICON = 'mdi:delete-empty'
SENSOR_PREFIX = 'trash_'

CONST_URL = 'url'
CONST_POSTCODE = 'postcode'
CONST_HUISNUMMER = 'huisnummer'
CONST_TOEVOEGING = 'toevoeging'
CONST_LABEL_NONE = 'label_geen'

SCAN_INTERVAL = timedelta(seconds=30)
MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=3600)

PLATFORM_SCHEMA = PLATFORM_SCHEMA.extend({
    vol.Optional(CONF_NAME, default=DEFAULT_NAME): cv.string,
    vol.Optional(CONST_URL, default="mijnafvalwijzer.nl"): cv.string,
    vol.Required(CONST_POSTCODE): cv.string,
    vol.Required(CONST_HUISNUMMER): cv.string,
    vol.Optional(CONST_TOEVOEGING, default=""): cv.string,
    vol.Optional(CONST_LABEL_NONE, default="Geen"): cv.string,
})


def setup_platform(hass, config, add_devices, discovery_info=None):
    """Setup the sensor platform."""
    # Setup JSON request (add sensor/devices)
    url = config.get(CONST_URL)
    postcode = config.get(CONST_POSTCODE)
    huisnummer = config.get(CONST_HUISNUMMER)
    toevoeging = config.get(CONST_TOEVOEGING)

    if None in (postcode, huisnummer):
        logger.error("Postcode or huisnummer not set!")

    url = ("https://json.{0}/?method=postcodecheck&postcode={1}&street=&huisnummer={2}&toevoeging={3}&platform=phone&langs=nl&").format(url,postcode,huisnummer,toevoeging)
    logger.debug("Json request url: {}".format(url))
    response = requests.get(url)

    if response.status_code != requests.codes.ok:
        logger.exception("Error doing API request")
    else:
        logger.debug("API request ok {}".format(response.status_code))

    json_obj = response.json()
    json_data = (json_obj['data']['ophaaldagen']['data'] + json_obj['data']['ophaaldagenNext']['data'])

    # Get unique trash shortname(s)
    uniqueTrashShortNames = []
    allTrashNames = ['firstdate', 'firstwastetype', 'today', 'tomorrow', 'dat', 'next']
    uniqueTrashShortNames.extend(allTrashNames)
    sensors = []

    for item in json_data:
        element = item["nameType"]
        if element not in uniqueTrashShortNames:
            uniqueTrashShortNames.append(element)

    logger.debug("uniqueTrashShortNames succesfully added: {}".format(uniqueTrashShortNames))

    data = (TrashCollectionSchedule(url, allTrashNames, config))

    for name in uniqueTrashShortNames:
        sensors.append(TrashCollectionSensor(name, data, config))
    add_devices(sensors)

    logger.debug("Object succesfully added as sensor(s): {}".format(sensors))


class TrashCollectionSensor(Entity):
    """Representation of a Sensor."""
    def __init__(self, name, data, config):
        """Initialize the sensor."""
        self._name = name
        self.data = data
        self.config = config
        self._state = self.config.get(CONST_LABEL_NONE)

    @property
    def name(self):
        """Return the name of the sensor."""
        return SENSOR_PREFIX + self._name

    @property
    def state(self):
        """Return the state of the sensor."""
        return self._state

    @property
    def icon(self):
        """Set the default sensor icon."""
        return ICON

    def update(self):
        """Fetch new state data for the sensor."""
        self.data.update()
        self._state = self.config.get(CONST_LABEL_NONE)

        for item in self.data.data:
            logger.debug("Update called for item: {}".format(item))
            if item['key'] == self._name:
                self._state = item['value'].strip()


class TrashCollectionSchedule(object):
    """Fetch new state data for the sensor."""
    def __init__(self, url, allTrashNames, config):
        """Fetch vars."""
        self._url = url
        self._allTrashNames = allTrashNames
        self.data = None
        self._config = config

    @Throttle(MIN_TIME_BETWEEN_UPDATES)
    def update(self):
        """Fetch new state data for the sensor."""
        response = requests.get(self._url)
        json_obj = response.json()
        json_data = (json_obj['data']['ophaaldagen']['data'] + json_obj['data']['ophaaldagenNext']['data'])

        today = datetime.today().strftime('%Y-%m-%d')
        dateConvert = datetime.strptime(today, '%Y-%m-%d') + timedelta(days=1)
        dateConvertDat = datetime.strptime(today, '%Y-%m-%d') + timedelta(days=2)
        tomorrow = datetime.strftime(dateConvert, '%Y-%m-%d')
        dat = datetime.strftime(dateConvertDat, '%Y-%m-%d')

        trashType = {}
        trashNext = {}
        trashToday = {}
        trashTomorrow = {}
        trashDat = {}
        multiTrashToday = []
        multiTrashTomorrow = []
        multiTrashDat = []
        trashSchedule = []

        # Some date count functions for next
        def d(s):
            [year, month, day] = map(int, s.split('-'))
            return date(year, month, day)

        def days(start, end):
            return (d(end) - d(start)).days

        # Collect upcoming trash pickup dates for unique trash
        for name in self._allTrashNames:
            for item in json_data:
                name = item["nameType"]
                dateConvert = datetime.strptime(item['date'], '%Y-%m-%d').strftime('%d-%m-%Y')

                if name not in trashType:
                    if item['date'] >= today:
                        trash = {}
                        trashType[name] = item["nameType"]
                        trash['key'] = item['nameType']
                        trash['value'] = dateConvert
                        trashSchedule.append(trash)

                    if item['date'] > today:
                        if len(trashNext) == 0:
                            trashType[name] = "next"
                            trashNext['key'] = "next"
                            trashNext['value'] = (days(today, item['date']))
                            trashSchedule.append(trashNext)

                    if item['date'] == today:
                        trashType[name] = "today"
                        trashToday['key'] = "today"
                        trashSchedule.append(trashToday)
                        multiTrashToday.append(item['nameType'])
                        if len(multiTrashToday) != 0:
                            trashToday['value'] = ', '.join(multiTrashToday)

                    if item['date'] == tomorrow:
                        trashType[name] = "tomorrow"
                        trashTomorrow['key'] = "tomorrow"
                        trashSchedule.append(trashTomorrow)
                        multiTrashTomorrow.append(item['nameType'])
                        if len(multiTrashTomorrow) != 0:
                            trashTomorrow['value'] = ', '.join(multiTrashTomorrow)

                    if item['date'] == dat:
                        trashType[name] = "dat"
                        trashDat['key'] = "dat"
                        trashSchedule.append(trashDat)
                        multiTrashDat.append(item['nameType'])
                        if len(multiTrashDat) != 0:
                            trashDat['value'] = ', '.join(multiTrashDat)

        # Setup scraper request
        url = self._config.get(CONST_URL)
        postcode = self._config.get(CONST_POSTCODE)
        huisnummer = self._config.get(CONST_HUISNUMMER)
        toevoeging = self._config.get(CONST_TOEVOEGING)
        scraper_url = ("https://www.{0}/nl/{1}/{2}/{3}").format(url, postcode, huisnummer, toevoeging)
        logger.debug("Scraper request url: {}".format(scraper_url))
        scraper_response = requests.get(scraper_url)

        if scraper_response.status_code != requests.codes.ok:
            logger.exception("Error doing scrape request")
        else:
            logger.debug("Scrape request ok {}".format(scraper_response.status_code))

        scraper_data = bs4.BeautifulSoup(scraper_response.text, "html.parser")

        # Append firstDate and firstWasteType
        trashFirstDate = {}
        trashFirstDate['key'] = 'firstdate'
        trashFirstDate['value'] = scraper_data.find('p', attrs={'class':'firstDate'}).text
        trashSchedule.append(trashFirstDate)
        logger.debug("Data succesfully added {}".format(trashFirstDate))

        firstWasteType = {}
        firstWasteType['key'] = 'firstwastetype'
        firstWasteType['value'] = scraper_data.find('p', attrs={'class':'firstWasteType'}).text
        trashSchedule.append(firstWasteType)
        logger.debug("Data succesfully added {}".format(firstWasteType))

        # Return collected data
        logger.debug("trashSchedule content {}".format(trashSchedule))

        self.data = trashSchedule

Ooops, sorry about that.

Fixed it in merge request #30

have one more ‘issue’ which is the component doesnt automatically update on sensor.date, meaning it can take up to an hour apparently to change the sensors accordingly.
This leads to some unexpected results:

55

trash_today still is ‘Geen’ while i fact it should be the plastic day. Because of that the other relevant sensors also are incorrect…

Ive tried updating the sensors manually with the homeassistant.update_entity service, but that doesnt workout.

Could we add some logic to the component, to have it update on the date change? Or is this simply a source issue? I mean could it be the api isn’t yet updated somehow and as a result of that the scraper shows what it is fed…

Hello all,

First let me start by saying: I LOVE what you guys did here with this!

Is this project still a thing and if so, where can the files/instruction be found now?
All the Github links I clicked lead to non-existing pages :frowning:

I am running Hass.io as VM on my Hyper-V machine (Hass.io On Hyper-V - Step By Step)

I get the same error for paper and pmd.
Is there a solution for it?

I’ve created the solution myself:

Icons can be found here: trash icons (afvalwijzer)

1 Like

Hello @beurdy,

Does this solve your issue (that error will stay though, to solve that you can edit trash.yaml).

I am trying to get this working, so I installed the Afvalwijzer via HACS.
Added the following to the configuration.yaml:

custom_updater:
component_urls:

Also didn’t find where to put my postal code and housenumber, but that might be in the entity itself (when adding it to lovelace?). But it’s not available after restarting HASS.

I got an error in the log:
Tue Jan 21 2020 08:25:34 GMT+0100 (Central European Standard Time)

Component error: custom_updater - Integration ‘custom_updater’ not found.

Any idea what I am doing wrong?

Hi
I tried getting this in. Copied all the files and installed the HACS Afvalwijzer.
Added the following to the configuration.yaml, but nothing is appearing.

sensor:

  • platform: afvalwijzer
    url: ‘mijnafvalwijzer.nl’
    postcode: ‘1111AA’
    huisnummer: ‘1’
    toevoeging: ‘’
    label_geen: ‘Geen’

what am I doing wrong?

Did you change your huisnummer and postcode to your personal values? What does the hass logfile show you? Please provide some more information, the problem description isn’t detailed enough.

(yes did change the postcode/housenumber, left it between quotes)

Might have to do with:

2020-01-23 15:34:28 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
File “/usr/local/lib/python3.7/site-packages/aiohttp/web_protocol.py”, line 418, in start
resp = await task
File “/usr/local/lib/python3.7/site-packages/aiohttp/web_app.py”, line 458, in _handle
resp = await handler(request)
File “/usr/local/lib/python3.7/site-packages/aiohttp/web_middlewares.py”, line 119, in impl
return await handler(request)
File “/usr/src/homeassistant/homeassistant/components/http/real_ip.py”, line 39, in real_ip_middleware
return await handler(request)
File “/usr/src/homeassistant/homeassistant/components/http/ban.py”, line 72, in ban_middleware
return await handler(request)
File “/usr/src/homeassistant/homeassistant/components/http/auth.py”, line 135, in auth_middleware
return await handler(request)
File “/usr/src/homeassistant/homeassistant/components/http/view.py”, line 123, in handle
result = await result
File “/usr/src/homeassistant/homeassistant/components/api/init.py”, line 355, in post
domain, service, data, True, self.context(request)
File “/usr/src/homeassistant/homeassistant/core.py”, line 1226, in async_call
await asyncio.shield(self._execute_service(handler, service_call))
File “/usr/src/homeassistant/homeassistant/core.py”, line 1251, in _execute_service
await handler.func(service_call)
File “/usr/src/homeassistant/homeassistant/components/hassio/init.py”, line 265, in async_handle_core_service
errors = await conf_util.async_check_ha_config_file(hass)
File “/usr/src/homeassistant/homeassistant/config.py”, line 805, in async_check_ha_config_file
res = await check_config.async_check_ha_config_file(hass)
File “/usr/src/homeassistant/homeassistant/helpers/check_config.py”, line 164, in async_check_ha_config_file
platform = p_integration.get_platform(domain)
File “/usr/src/homeassistant/homeassistant/loader.py”, line 265, in get_platform
f"{self.pkg_path}.{platform_name}"
File “/usr/local/lib/python3.7/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1006, in _gcd_import
File “”, line 983, in _find_and_load
File “”, line 967, in _find_and_load_unlocked
File “”, line 677, in _load_unlocked
File “”, line 724, in exec_module
File “”, line 860, in get_code
File “”, line 791, in source_to_code
File “”, line 219, in _call_with_frames_removed
File “/config/custom_components/afvalwijzer/sensor.py”, line 1

^
SyntaxError: invalid syntax

But I can’t fix it, there is no strange syntax or so?

Hi @Mariusthvdb,

Could you post the scripts/sensors needed for the list view and for the calendar? Scrolling through the topic with all the different views confuses me… :slight_smile:

not are what you really want. which ‘list-view’ ? also with calendar you mean what?
this is my view, maybe point to what you want?

I’m sorry, i thought it would show the post i replied.

The views in this post:

edit: the view in your last reply looks way better. what about the calendar?

For the record, a HA noob over here :wink:

Ok I see. no problem.
the calendar is the mijnafvalwijzer website :wink:

the list view has become a glance now, this is the Lovelace config I use ( note I changed the component to include a day-after-tomorrow sensor, trash_dat in the setup below):

type: custom:vertical-stack-in-card
cards:
  - type: markdown
#    style: |
#      ha-card {
#      background: url("/local/mijnafvalwijzer/saver_logo.png");
#      }

    content: >
      # <font color = {{states('sensor.trash_color')}}>Afvalwijzer</font>

      <ha-icon icon= {{'mdi:delete-alert' if states('sensor.trash_firstdate') == 'vandaag' else 'mdi:delete-outline'}}> </ha-icon> Vandaag, {{states('sensor.vandaag')}}: <font color= {{states('sensor.trash_color')}}>**{{states('sensor.trash_today')}}</font>**

      <ha-icon icon= {{states('sensor.trash_next_icon')}}> </ha-icon> Volgende afval-ophaal over **<font color={{states('sensor.trash_next_color')}}>{{states('sensor.trash_next')}}</font>** {{'dag' if states('sensor.trash_next') == '1' else 'dagen'}} op:

      <ha-icon icon= 'mdi:delete-alert'> </ha-icon> {{states('sensor.trash_firstdate')|capitalize}}: **<font color={{states('sensor.trash_next_color')}}>{{states('sensor.trash_firstwastetype')}}</font>**

  - type: horizontal-stack
    cards:
      - type: picture-entity
        style: |
          ha-card {
            border-radius: 0px;
            animation: {% if is_state('persistent_notification.trash_notification', 'notifying') and
                               states('sensor.trash_today') != 'Geen' %} blink 2s linear infinite;
                       {% else %} none
                       {% endif %}
          }
          @keyframes blink {
            100% {opacity: 0;}
            }

        entity: sensor.trash_today
        name: Vandaag
        show_state: false
        state_image:
          'gft': /local/mijnafvalwijzer/gft3.png
          'papier': /local/mijnafvalwijzer/papier3.png
          'restafval': /local/mijnafvalwijzer/restafval3.png
          'plastic verpakkingsafval': /local/mijnafvalwijzer/plastic3.png
          'Geen': /local/mijnafvalwijzer/kliko_geen.png
      - type: picture-entity
        style: |
          ha-card {
            border-radius: 0px;
            animation: {% if is_state('persistent_notification.trash_notification', 'notifying') and
                               states('sensor.trash_tomorrow') != 'Geen' %} blink 2s linear infinite;
                       {% else %} none
                       {% endif %}
          }
          @keyframes blink {
            100% {opacity: 0;}
            }
        entity: sensor.trash_tomorrow
        name: Morgen
        show_state: false
        state_image:
          'gft': /local/mijnafvalwijzer/gft3.png
          'papier': /local/mijnafvalwijzer/papier3.png
          'restafval': /local/mijnafvalwijzer/restafval3.png
          'plastic verpakkingsafval': /local/mijnafvalwijzer/plastic3.png
          'Geen': /local/mijnafvalwijzer/kliko_geen.png
      - type: picture-entity
        style: |
          ha-card {
            border-radius: 0px;
        entity: sensor.trash_dat
        name: Overmorgen
        show_state: false
        state_image:
          'gft': /local/mijnafvalwijzer/gft3.png
          'papier': /local/mijnafvalwijzer/papier3.png
          'restafval': /local/mijnafvalwijzer/restafval3.png
          'plastic verpakkingsafval': /local/mijnafvalwijzer/plastic3.png
          'Geen': /local/mijnafvalwijzer/kliko_geen.png

  - type: glance
    entities:
     - entity: sensor.gft_formatted
       name: GFT
     - entity: sensor.papier_formatted
       name: Papier
     - entity: sensor.plastic_verpakkingsafval_formatted
       name: Plastic
     - entity: sensor.restafval_formatted
       name: Restafval

  - type: divider

  - type: entities
#    style: |
#      ha-card {
#      background: url("/local/mijnafvalwijzer/background.png");
#      background-size: auto;
#      }
    entities:
     - entity: automation.afval_vandaag
       name: Notify vandaag
     - entity: automation.afval_morgen
       name: Notify morgen
     - entity: input_boolean.trash_reminder
     - entity: input_boolean.trash_outside

#      - type: glance
#        entities:
#         - entity: sensor.afval_vandaag
#           name: Vandaag
#         - entity: sensor.trash_tomorrow
#           name: Morgen
#         - entity: sensor.afval_overmorgen
#           name: Overmorgen

#  - type: entities
#    title: 'Afvalwijzer ophaaldata'
#    show_header_toggle: false
#    entities:
#      - sensor.trash_firstdate
#      - sensor.trash_firstwastetype
#      - sensor.trash_next
##      - sensor.trash_today
#      - sensor.afval_vandaag
#      - sensor.trash_tomorrow
#      - sensor.afval_overmorgen
#      - sensor.restafval
#      - sensor.gft
#      - sensor.plastic_verpakkingsafval
#      - sensor.papier
#      - automation.afval_vandaag
#      - automation.afval_morgen

Thanks!! I will configure it later :slight_smile:

edit: stupid about the calendar… haha

Short question:

For some reason I seem to be missing an PMD-sensor. What’s the best way to recreate that?

@xirixiz @Mariusthvdb the names of ‘plastic, metalen en drankkartons’ and ‘papier en karton’ are too long for my custom view. Is it possible to shorten those names?

Sure, as you can template anything you want :wink:

That’s what I did if you check my glance setup