PostNL Integration

[ptnijssen (Peter Nijssen)]

You can contribute to the existing integration and update it; GitHub - iMicknl/python-postnl-api: Python wrapper for the PostNL API, a way to track packages using their online portal.

Wondering though how you passed the bot protection (if still in place).

[arjenbos]

The integration you are linking is just a wrapper around the PostNL API. Not a Home Assistant Integration. Or am I missing something? My question is about when you have the data.

EDIT: or, are you thinking of updating the wrapper so the existing lovelace will work (again)?

[arjenbos]

Let me share what I did.

1. I checked the web app and went trough the network activity logs. I noticed that they use oAuth to retrieve an access token for the GraphQL api. Nothing special here.
2. I decompiled the Android app. To see if they are using oAuth and GraphQL as well. They do. I copied the client_id and scope from the configuration.
3. I found out the redirect_uri in the mobile app is postnl://login. The scheme protocol (postnl://) is used to redirect a user back to the PostNL app after they filled in their credentials at https://login.postnl.nl.
4. The biggest challenge was to get back from https://login.postnl.nl to Home Assistant. I had a couple of ideas.
   1. I tried to create a scheme handler for Chrome and Firefox. But it turns out that there is a safelist for scheme handlers - HTML Standard.
   2. I tried to use https://selenium-python.readthedocs.io/ to use a headless browser. However, this is a bit against how oAuth should work and it can easily break if PostNL updates their login page.
   3. My current solution is a bit like my first solution: try to catch the postnl:// redirect. I created a Chrome extension (but this will also work in Firefox) to listen a webRequest event. When the event webRequest.onBeforeRedirect is fired, I replace postnl:// with the oAuth redirect_uri for HA. Now, HA will receive all the data it needs to store the Application Credentials. The extension is only 3 simple lines of JavaScript, nothing fancy at all. It’s the safest, less hacky method. It only needs a little help from a browser extension.
5. I looked at the GraphQL requests in the web app, and copied the query to validate if the access token is valid. It is!

[danieldotnl (Daniel)]

That sounds awesome! My vote (on this voting day) goes to starting with the calendar approach!

[danieldotnl (Daniel)]

Wait, now I’m less enthusiastic… Are you saying it requires one to install a browser extension? I guess it will not work from the companion app in that case?

[arjenbos]

The browser extension is only needed on install of the integration to retrieve the access and refresh token. After that it’s not needed anymore. It’s just oAuth.

If you want to learn about oAuth, then you can have a look here:

• Redirect URLs - OAuth 2.0 Simplified
• Access Tokens - OAuth 2.0 Simplified.

[danieldotnl (Daniel)]

Thanks, that’s better. I was thinking with my limited oauth knowledge that you normally provide the redirect_uri as a url parameter in the authentication request?

[arjenbos]

You need to provide the redirect_uri in the auth request, but that’s only when you are authorizing the client the first time. To retrieve a new access token you need to POST the refresh token (and some more parameters) to the token endpoint.

[ptnijssen (Peter Nijssen)]

This wrapper always has been the base for the integration The author of the wrapper is also the one that implemented it within Home Assistant in the first place,

[arjenbos]

Alright, but the wrapper you linked is not a Home Assistant Integration. When I’m done, then I can abstract API classes from the integration and maybe create a pull request to the wrapper.

I pushed my (very work in progress) project to Github. You can have a look here: GitHub - arjenbos/ha-postnl: Custom PostNL Home Assistant integration

[JelleKoster]

Hi Marius
I did a little googling, and it should be possible to use the e-mails as a trigger using the IMAP integration:

Using the IMAP integration to create a custom event from an email - HELP PLEASE!

Hi I want to use the Home Assistant IMAP integration to create a custom event to trigger an automation when my Gmail receives certain emails. I’ve been trying to follow the steps at IMAP - Home Assistant but am struggling a bit. Here’s the background: My home alarm does not have an official Home Assistant integration. I want my Home Assistant to monitor the status of my home alarm (which is either Fully Armed, Part Armed or Disarmed). My home alarm can send emails upon a change in state to …

Then it’s a matter of parsing the payload using templates - I guess this can be a struggle but should be achievable.

It would be great if this solution (when it’s made into a proper / HACS integration) can leverage the new 2023.11 to-do list functionality, that sounds like it’s appropriate to use dynamically for a constantly updating list of incoming parcels.

To-do list

[arjenbos]

@ptnijssen still work in progress, but I’m getting there.

[Mariusthvdb (Marius)]

do you think this would also go for regular mail, so letters etc, and not only packages which are delivered via the PostNL expedition.

when logged in into the app, we get a daily notification when mail is on the way. That is what I was looking for.
seems another PostNL service altogether?

[arjenbos]

I released the first “beta”. It’s more an alpha. So, if people would like to test, then please feel free to do so. Also, pull requests are of course welcome!

I didn’t add support for letters, because the old mobile API is dead. With a man in the middle attack I saw how PostNL retrieves the data for their mobile app. This is (again) another API. It will take some time to figure out what the best approach is.

Some other things I want to do:

• Less API requests.
  • It’s not needed to fetch all data for already delivered packages.
  • The track and trace details JSON api endpoint returns a lastObservation timestamp. Hopefully I can get the same data from the GraphQL API and determine if a call to the track and trace details endpoint is needed.
• Reduce the amount of data that GraphQL returns.
• Restructure the various API classes and use structs for packages.
• Some code clean up.
• Add tests.

GitHub - arjenbos/ha-postnl: Custom PostNL Home Assistant integration

[Kliko (Jasper)]

Hi Arjen,

Thanks for all your efforts in this. I do get the integration linked with my PostNL account however, it does not create any sensors. Here’s some debug logs. Hope this helps.

2023-11-23 22:29:12.578 DEBUG (MainThread) [custom_components.postnl.sensor] Update shipments
2023-11-23 22:29:12.579 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.postnl_delivery_2 fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 696, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 957, in async_device_update
    await self.async_update()
  File "/config/custom_components/postnl/sensor.py", line 98, in async_update
    await self.update_access_token()
  File "/config/custom_components/postnl/sensor.py", line 83, in update_access_token
    implementation = await async_get_config_entry_implementation(self.hass, self.entry)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 405, in async_get_config_entry_implementation
    raise ValueError("Implementation not available")
ValueError: Implementation not available

[arjenbos]

Thanks for sharing the logs! I created an issue on Github to track the issue.

ValueError: Implementation not available

[arjenbos]

I changed a couple of things in the integration (Use coordinator by arjenbos · Pull Request #2 · arjenbos/ha-postnl · GitHub). Could you fetch the latest beta version and test again? I would suggest to do a fresh install, just to be sure.

If it doesn’t work, then please message me in a DM how you implemented the integration.

[Kliko (Jasper)]

Yes! That did work! Great job. Thanks a lot Arjen

[Remco_van_Geel]

Works! but the whole weekend I tried end ended with blank screen because i didn’t saw the line that I must download the chrome extension hehe

[Kliko (Jasper)]

At my installation the integration worked through the whole weekend, but now the sensor is ‘unavailable’. See the logs below (btw: I replaced the actual numbers with 1s)

Deze fout is ontstaan door een aangepaste integratie.

Logger: custom_components.postnl.coordinator
Source: helpers/update_coordinator.py:290
Integration: PostNL (documentation, issues)
First occurred: 26 november 2023 om 18:12:09 (383 occurrences)
Last logged: 11:44:43

Unexpected error fetching PostNL data: '3SBOL1111111111'
Unexpected error fetching PostNL data: '3SBUSD11111111'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 290, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/postnl/coordinator.py", line 48, in _async_update_data
    colli = track_and_trace_details['colli'][shipment['barcode']]
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
KeyError: '3SBOL1111111111'