Screen scrape airport security times

Configuration
1

Hi,

I’m trying to scrape the security waiting times at the DUS airport. They are published here and show “Aktuelle Wartezeiten an Flugsteig
A: 3 MinB: 4 MinC: 1 Min”. The related CSS code looks as follows:

<div class="waiting-times__gates"><span 
class="gateTerminal">A:</span> <span 
class="waitingTimeText">1 Min</span><span 
class="gateTerminal">B:</span> <span 
class="waitingTimeText">6 Min</span><span 
class="gateTerminal">C:</span> <span 
class="waitingTimeText">0 Min</span> <!----></div>

Scrape is set up with a resource https://www.dus.com/de-de/fliegen/abflug as method GET and the rest of the options all empty. For the sensor I have tried “.waiting-times__gates .waitingTimeText:nth-of-type(2)” as selector, Index 0 and the rest all empty.
The debug log shows the site being loaded but the parsed value remains empty. Any idea of what I am doing wrong here?

Thanks,
Klayman

2

The information is coming from a json api call at https://www.dus.com/api/sitecore/flightapi/WaitingTimes
Not working directly, though, so you’ll likely have to reverse engineer the cookie that must be passed…

3

Thanks for your reply. I thought the scraper would get a web page just as a normal browser and then I could scrape the numbers from the page… So there is no way to get this done without using the API?

4

It is preferred to go through an API.
Anyway, scrapping won’t work, here, as the web page is dynamically build in JavaScript through API calls as well, and the HA scrapper only work for simple html pages.

5

Thanks. I have to give this up, far beyond my knowledge :wink:

6

maybe not just yet… Answer to self: The following curl command spits out the required data:

curl -X GET "https://www.dus.com/api/sitecore/flightapi/WaitingTimes?lang=de" -H "X-Requested-With:XMLHttpRequest"

The returned data then looks as follows:

{"data":[{"id":1,"name":"Sicherheitskontrolle A","waitingTime":1,"gateTerminal":"A","waitingTimeText":"1 Min"},{"id":2,"name":"Sicherheitskontrolle B","waitingTime":3,"gateTerminal":"B","waitingTimeText":"3 Min"},{"id":3,"name":"Sicherheitskontrolle C","waitingTime":3,"gateTerminal":"C","waitingTimeText":"3 Min"}],"serviceType":"Internal","hasError":false,"internalErrorMessage":"","errorCode":""}

Thanks to ChatGPT I was able to figure out how to add it to HomeAssistant. First create a REST Sensor:

sensor:
  - platform: rest
    name: "DUS Waiting Times Raw"
    resource: "https://www.dus.com/api/sitecore/flightapi/WaitingTimes?lang=de"
    headers:
      X-Requested-With: "XMLHttpRequest"
    scan_interval: 60
    value_template: "OK"  # Placeholder value
    json_attributes:
      - data

Then create 3 template sensors within the same (sensor) section:

  - platform: template
    sensors:
      terminal_a_waiting_time:
        friendly_name: "DUS Terminal A Waiting Time"
        unit_of_measurement: "min"
        value_template: >-
          {% set items = state_attr('sensor.dus_waiting_times_raw', 'data') | selectattr('gateTerminal', 'equalto', 'A') | list %}
          {{ items[0].waitingTime if items else 'unknown' }}

      terminal_b_waiting_time:
        friendly_name: "DUS Terminal B Waiting Time"
        unit_of_measurement: "min"
        value_template: >-
          {% set items = state_attr('sensor.dus_waiting_times_raw', 'data') | selectattr('gateTerminal', 'equalto', 'B') | list %}
          {{ items[0].waitingTime if items else 'unknown' }}

      terminal_c_waiting_time:
        friendly_name: "DUS Terminal C Waiting Time"
        unit_of_measurement: "min"
        value_template: >-
          {% set items = state_attr('sensor.dus_waiting_times_raw', 'data') | selectattr('gateTerminal', 'equalto', 'C') | list %}
          {{ items[0].waitingTime if items else 'unknown' }}

et voila - three new entities with the security waiting times for Düsseldorf airport security. Love it :smiley:

Regards,
Klayman