How to extract data from JSON (again)

I thought I was getting the hang of JSON extraction. Obviously not so much…

I’m trying to use the Transport for London api but it is returning data (excuse me if my terminology is wrong here) in an array with no top level name.

Some elements within that array are also arrays, again with no top level name

Rather than paste the entire api call response, here is a simplified example of what I mean showing some data for two lines each with a different number of ‘lineStatuses’.

It is all the ‘lineStatuses’ data that I am primarily interested in (at the moment).

First my sensor so far which just counts the disrupted lines for the state (with my failed attempts and capturing attributes removed :wink:
(The resource url will work for anyone if you care to get all the data…)

  - platform: rest
    resource: https://api.tfl.gov.uk/Line/Mode/dlr,elizabeth-line,overground,tube/Status
    name: TfL All Lines Status
    value_template: >
      {% if value_json is defined %}
        {% set ns = namespace (disrupted_line_count = 0) %}
        {% for line in value_json %}
          {% if line.lineStatuses[0].statusSeverityDescription != 'Good Service' %}
            {% set ns.disrupted_line_count = ns.disrupted_line_count + 1 %}
          {% endif %}
        {% endfor %}

        {{ ns.disrupted_line_count }}
      {% else %}
        unavailable
      {% endif %}


[
	{
		"$type": "Tfl.Api.Presentation.Entities.Line, Tfl.Api.Presentation.Entities",
		"id": "london-overground",
		"name": "London Overground",
		"modeName": "overground",
		"lineStatuses": [
			{
				"$type": "Tfl.Api.Presentation.Entities.LineStatus, Tfl.Api.Presentation.Entities",
				"lineId": "london-overground",
				"statusSeverityDescription": "Severe Delays",
				"reason": "No service between Hackney Downs and Walthamstow Central. SEVERE DELAYS between Walthamstow Central and Chingford, while we make urgent repairs to the track at Clapton. GOOD SERVICE on all other London Overground routes. "
			},
			{
				"$type": "Tfl.Api.Presentation.Entities.LineStatus, Tfl.Api.Presentation.Entities",
				"lineId": "london-overground",
				"statusSeverityDescription": "Part Suspended",
				"reason": "No service between Hackney Downs and Walthamstow Central. SEVERE DELAYS between Walthamstow Central and Chingford, while we make urgent repairs to the track at Clapton. GOOD SERVICE on all other London Overground routes. "
			}
		]
	},
	{
		"$type": "Tfl.Api.Presentation.Entities.Line, Tfl.Api.Presentation.Entities",
		"id": "piccadilly",
		"name": "Piccadilly",
		"modeName": "tube",
		"lineStatuses": [
			{
				"$type": "Tfl.Api.Presentation.Entities.LineStatus, Tfl.Api.Presentation.Entities",
				"lineId": "piccadilly",
				"statusSeverityDescription": "Severe Delays",
				"reason": "Piccadilly Line: Severe delays westbound and MINOR DELAYS eastbound due to an earlier fire alert at Acton Town. Tickets are being accepted on London Buses. "
			}
		]
	}
]

Answer (again (not really just beeing cheeky))

value_template: >
  {{ value_json[0].lineStatuses | selectattr('statusSeverityDescription', 'eq', 'Good Service') | list | count }}

if you want to select via the line name…

value_template
  {% set line = value_json | selectattr('name', 'eq', 'London Overground') | first | default(none) %}
  {% if line %}
    {{ line | selectattr('statusSeverityDescription', 'eq', 'Good Service') | list | count }}
  {% else %}
    unavailable
  {% endif %}

Thanks! Not as quick as yesterday though :wink:

That is really helpful, apart from anything it reminds me that there is now a way to do things without namespaces and for loops…

But…

doesn’t that only return the first element in the array? It seems to when I test it.

Both your examples return 0
There are currently five lines with disruptions including London Overground (My for loop returns 5).

What I’m aiming for is a single sensor with the state being the total number of lines with disruptions (not the total number of disruptions as that can be more as each line can have several).

I want an attribute for each line. Each line is an array of disruptions.

I can do all of that programmatically but I can’t see anyway to get all the data into HA so that I can manipulate it. A separate sensor for each line would be ok, it doesn’t need to be one sensor but I still don’t see how to do that.


As I said, if you want to try it and get actual data (and I’m not suggesting you should) the rest sensor I posted uses a public api.

  - platform: rest
    resource: https://api.tfl.gov.uk/Line/Mode/dlr,elizabeth-line,overground,tube/Status
    name: TfL All Lines Status
    value_template: >
      {% if value_json is defined %}
        {% set ns = namespace (disrupted_line_count = 0) %}
        {% for line in value_json %}
          {% if line.lineStatuses[0].statusSeverityDescription != 'Good Service' %}
            {% set ns.disrupted_line_count = ns.disrupted_line_count + 1 %}
          {% endif %}
        {% endfor %}

        {{ ns.disrupted_line_count }}
      {% else %}
        unavailable
      {% endif %}

Well, for london overground, both lines do not have good service at the moment, one has Part Suspended, the other has Severe Delays. So that sensor will be zero.

And for the one using [0], i.e. the first line (Bakerloo), it also has Severe Delays, so that will also be zero.

Are you trying to get ALL lines?

All lines…

{{ value_json | map(attribute='lineStatuses') | selectattr('statusSeverityDescription', 'eq', 'Good Service') | list | count }}

oh, there’s some black magic that is needed for all lines because of the object structure.

{% set ns = namespace(lines=[]) %}
{% for station in value_json | map(attribute='lineStatuses') %}
  {% for line in station %}
    {% set ns.lines = ns.lines + [ line ] %}
  {% endfor %}
{% endfor %}
{{ ns.lines | selectattr('statusSeverityDescription', 'search', 'Good') | list | count }}

A quick clarification (and I’m really not being patronising, I am pretty sure you don’t live in London), the London Overground is one line but it has two disruptions on different sections. The TfL website Status update page reports it like this:
image

i.e. One line with two disruptions (Also see Piccadilly).


Just seen your update…
That doesn’t return the correct figure either, it is giving 14 which is the number of lines?

(I also missed that you are checking for equal to ‘Good Service’, I want to count the lines that are NOT ‘Good Service’ but that is a minor point.)

Just before I posted this I saw your second update. I’ll post this anyway and then look into your ‘black magic’… and comment back later…


Yes I am. I’d like to end up ideally with one sensor (and so only one api call) with the following

state: Number of lines with disruption.
attributes:

  • line1 name:
    • disruption1: details
    • disruption2: details
    • disrputionX: details
  • line2 name:
    • disruption1: details
    • disruption2: details
    • disrputionX: details

Alternatively I can have one sensor per line but either way I can’t see how to get the info I need into the attributes using the REST sensor, the source of data for the attributes in those sensors seems very restrictive. (Or anywhere else in HA so that I can manipulate it programmatically)

I’m getting 8 with good service and it correlates with the data coming from the api.

this is gettign all the delays

{% set ns = namespace(lines=[]) %}
{% for station in value_json | map(attribute='lineStatuses') %}
  {% for line in station %}
    {% set ns.lines = ns.lines + [ line ] %}
  {% endfor %}
{% endfor %}
{{ ns.lines | rejectattr('statusSeverityDescription', 'search', 'Good') | map(attribute='disruption.description') | list }}

returns

[
  "Bakerloo Line: Severe delays between Stonebridge Park and Harrow & Wealdstone due to a track fault at Harrow & Wealdstone. GOOD SERVICE on the rest of the line. ",
  "Central Line: Minor delays due to train cancellations. ",
  "Docklands Light Railway: Minor delays between Beckton and Tower Gateway due to a faulty train at Gallions Reach. GOOD SERVICE on other DLR routes. ",
  "Hammersmith and City Line: Minor delays due to train cancellations. ",
  "No service between Edmonton Green and Cheshunt and SEVERE DELAYS between Liverpool Street and Enfield Town / Cheshunt due to an earlier track fault at Clapton. GOOD SERVICE on all other London Overground routes. ",
  "No service between Edmonton Green and Cheshunt and SEVERE DELAYS between Liverpool Street and Enfield Town / Cheshunt due to an earlier track fault at Clapton. GOOD SERVICE on all other London Overground routes. ",
  "Piccadilly Line: Severe delays due to an earlier fire alert at Acton Town. Tickets are being accepted on Elizabeth Line and London Buses. "
]

So if I understand the data better, you have to use namespace.

{% set ns = namespace(lines=[]) %}
{% for line in value_json %}
  {% for status in line.lineStatuses if status.disruption is defined %}
    {% set ns.lines = ns.lines + [ {'line': line.name, 'severity':status.statusSeverity, 'description': status.statusSeverityDescription, 'info':status.disruption.description } ] %}
  {% endfor %}
{% endfor %}
{{ ns.lines }}

outputs

[
  {
    "line": "Bakerloo",
    "severity": 6,
    "description": "Severe Delays",
    "info": "Bakerloo Line: Severe delays between Stonebridge Park and Harrow & Wealdstone due to a track fault at Harrow & Wealdstone. GOOD SERVICE on the rest of the line. "
  },
  {
    "line": "Central",
    "severity": 9,
    "description": "Minor Delays",
    "info": "Central Line: Minor delays due to train cancellations. "
  },
  {
    "line": "DLR",
    "severity": 9,
    "description": "Minor Delays",
    "info": "Docklands Light Railway: Minor delays between Beckton and Tower Gateway due to a faulty train at Gallions Reach. GOOD SERVICE on other DLR routes. "
  },
  {
    "line": "Hammersmith & City",
    "severity": 9,
    "description": "Minor Delays",
    "info": "Hammersmith and City Line: Minor delays due to train cancellations. "
  },
  {
    "line": "London Overground",
    "severity": 6,
    "description": "Severe Delays",
    "info": "No service between Edmonton Green and Cheshunt and SEVERE DELAYS between Liverpool Street and Enfield Town / Cheshunt due to an earlier track fault at Clapton. GOOD SERVICE on all other London Overground routes. "
  },
  {
    "line": "London Overground",
    "severity": 3,
    "description": "Part Suspended",
    "info": "No service between Edmonton Green and Cheshunt and SEVERE DELAYS between Liverpool Street and Enfield Town / Cheshunt due to an earlier track fault at Clapton. GOOD SERVICE on all other London Overground routes. "
  },
  {
    "line": "Piccadilly",
    "severity": 6,
    "description": "Severe Delays",
    "info": "Piccadilly Line: Severe delays due to an earlier fire alert at Acton Town. Tickets are being accepted on Elizabeth Line and London Buses. "
  }
]

change last line to {{ ns.lines | count }} for number.

So for a full sensor… you’ll have a hard time with this because your top element is a list. The rest sensor will only be able to output a count otherwise you’ll hit the character limit.

Yeah… that was what really brought me here.
Is there no way to get this info into attributes?

Nope, no way.

Oh well…

Back to using the integration I suppose.
Which is a bit useless as it hasn’t been updated for the new Elizabeth Line which is about 5 minutes from me!

Thanks anyway.
I learnt some stuff.

There’s a custom integration that does this.

Yes I know.
It’s not exactly what I want so I thought I’d do my own.

I will probably be able to mange to get by with the CC though…

Ah, well you can get the count.

do you have a link to the restful api for the tfl?

Do you mean this?
Home - Transport for London - API (tfl.gov.uk)

You don’t need to register for the ‘Open Data’

Yes, hold on a sec