Hi all, asking for your view/ideas on this.
For my bus-lines, I have a ‘real-time’ gtfs link, this I want to load every 5 mins.
The corresponding dataset is about 1.7Mb and I see that the sensors are not always calculated where they should (the data is in there)
The logs show nothing, i.e. I see the whole data-set loaded but no log entries after that.
My guess is that the dataset is too big to be dealt with at times but this is plain guessing
Although I only need 4 buslines from this dataset, I cannot change the source to provide me less data.
Question 1: can it indeed be the size of the data that obstructs the sensor to calculate?
Question 2: is there a way to download the data and cut it in pieces?
Link is here: https://proxy.transport.data.gouv.fr/resource/region-sud-gtfs-rt-trip-update
Example sensor is:
sensor:
- name: 530_rt_retour_Sophia2
icon: mdi:bus
unique_id: 530_rt_retour_Sophia2
value_template: >
{% if value_json is defined %}
{% set y = value_json.entity | count %}
{% set ns = namespace(val="unknown") %}
{% for x in range(0,y) %}
{% set trip = value_json["entity"][x]["trip_update"]["trip"]["trip_id"] %}
{% if trip[:4] == "653R" %}
{% set y2 = value_json["entity"][x]["trip_update"]["stop_time_update"] | count %}
{% for x2 in range(0,y2) %}
{% set stop = value_json["entity"][x]["trip_update"]["stop_time_update"][x2]["stop_id"] %}
{% if stop[:5] == "01998" %}
{% set ns.val = (as_local(as_datetime(value_json["entity"][x]["trip_update"]["stop_time_update"][x2]["departure"]["time"] + value_json["entity"][x]["trip_update"]["stop_time_update"][x2]["departure"]["delay"]))).strftime('%H:%M') %}
{% endif %}
{% endfor %}
{% endif %}
{% endfor %}{{ns.val}}
{% else %}
unknown
{% endif %}
Thanks for the response and I now know jq exists but that is all…any pointers for documentation/examples on how to use it wrt to filtering? In this area total noob but happy to learn the hard way
Hard way is to search for JQ playground. In an online browser you can paste in your JSON and start to learn. There are links to documentation there. I have some time today to look if you tell me exactly what you would want from the larger file.
Simple example on how to use curl with jq is here:
This actually gets CSV file and parses with JQ to JSON structure making sensor from the data.
Much more complex example is here:
See the top sensor which is using JQ to parse the locally stored Insteon devices JSON to completely reorganize it into groups.
While jq ain’t easy (I come from the XML/XSL world), it is great when it comes to what you can do to slice/dice and reorganize JSON. Good luck with it and feel free to post questions, always happy to help.
