REST: not always returning sensor with data

Hi all, asking for your view/ideas on this.
For my bus-lines, I have a ‘real-time’ gtfs link, this I want to load every 5 mins.
The corresponding dataset is about 1.7Mb and I see that the sensors are not always calculated where they should (the data is in there)
The logs show nothing, i.e. I see the whole data-set loaded but no log entries after that.
My guess is that the dataset is too big to be dealt with at times but this is plain guessing
Although I only need 4 buslines from this dataset, I cannot change the source to provide me less data.
Question 1: can it indeed be the size of the data that obstructs the sensor to calculate?
Question 2: is there a way to download the data and cut it in pieces?
Link is here:

Example sensor is:

      - name: 530_rt_retour_Sophia2
        icon: mdi:bus
        unique_id: 530_rt_retour_Sophia2
        value_template: >
             {% if value_json is defined %}
             {% set y = value_json.entity | count %}
             {% set ns = namespace(val="unknown") %}
             {% for x in range(0,y) %}           
                {% set trip = value_json["entity"][x]["trip_update"]["trip"]["trip_id"] %}
                    {% if trip[:4] == "653R" %}
                        {% set y2 = value_json["entity"][x]["trip_update"]["stop_time_update"] | count %}
                        {% for x2 in range(0,y2) %}
                            {% set stop = value_json["entity"][x]["trip_update"]["stop_time_update"][x2]["stop_id"] %}
                                {% if stop[:5] == "01998" %}
                                    {% set ns.val = (as_local(as_datetime(value_json["entity"][x]["trip_update"]["stop_time_update"][x2]["departure"]["time"] + value_json["entity"][x]["trip_update"]["stop_time_update"][x2]["departure"]["delay"]))).strftime('%H:%M') %}
                                {% endif %}                              
                        {% endfor %} 
                    {% endif %}
             {% endfor %}{{ns.val}}
             {% else %}
             {% endif %}

Use a command_line sensor with curl pass the data to jq and you can parse out and make new JSON containing only what you need.

Thanks for the response and I now know jq exists but that is all…any pointers for documentation/examples on how to use it wrt to filtering? In this area total noob but happy to learn the hard way :slight_smile:

Hard way is to search for JQ playground. In an online browser you can paste in your JSON and start to learn. There are links to documentation there. I have some time today to look if you tell me exactly what you would want from the larger file.

Simple example on how to use curl with jq is here:

This actually gets CSV file and parses with JQ to JSON structure making sensor from the data.

Much more complex example is here:

See the top sensor which is using JQ to parse the locally stored Insteon devices JSON to completely reorganize it into groups.

1 Like

Much appreciated… learning always (euh?) fun…

While jq ain’t easy (I come from the XML/XSL world), it is great when it comes to what you can do to slice/dice and reorganize JSON. Good luck with it and feel free to post questions, always happy to help.