REST - Template like payload transformation

Hello,

The current rest integration allows for pulling json attributes and populating them as attributes to sensors or binary sensors.

Today, if you wanted to do some basic transformation, like mapping a property of a json response to another attribute (eg.: Response provides ‘Lat’ and ‘Lng’ keys, need to map to ‘latitude’ and ‘longitude’ respectively for mapping purposes), you’d have to do it with an intermediate template sensor.

This could be simplified by baking templates into json_attributes, also allows for basic data transformation right from the integration without the need for intermediate templates.

It would be nice to be able to name rest integration sensors (multiple sensors) with templates.
I think this would be nice, and would really clean up the entity list.

It would also be nice to have the template style block configuration unique ID for REST integrations with multiple sensors.

Using the new Rest integration format: https://www.home-assistant.io/integrations/rest/ you can directly create a sensor for each attribute, each with their own value template.

Or maybe I don’t understand your use case. Can you give an actual example?

TL;DR: I think there’s a lot to be gained if we could define sensors in REST integration just like defining sensors in Template integration, benefits include ability to map values, ability to do basic transformation and enumeration.

Well in my case, here’s my use case, sorry for verbosity, I don’t know how to make it any shorter but keep the clarity.


Data and Current Configuration

To give some context, I have a RESTful integration that polls an API with the following response:

{
    "BusStopCode": "20251",
    "Services": [
        {
            "ServiceNo": "176",
            "Operator": "MY_BUS_OPERATOR",
            "NextBus": {
                "OriginCode": "10009",
                "DestinationCode": "45009",
                "EstimatedArrival": "2020-02-12T14:09:11+08:00",
                "Latitude": "SOME_COORDINATE",
                "Longitude": "SOME_COORDINATE",
                "VisitNumber": "1",
                "Load": "SEA",
                "Feature": "WAB",
                "Type": "DD"
            },
            "NextBus2": {
                "OriginCode": "10009",
                "DestinationCode": "45009",
                "EstimatedArrival": "2020-02-12T14:21:19+08:00",
                "Latitude": "SOME_COORDINATE",
                "Longitude": "SOME_COORDINATE",
                "VisitNumber": "1",
                "Load": "SEA",
                "Feature": "WAB",
                "Type": "DD"
            },
            "NextBus3": {
                "OriginCode": "10009",
                "DestinationCode": "45009",
                "EstimatedArrival": "2020-02-12T14:44:30+08:00",
                "Latitude": "SOME_COORDINATE",
                "Longitude": "SOME_COORDINATE",
                "VisitNumber": "1",
                "Load": "SEA",
                "Feature": "WAB",
                "Type": "DD"
            }
        },
        ...otherServices // Right now limited to 1 since I don't know how to dynamically create sensors. 
    ]
}

So then, I have sensors for each “NextBus” instance (each bus service only returns up to 3 next bus instances), exposing every json data inside each json object. For brevity, I only included one ‘next bus’ from my instance, the other two are pretty much the same as the one defined below.

resource: MY_3RD_PARTY_RESOURCE_URL
sensor:
  - name: "REST: My Next Bus"
    value_template: "{{ value_json.Services[0].NextBus.EstimatedArrival }}"
    device_class: timestamp
    json_attributes_path: "$.Services[0].NextBus"
    json_attributes:
      - OriginCode # Basically all the attributes inside each next bus instance.
      - DestinationCode
      - EstimatedArrival
      - Latitude
      - Longitude
      - VisitNumber
      - Load
      - Feature
      - Type

Here is a list of items I feel could be improved on:


Inconvenience 1:

Mapping properties from JSON to attribute

The GPS coordinates of the bus are listed in the JSON object as Latitude and Longitude. When adding those to the attribute, the map ignores the coordinates and does not display the entities on the map.

I have to create another template sensor which maps Latitude and Longitude to latitude and longitude (yes just making the names lower case :joy:) for it to show up.


Inconvenience 2:

Template string based sensor name

The bus services also end at about 1am daily (depending on traffic).
Afterwhich, the data of each field becomes an empty string.

Eg: "{{ value_json.Services[0].NextBus.EstimatedArrival }}"
Instead of an ISO8601 datetime string, if bus service ended, it will return: ''.

When this is the case, I want to name (friendly name) the sensor as {{ Bus Service Number }} - Service Ended.
This is also only possible right now with template sensors.


Inconvenience 3:

Template string based attributes and basic data transformation

The data provided by the API is extremely abbreviated and does not make sense to be used immediately.

For example:

{
    // This means 'SEAting is available' 
    "Load": "SEA",  // Seats Available (SEA), Standing Available (SDA), Limited Standing (LDA)
    "Feature": "WAB", // Wheelchair Accessible Bus
    "Type": "DD" // Double Deck (DD), Single Deck (SD), Bendy Bus (BD)
}

It would be great to have just a template built right in to parse these enumerated values into something that’s human readable.


Inconvenience 4:

Template string based sensor icon

Same for icons, would be nice to set an icon template to RESTful sensors by enumerating a property on value_json using template strings.

From the above examples, I am show the type of bus using the icon (double deck / single deck), and bus with alert icon when the service has ended. This is already possible using a second template sensor, but not possible directly from RESTful sensor.


Inconvenience 5:

Config block based unique_id

I couldn’t seem to set a unique_id directly from the RESTful sensors. Would be nice if we could do block configuration for that just like in template sensors (the non-legacy way).

For example, I would name the entire block with {bus stop code}_{bus service number}, then each sensor would only need to append next_bus, next_bus_2, next_bus_3


Conclusion

If you’re still here, thank you for reading this.

The above points are why I think it would be nice to integrate how things are done in the Template integration today with the RESTful integration.

I’m sure others have faced the same situation while using other APIs that are out of their control, and thought this is a pretty generic use case that can benefit the length of everybody’s entity list. :joy:

All the cases above can be solved today with Template integration, just would be nice to not have 2 entities for each bus.

I read it all and you make some good arguments. :+1:

  1. That’s annoying. It should be pretty easy to submit a PR for the map to accept capitalised lat and lon.

  2. Why not change the state to that message rather than the name? That can be done in the value_template.

  3. This is the best example for your request.

  4. That would be a nice feature.

  5. That would be a nice feature.

So you get my vote. Don’t forget to vote for your own request.

Sadly, in case 2, I’ve simplified the case, there are other exceptions that I have to take care of, which also could be solved with templates integration. I have thought of that but still not the cleanest solution. :frowning:

Ideally I wanna just copy and paste this file, search and replace the bus service number for any number of buses I need. :joy:
Maybe eventually I might just make a custom integration to do this, still too new at Home Assistant at this point.

Also thanks for the vote! Appreciate it! :smiley:

Hi Samuel, I am working on a similar usage for next bus . My Paylod is different:
{
“result”: {
“schedules”: [
{
“message”: “A l’arret”,
“destination”: “Parc de Saint-Cloud”
},
{
“message”: “17 mn”,
“destination”: “Parc de Saint-Cloud”
}
]
},
“_metadata”: {
“call”: “GET /schedules/buses/52/assomption+++radio+france/R”,
“date”: “2022-01-23T10:25:37+01:00”,
“version”: 4
}
}

  - platform: rest
    resource: https://api-ratp.pierre-grimaud.fr/v4/schedules/buses/52/assomption+++radio+france/R
    name: Bus_52_Saint_Cloud_Assomption
    value_template: >- 
      {%set arret="A l'arret" %}
      {%set approche="A l'approche" %}
      {{ value_json.result.schedules[0].message|regex_replace(find='mn', replace='', ignorecase=False)| regex_replace(find=approche, replace='1', ignorecase=False)| regex_replace(find=arret, replace='0', ignorecase=False)}}
    json_attributes_path: "$.result.schedules[1]"
    json_attributes:
      - message
    unit_of_measurement: 'mn'

This is working fine

1 Like

Oh this is interesting, never thought of piping it through a regex filter. Thanks for the suggestion!