RESTful sensor: Filtering return data before JSON parsing?

Configuration
1

Hi,

I am integrating my Internorm i-tec window blinds with HA using the Internorm Smart Window Gateway.

While I am able to control the blinds, I am having problems receiving their state, position, and tilt position from the gateway, because the gateway returns JSON formatted data which is unfortunately prefixed with a non-standard status code string, e.g. {XC_SUC} on success or {XC_ERR} on error.

Example:

{XC_SUC}[{"type":"IN","sid":"01","adr":"01","config":"","state":"012400","deviceType":"01"}]

Is it possible to filter the return data of a RESTful sensor call, before JSON is parsed and stored in json_data, so I can remove the status string?

Alternatively, any hints on how to receive via HTTP, filter and parse this data directly from HA without executing external processes such as curl? I’d like to spare my HA of forking processes every few seconds.

Any help or idea appreciated!

Best regards,
// Veit

2

No, the rest integration is expecting valid JSON, and this prefix does not make this valid JSON.

I’m not aware of any existing integration that could do this for you, but the only thing I can think of would be a custom component that does almost the same as the rest integration, but where you would include a pre-parser to remove that prefix.

3

You could load the entire raw string in the rest sensor (note: there’s deliberately no value_template defined here, we want the whole ugly string to work with later):

  - platform: rest
    resource_template: "https://host_name/response.json"
    name: REST Test

In this example, I put your exact example data in a file “response.json”.

Then use template sensors to tease out the interesting bits:

template:
  - sensor:
      - name: Rest Test state
        state: '{{ (states("sensor.rest_test")[8:] | from_json )[0].state }}'

This chops off the first 8 characters of the state of sensor.rest_test string, deserializes the json, selects the first list item (since the remaining valid json is wrapped in square brackets it is treated as a single-item list), then returns the value of the “state” key from the json. This example makes some assumptions to simplify the explanation: that the bad data of the success or error message will always be the first 8 characters, that the remaining string will always be a json list (square brackets), and that “state” will always be one of the keys. Once you’ve got the valid json, you could be up more complex templates if needed.

image
image2160Ă—454 51.4 KB

4

This sounds great, @gonzotek. I was not aware that leaving value_template out loads the full string there.

I tried it, unfortunately, the full string (not the example) is 424 bytes long, and thus does not fit into the value field:

homeassistant.exceptions.InvalidStateError: Invalid state encountered for entity ID: sensor.mediola_device_states. State max length is 255 characters.

Is it maybe possible to store the whole thing in an attribute instead?

5

Hmm…not sure. Can you share the full string (redacting any private info if needed, but try to keep the length the same if possible)? I’d like to test a bit before throwing out any bad guesses. And indicate which fields are of the most interest (can just say all if that’s the case).

6

Another option might be to download it as a file (curl?) and then use command line stuff to cut-off the frist characters, after which you can then use the file via rest again.
Or…ask the provider of the data to just provide sensible data, where the status (SUC / ERR) is part of the json

7

@gonzotek : Thanks for your assistance. :slight_smile:

This is the full string that I get from the gateway:

{XC_SUC}[{"type":"IN","sid":"01","adr":"01","config":"","state":"012400","deviceType":"01"},{"type":"IN","sid":"02","adr":"02","config":"","state":"241C00","deviceType":"01"},{"type":"IN","sid":"03","adr":"03","config":"","state":"002400","deviceType":"01"},{"type":"IN","sid":"04","adr":"04","config":"","state":"241400","deviceType":"01"},{"type":"IN","sid":"05","adr":"05","config":"","state":"243000","deviceType":"01"}]

I am interested in the state attribute of all elements of type == "IN", addressed through the adr attribute, so a hash map of pairs adr => state (filtered for type == "IN" would include all data I need.

state is always a 6-digit all-uppercase hexadecimal string with the first byte representing some bit-encoded status flags, the second byte representing the tilt position of the blinds (0x00 is fully closed, 0x3c is fully open), and the third byte represents the position in percent (0x00 … 0x64).

So far my plan was to receive the data once from the gateway and use JSONPath to filter and select the state string presented as a sensor for every single window, here an example of the blind with adr == "04":

rest:
  - resource: http://192.168.88.101/command?XC_FNC=GetStates
    method: GET
    scan_interval: 5
    sensor:
      - name: "Arbeitszimmer Beschattung"
        json_attributes_path: '$.[?(@.adr == "04" && @.type == "IN")]'
        value_template: "OK"
        json_attributes:
          - "state"

And then to decode the data for position and tilt position inside the cover entity, likewise for the blind with adr == "04" as an example:

cover:
  - platform: template
    covers:
      arbeitszimmer_beschattung:
        friendly_name: "Arbeitszimmer Beschattung"
        unique_id: arbeitszimmer_beschattung
        device_class: blind
        position_template: "{{ state_attr('sensor.arbeitszimmer_beschattung', 'state')[4:6] | int(base=16) % 101 }}"
        tilt_template: "{{ (state_attr('sensor.arbeitszimmer_beschattung', 'state')[2:4] | int(base=16) * 5 / 3) | int % 101 }}"
        icon_template: >-
          {% if state_attr('sensor.arbeitszimmer_beschattung', 'state')[4:6] != '00' %}
            mdi:blinds-horizontal
          {% else %}
            mdi:blinds-horizontal-closed
          {% endif %}
        open_cover:
          service: rest_command.internorm_blinds_open
          data:
            gateway: 192.168.88.101
            adr: '04'
        close_cover:
          service: rest_command.internorm_blinds_close
          data:
            gateway: 192.168.88.101
            adr: '04'
        stop_cover:
          service: rest_command.internorm_blinds_stop
          data:
            gateway: 192.168.88.101
            adr: '04'
        set_cover_position:
          service: rest_command.internorm_blinds_set_position
          data:
            gateway: 192.168.88.101
            adr: '04'
            position: "{{position}}"
        set_cover_tilt_position:
          service: rest_command.internorm_blinds_set_tilt
          data:
            gateway: 192.168.88.101
            adr: '04'
            tilt: "{{tilt}}"

With your example in mind, my new plan was to store all JSON data in a single sensor.mediola_device_states entity and let the cover templates filter and select the state attribute to decode position and tilt position directly from this entity:

sensor:
  - platform: rest
    name: Mediola Device States
    resource: http://192.168.88.101/command?XC_FNC=GetStates
    method: GET
    scan_interval: 5

This ressource returns in the 424 bytes long string at the beginning of this reply.

8

Thanks @vingerha , but using curl is what I am trying to avoid to not produce needless system load through forks and process management. If this can’t be solved with HA’s in-house means, I’d either build a proxy service correcting the string or even replace the JSON with a new one with decoded data so HA does not have to, or try my best to find a custom component doing similar stuff that I could transform with my humble Python kung-fu into what I need such as removing the {XC_...} as @gonzotek suggested, or to replace either the string’s 8th character (resp. the first occurance of } via regexp) by : and add a } to the end of the string, to transform the whole thing into valid JSON.

This is the API that the vendor’s apps use to talk to the gateway, so they will not change that for a more sane one. They did not even respond to my request for the API documentation that I sent them through their official API documentation request form… So Wireshark is my only friend in this regard. :wink:

9

OK, how does the below look? You’ll still need to build your template covers and decode the hex (I haven’t had enough caffeine today for that :slight_smile: ), but it seems like this should give you what you need to be able to do that. Just add as many sensors under the resource key as you need and change the device.adr in the if clause for each unique device address.

The JSON will only be pulled once for as many device sensors as you place under it, and the list is filtered by the if clause before each for loops begin(actually, that’s not how that works…it has to loop over each item in the list to test for the values, but it’s still the most efficient way I know of to do this in jinja), so this should still be fairly efficient.

I haven’t added any error checks (for instance: if a given adr doesn’t appear in the results for some reason). That kind of situation should be possible to account for in the jinja template, but you’d need to make some decisions about how that would work.

rest:
  - resource: "http://192.168.88.101/command?XC_FNC=GetStates"
    scan_interval: 5
    sensor:
      - name: "REST Device 1"
        value_template: >-
          {% set devices = (value[8:] | from_json ) %}
          {% for device in devices if (device.adr == "01" and device.type == "IN") %}
            {{ device.state }}
          {% endfor %}
      - name: "REST Device 2"
        value_template: >-
          {% set devices = (value[8:] | from_json ) %}
          {% for device in devices if (device.adr == "02" and device.type == "IN") %}
            {{ device.state }}
          {% endfor %}
      - name: "REST Device 3"
        value_template: >-
          {% set devices = (value[8:] | from_json ) %}
          {% for device in devices if (device.adr == "03" and device.type == "IN") %}
            {{ device.state }}
          {% endfor %}
      - name: "Arbeitszimmer Beschattung"
        value_template: >-
          {% set devices = (value[8:] | from_json ) %}
          {% for device in devices if (device.adr == "04" and device.type == "IN") %}
            {{ device.state }}
          {% endfor %}
      - name: "REST Device 5"
        value_template: >-
          {% set devices = (value[8:] | from_json ) %}
          {% for device in devices if (device.adr == "05" and device.type == "IN") %}
            {{ device.state }}
          {% endfor %}

BTW, I came across this project while trying to learn if it was possible to get an individual device state (although I didn’t go very far down that path in the end). Just figured I’d share in case you hadn’t seen it already.

10

Thank you very much, @gonzotek !
This is great, I was not aware that the whole input is available in value. :smiley:

I ended up filtering in Jinja, because I think it looks nicer:

  - resource: http://192.168.88.101/command?XC_FNC=GetStates
    method: GET
    scan_interval: 5
    sensor:
      - name: "Schlafzimmer Beschattung links"
        value_template: '{{ value[8:] | from_json | selectattr("type", "equalto", "IN") | selectattr("adr", "equalto", "03") | map(attribute="state") | first }}'
      - name: "Schlafzimmer Beschattung rechts"
        value_template: '{{ value[8:] | from_json | selectattr("type", "equalto", "IN") | selectattr("adr", "equalto", "01") | map(attribute="state") | first }}'
      - name: "Arbeitszimmer Beschattung"
        value_template: '{{ value[8:] | from_json | selectattr("type", "equalto", "IN") | selectattr("adr", "equalto", "04") | map(attribute="state") | first }}'
      - name: "Gästezimmer Beschattung"
        value_template: '{{ value[8:] | from_json | selectattr("type", "equalto", "IN") | selectattr("adr", "equalto", "02") | map(attribute="state") | first }}'
      - name: "Wannenbad Beschattung"
        value_template: '{{ value[8:] | from_json | selectattr("type", "equalto", "IN") | selectattr("adr", "equalto", "05") | map(attribute="state") | first }}'

I am not sure whether your or my solution is actually more efficient – mine walks through the list several times, but yours needs to be interpreted instead of using specialised functions.
Regarding to your experience, do you expect your solution to be more efficient? I have no idea whether Jinja is runtime compiled and optimised…

2 Likes
11

Lol, no problem! I also didn’t really know if this could actually be achieved entirely within the Rest sensor when I first started looking…it just seemed like an interesting problem :-). I’ve done a few rest and scrape sensors in the past to pull data from public websites lacking real APIs, I’m sure what I learned with this will be useful in the future.

I also don’t know which method would be more efficient. They may even be more-or-less equivalent under the hood. I do recall HA devs mentioning that Jinja templates can be a performance hit, but in this case, either way we’re going to end up using templates. At least you aren’t flooding the network unnecessarily, just one request and its response every 5 seconds. I like your solution better as well. I didn’t comprehend what could be done with selectattr and map.

12

Hey @veitw , what “Internorm Smart Window Gateway” are you using?
Getting Internorm installed shortly with smart blinds but told I need s Gateway / Server to be able to control them on my phone.
Thanks

13

Hey @jsstevo,
sorry, I missed the notification on your reply.

I am using a Mediola AiO Gatewy v6, which was the same device Internorm sold branded as their own. As I was not willing to pay double the price just for a Internorm logo on top, I opted for this one.

In the meantime, Internorm has stopped developing their own app, which was a simplified version of the Mediola app and only suppoerted a few devices, and they are pushing you to switch to the Mediola app instead. Once your data is migrated, there is no way back either.

I used the Mediola and Internorm apps only a few times over two years ago, but at that time is was less polished than the Internorm app, especially regarding slat tilt, and the Internorm app also allowed to set a default position.

Since then I am using Home Assistant, and I figured out that it is possible to store a full set of 8 positions to recall instead of just one, so I gave every window the following positions to integrate into scenes:

  • open (blinds fully opened)
  • closed dark/sleeping/max temperature blocking (blinds fully closed, slats fully turned outwards)
  • closed bright/privacy (blinds fully closed, slats fully turned inwards, thus letting in light from the sky)
  • see-thru (blinds fully closed, slats in horizontal position)
  • shadowing/direct sunlight blocking (blinds fully closed, slats slightly turned outwards, so no direct sunlight hits my room)

These are well integrated into my automations:
If the outside temperature is low, all blinds that are not in sleeping or privacy mode, fully open to let in as much sunlight as possible to support heating.
If the outside temperature is high, when the sun comes around the corner, all windows on the corresponding side of the house that are open or see-thru are turned to shadowing for rooms that are occupied, and for rooms that are not occupied (also assumed for all rooms if house door is locked externally), the windows are put into max temperature blocking mode,

1 Like