How do I transform two lists into a dict in a rest sensor's value template?

[mrmachine]

Given that my REST resource returns the following JSON:

{"range":[1572526800,1572605220],"labels":["voltage","mains","oven","air_conditioners_1","hot_water_system","solar_analytics","power_points_1","lights_1","power_points_2","lights_2","air_conditioners_2","fronius_inverter","latronics_inverter","air_conditioners","consumption","grid_export","grid_import","grid_import_neg","lights","mains_inverted","power_points","production","self_consumption","unmeasured"],
"data":[[244.8,-863.9,0.1,15.1,99.5,1.2,100.8,17.6,186.9,15.7,16.8,1098,224.5,32,458.5,863.9,0,-0,33.3,863.9,287.7,1322.5,458.5,4.8]]}

How would I zip the labels and data lists into a dict, so I end up with:

{'voltage': 244.8, 'mains': -863.9, 'oven': 0.1, 'air_conditioners_1': 15.1, 'hot_water_system': 99.5, 'solar_analytics': 1.2, 'power_points_1': 100.8, 'lights_1': 17.6, 'power_points_2': 186.9, 'lights_2': 15.7, 'air_conditioners_2': 16.8, 'fronius_inverter': 1098, 'latronics_inverter': 224.5, 'air_conditioners': 32, 'consumption': 458.5, 'grid_export': 863.9, 'grid_import': 0, 'grid_import_neg': 0, 'lights': 33.3, 'mains_inverted': 863.9, 'power_points': 287.7, 'production': 1322.5, 'self_consumption': 458.5, 'unmeasured': 4.8}

as the sensor value?

I was thinking something like:

{% set foo = {} %}
{% for i in range(0, value_json.labels|length) %}
{% set foo[value_json.labels[i]] = value_json.data[0][i] %}
{% endfor %}
{{ foo }}

But when I test that in the template editor, I get:

Error rendering template: TemplateSyntaxError: expected token 'end of statement block', got '['

[BlueBlueBlob]

Try with a persistent notification to have “logs”.
And display value between for and set.
And check value_json to be sure…
Sorry no idea, seems good for me

[petro]

You’re limited to 255 characters for a state. I don’t think this will ever work for you. Instead, use the json_atrributes field to place each kvp as an attribute. Also, states are only strings, so you won’t be able to use it like a dictionary.

[mrmachine]

@petro I’d be fine having the zipped dict as an attribute instead. But how do I transform the two lists into a dict?

[petro]

Ahh, it’s a pain to view posts on mobile. I see what you’re asking now.

You cannot create dictionaries from scratch in jinja.

A new macro was added to convert a json string to a json object. You may be able to iterate through both lists in a macro and return a json_dict as a sting. Then use the new from_json macro to convert it. But that will have to be done in a template sensor as rest does not have the tools to extract that properly every time

[SteveDinn]

How about this? Don’t try to create a dictionary, just write JSON to the output as you go.

{%- set value_json = {"range":[1572526800,1572605220],"labels":["voltage","mains","oven","air_conditioners_1","hot_water_system","solar_analytics","power_points_1","lights_1","power_points_2","lights_2","air_conditioners_2","fronius_inverter","latronics_inverter","air_conditioners","consumption","grid_export","grid_import","grid_import_neg","lights","mains_inverted","power_points","production","self_consumption","unmeasured"],
"data":[[244.8,-863.9,0.1,15.1,99.5,1.2,100.8,17.6,186.9,15.7,16.8,1098,224.5,32,458.5,863.9,0,-0,33.3,863.9,287.7,1322.5,458.5,4.8]]} -%}

{{ "{\n" }}
{%- for i in range(0, value_json.labels|length) -%}
  {%- set key = value_json.labels[i] -%}
  {%- set value = value_json.data[0][i] -%}
  {{ '  "' + key + '": ' }}{{ value }}{{ "," if not loop.last else "" }}{{"\n"}}
{%- endfor -%}
{{ "}\n" }}

Outputs as:

{
  "voltage": 244.8,
  "mains": -863.9,
  "oven": 0.1,
  "air_conditioners_1": 15.1,
  "hot_water_system": 99.5,
  "solar_analytics": 1.2,
  "power_points_1": 100.8,
  "lights_1": 17.6,
  "power_points_2": 186.9,
  "lights_2": 15.7,
  "air_conditioners_2": 16.8,
  "fronius_inverter": 1098,
  "latronics_inverter": 224.5,
  "air_conditioners": 32,
  "consumption": 458.5,
  "grid_export": 863.9,
  "grid_import": 0,
  "grid_import_neg": 0,
  "lights": 33.3,
  "mains_inverted": 863.9,
  "power_points": 287.7,
  "production": 1322.5,
  "self_consumption": 458.5,
  "unmeasured": 4.8
}

[petro]

That would work if it stays under 255 characters.

[SteveDinn]

You could always set it to an input_text rather than a sensor.

[petro]

Pretty sure all states are limited. Attributes are unlimited. The problem is that he’s using rest sensor and it has limited ways to get the data out.

[SteveDinn]

I didn’t realize that was a limit on all state values. I’m surprised I haven’t run up against that yet.

[mrmachine]

Thanks. I’m making progress. Here’s my config now:

sensor:
  - platform: rest
    name: IoTaWatt (Show)
    resource: http://iotawatt.home/query?show=series
    json_attributes:
      - series
    value_template: 'on'
  - platform: rest
    name: IoTaWatt (Select)
    # resource_template: http://iotawatt.home/query?select=[{% for label in states.sensor.iotawatt_show.attributes.series %}{{ label.name }},{% endfor %}]&begin=s-5s&end=s&group=5s&header=yes
    resource_template: http://iotawatt.home/query?select=[voltage,mains,oven,hot_water_system,solar_analytics,fronius_inverter,latronics_inverter,air_conditioners,consumption,grid_export,grid_import,lights,power_points,production,self_consumption]&begin=s-5s&end=s&group=5s&header=yes
    json_attributes:
      - labels
      - data
    value_template: 'on'
  - platform: template
    sensors:
      iotawatt:
        friendly_name: IoTaWatt
        value_template: 'on'
        attribute_templates:
          data: >-
            {% macro zip_labels_and_data() %}
              {
              {%- for i in range(0, state_attr('sensor.iotawatt_select', 'labels')|length|default(0)) -%}
                "{{ state_attr('sensor.iotawatt_select', 'labels')[i] }}": {{ state_attr('sensor.iotawatt_select', 'data')[0][i] }}{% if not loop.last %},{% endif %}
              {%- endfor -%}
              }
            {% endmacro %}
            {{ zip_labels_and_data()|from_json }}
      hot_water_system:
        friendly_name: Hot Water System
        icon_template: >-
          {% if state_attr('sensor.iotawatt', 'data').hot_water_system > 100 %}
            mdi:water
          {% elif state_attr('sensor.iotawatt', 'data').hot_water_system > 1 %}
            mdi:water-outline
          {% else %}
            mdi:water-off
          {% endif %}
        value_template: >-
          {% if state_attr('sensor.iotawatt', 'data').hot_water_system > 100 %}
            On
          {% elif state_attr('sensor.iotawatt', 'data').hot_water_system > 1 %}
            Standby
          {% else %}
            Off
          {% endif %}

A few problems remain:

1. If I use the commented resource_template for the IoTaWatt (Select) sensor, I don’t get any data at all for that sensor. But I’ve confirmed via the template editor that the template does produce the exact same hard coded URL that I am using instead. I’d like to use the template to automatically pull in data for all channels without having to specifically name them.

   

2. The data attribute on the iotawatt sensor does appear to be parsed as JSON into an object (per |from_json), BUT the hot_water_system sensor is showing as unavailable (see states screenshot below).

   

3. The template editor renders nothing (an empty string) for state_attr('sensor.iotawatt', 'data').hot_water_system and state_attr('sensor.iotawatt', 'data')["hot_water_system"], and an error for (state_attr('sensor.iotawatt', 'data')|from_json).hot_water_system.

   
   

   It looks like |from_json is actually converting the rendered JSON string into an object and then re-rendering it as an invalidly formatted JSON string (with single quotes instead of double quotes)?

   If I remove |from_json from the macro and instead use it in the hot_water_system sensor directly (and every other sensor I want to create), then everything works:

   

Here’s the final working config (with workarounds):

sensor:
  - platform: rest
    name: IoTaWatt (Show)
    resource: http://iotawatt.home/query?show=series
    json_attributes:
      - series
    value_template: 'on'
  - platform: rest
    name: IoTaWatt (Select)
    # resource_template: http://iotawatt.home/query?select=[{% for label in states.sensor.iotawatt_show.attributes.series %}{{ label.name }},{% endfor %}]&begin=s-5s&end=s&group=5s&header=yes
    resource_template: http://iotawatt.home/query?select=[voltage,mains,oven,hot_water_system,solar_analytics,fronius_inverter,latronics_inverter,air_conditioners,consumption,grid_export,grid_import,lights,power_points,production,self_consumption]&begin=s-5s&end=s&group=5s&header=yes
    json_attributes:
      - labels
      - data
    value_template: 'on'
  - platform: template
    sensors:
      iotawatt:
        friendly_name: IoTaWatt
        value_template: 'on'
        attribute_templates:
          data: >-
            {% macro zip_labels_and_data() %}
              {
              {%- for i in range(0, state_attr('sensor.iotawatt_select', 'labels')|length|default(0)) -%}
                "{{ state_attr('sensor.iotawatt_select', 'labels')[i] }}": {{ state_attr('sensor.iotawatt_select', 'data')[0][i] }}{% if not loop.last %},{% endif %}
              {%- endfor -%}
              }
            {% endmacro %}
            {{ zip_labels_and_data() }}
      hot_water_system:
        friendly_name: Hot Water System
        icon_template: >-
          {% if (state_attr('sensor.iotawatt', 'data')|from_json).hot_water_system > 100 %}
            mdi:water
          {% elif (state_attr('sensor.iotawatt', 'data')|from_json).hot_water_system > 1 %}
            mdi:water-outline
          {% else %}
            mdi:water-off
          {% endif %}
        value_template: >-
          {% if (state_attr('sensor.iotawatt', 'data')|from_json).hot_water_system > 100 %}
            On
          {% elif (state_attr('sensor.iotawatt', 'data')|from_json).hot_water_system > 1 %}
            Standby
          {% else %}
            Off
          {% endif %}

But I’d still like to be able to fetch all data without having to name individual channels, and I’d like to not have to use |from_json every time I access the transformed data attribute.

[SteveDinn]

You’re not using from_json in the right way. Most of the time, templates take a JSON string and convert it to an object before you even see it so that you can access fields. A JSON string is literally just a string. You had a JSON string before you called from_json. You ended up with an object that when converted to a string for display in the default way is not valid JSON.

Edit: When you use it to read fields in the sensor’s data attribute, that’s how it was intended. Attributes containing JSON don’t get parsed into objects automatically.

You can get your stuff in individual attributes if you want that, by naming each one under attribute_templates like this:

  attribute_templates:
    voltage: '{{ value.json.data[0][0] }}'
    mains: '{{ value_json.data[0][1] }}'
    etc.

[mrmachine]

But is there any way to create an attribute for each value (like you have) without relying on the numerical indexes? If I add a new output or to IoTaWatt or rename an input, the indexes may change. Hence why I wanted to zip the list of labels and data values into a dict to begin with.

So if I understand correctly:

• I have to capture the raw labels and data attributes as JSON in the rest sensor to avoid a 255 character limit.
• I have to make an additional template sensor purely so I can reformat labels and data into a JSON string attribute.
• Everywhere I want to access this attribute (in additional template sensors for each label I want to expose) I need to use |from_json to convert it into an object first.

And why does the resource_template URL not work, but the hard coded URL does? The hard coded URL was obtained by pasting the template into the template editor and copying the rendered result.

[SteveDinn]

Yeah. You got it right. Sometimes you have to jump through some hoops. I didn’t even know about the 255 character limit until this thread.

You can get your stuff in individual attributes without indexes, but you would have to have the same parsing code in each attribute template. I think that would suck even more. If you can be reasonably sure that those indexes won’t change, I’d totally go the attribute_templates route.

[mrmachine]

For anyone else who runs into this issue, here is the complete and final answer:

sensor:
  - platform: rest
    name: IoTaWatt (Select)
    resource: http://iotawatt.home/query?select=[voltage,mains,oven,hot_water_system,solar_analytics,fronius_inverter,latronics_inverter,air_conditioners,consumption,grid_export,grid_import,lights,power_points,production,self_consumption]&begin=s-5s&end=s&group=5s&header=yes
    json_attributes:
      - labels
      - data
    value_template: 'on'
  - platform: template
    sensors:
      iotawatt:
        friendly_name: IoTaWatt
        attribute_templates:
          data: >-
            {
            {%- for i in range(0, state_attr('sensor.iotawatt_select', 'labels')|length|default(0)) -%}
            "{{ state_attr('sensor.iotawatt_select', 'labels')[i] }}": {{ state_attr('sensor.iotawatt_select', 'data')[0][i] }}{% if not loop.last %}, {% endif %}
            {%- endfor -%}
            }
        value_template: 'on'
      iotawatt_air_conditioners:
        friendly_name: Air Conditioners
        unit_of_measurement: Watts
        value_template: '{{ (state_attr("sensor.iotawatt", "data")|from_json).air_conditioners }}'

I still don’t know why I can’t use resource_template instead of resource, or why I have to include value_template: 'on', when I only care about the attributes.