Creating valid JSON output from jinja template

[mirekmal (Mirek Malinowski)]

I’m trying to finetune fing API output to my need. In particular there is a problem with easy sorting by IP address of device. I would like to solve it by adding to original json output one node with sort order (actually just lasy segment of IP address, but as initeger). Currentlu output of fing sensor looks like that (credit goes to @RoyOltmans for his work on fing):

I’d like to replicate this takinig as source data from this sensor and adding one more key to json output, that would be just last segment of IP address. For this I created following template:

As you can see it renders proper json oputput with exception of adding ‘order’ key and translating list of IP addresses to single address.
BUT this just looks like valid json, this is text, not json object that could be easily iterated in template. Seems there is some extra step missing with conversion. I found some references to jinja filters tojson of to_json or even from_json but using them either renders error or changes output of template to something for sure different than json.

Here is full sensor code for reference:

      - name: ordered_network_devices
        state: 'on'
        attributes:
          details: >
            {% set dev_list = namespace(devs = '') %}
            {% set json_string = '' %}
            {% set devices = state_attr('sensor.all_network_devices', 'devices') %}
            {%- for device in devices -%}
            {%- set order = '{"order":"' + (device.ip[0] | default('N/A')).split('.')[3] + '",' -%}
            {%- set mac = '"mac":"' + (device.mac | default('N/A')) + '",' -%}
            {%- set ip = '"ip":"' + (device.ip[0] | default('N/A')) + '",' -%}
            {%- set state = '"state":"' + (device.state | default('N/A')) + '",' -%}
            {%- set name = '"name":"' + (device.name | default('N/A')) + '",' -%}
            {%- set type = '"type":"' + (device.type | default('N/A')) + '",' -%}
            {%- set first_seen = '"firts_seen":"' + (device.first_seen | default('N/A')) + '",' -%}
            {%- set last_changed = '"last_changed":"' + (device.last_changed | default('N/A')) + '"' -%}
            {%- set json_string = json_string + order + mac + ip + state + name + type + first_seen + last_changed + '},' -%}
            {% set dev_list.devs = dev_list.devs + json_string  %}
            {% endfor %}
            {{ dev_list.devs }}

and output it renders in developer toole:

Any hionts on how to get around this problem?

[123 (Taras)]

Try this version in the Template Editor.

{% set devices = state_attr('sensor.all_network_devices', 'devices') %}
{% set ns = namespace(devices=[]) %}
{% for d in devices %}
  {% set ns.devices = ns.devices + [dict(
    order=d.ip[0].split('.')[3], 
    mac=d.mac,
    ip=d.ip[0],
    state=d.state,
    name=d.name,
    type=d.type,
    first_seen=d.first_seen,
    last_changed=d.last_changed)] %}
{% endfor %}
{{ ns.devices }}

I tested it (using other data) and it produces a valid list of dictionary items.

Feel free to add/modify the dictionary keys to meet your requirements.

[mirekmal (Mirek Malinowski)]

Well, it works! Though not entirely as I expected
So your template definitely renders output I expected and after modyfying rest of my config to adjust to changed sensor it almost works. There are, however 2 issues.

The first one is something I can’t even think how to explain. Sensor definition is exactly as you proposed, so one of attributes is called first_seen… but in the state table this name is changed to firts_seen:

I double and triple checked code:

Second issue is related to behavior after implementation of this change… I hoped, that after applying sorting to list of devices by order, which is now simple number, not string representing IP address (though probably text, not integer) I’ll get proper sorting order by IP address… but not, it works exactly like it was previously, e.g. 1, 10, 11, 2, 20, 12 instead of 1, 2, 10, 11, 20… I do not think there is any simple solution to this So outcome is not exactly I was hoping for:

[123 (Taras)]

@mirekmal

but in the state table this name is changed to firts_seen:

Your original example did that.

{%- set first_seen = '"firts_seen":"' + (device.first_seen | default('N/A')) + '",' -%}
                       ^^^^^^^^^^

I don’t see how my example is capable of doing it.

first_seen=d.first_seen,

@mirekmal

which is now simple number, not string representing IP address

The value is a numeric string, so it will be sorted alphabetically not numerically. Convert the value to an integer.

order=d.ip[0].split('.')[3]|int(0),

[mirekmal (Mirek Malinowski)]

Now I got some disaster

@123

Your original example did that.

So seems that was some sort of caching issue or something? Perhaps, since source sensor refreshes only every 1 minute I was not waiting long enough and got this error. But strange as later playing with code in UI took for sure more and finally I got it working.

Now after suggested update (adding int filter) I again restarted HA and sensor shows proper first_seen key.
But then things get more complicated as everything else stopped to work. Now whole output of senor is no longer seen as json object and again it counts to some ~20000 elemnts insted of ~100… So if I try to extract key value from sensor sttribute I get info that it does not have specific element… any element in fact, I tested for all…
I tried to revert to previous state (without int) and it also does not work…
For comparison, original sensor:

And one based on your code:

[123 (Taras)]

There’s something weird going on with your system. The first evidence of its weirdness is when it reported firts_seen even when the template used first_seen. Now it reports the template’s result as a string even though the template clearly generates a list.

[mirekmal (Mirek Malinowski)]

OK, I think I figured out the reasons for this strange behavior.
First, wrong names of keys in json - my sensor in fact is trigger sensor, so new values should be rendered when base sensor changes its state. As status of devices (connecting/disconnecting from WiFi) is changing rather rarely, most likely I was observing the old state of sensor, as I initially created it, not rendered in line with it most up to date definition.
Second, issues with proper rendering of json output… Apparently both your and mine method of creation of proper json output is vulnerable to structure of source file. While troubleshooting output I found at least 2 main resons for screewing up the output:

• if the specific key is not present - in this particular case model key is not present or is empty for some of devices.
• if any value contains special characters ( " ) it also screews the flow of template rendering wrong file.

I updated my template to remediate these issues and now I get proper json as output! As I can’t resolve order value to be INT, it is obviously not working fully as expected yet. Here is updated sensor code:

template:
  - trigger:
    - platform: time_pattern
      minutes: '/01'
    sensor:
      - name: ordered_network_devices
        state: 'on'
        attributes:
          details: >     
            {% set devices = state_attr('sensor.all_network_devices', 'devices') %}
            {% set dev_list = namespace(devs = '') %}
            {% set json_string = '' %}
            {%- for device in devices -%}
            {%- set order = '{"order":"' + (device.ip[0] | default('N/A')).split('.')[3] + '",' -%}
            {%- set mac = '"mac":"' + (device.mac | default('N/A')) + '",' -%}
            {%- set ip = '"ip":"' + (device.ip[0] | default('N/A')) + '",' -%}
            {%- set state = '"state":"' + (device.state | default('N/A')) + '",' -%}
            {%- set name = '"name":"' + (device.name | default('N/A')) + '",' -%}
            {%- set type = '"type":"' + (device.type | default('N/A')) + '",' -%}
            {%- set make = '"make":"' + (device.make | default('N/A')) + '",' -%}
            {% if 'model' in device %}
              {% if device.model == "" %}
                {%- set model = '"model":"' + 'N/A' + '",' -%}
              {% else %}
                {%- set model = '"model":"' + (device.model | replace('"', '') | default('N/A')) + '",' -%}
              {% endif %}
            {% else %}
              {%- set model = '"model":"' + 'N/A' + '",' -%}
            {% endif %}
            {%- set first_seen = '"first_seen":"' + (as_timestamp(strptime((device.first_seen  | replace('T', ' ') | replace('Z', '')).split('.')[0], '%Y-%m-%d %H:%M:%S')) | timestamp_custom('%d/%m/%y %H:%M:%S')| default('N/A')) + '",' -%}
            {%- set last_changed = '"last_changed":"' + (as_timestamp(strptime((device.last_changed  | replace('T', ' ') | replace('Z', '')).split('.')[0], '%Y-%m-%d %H:%M:%S')) | timestamp_custom('%d/%m/%y %H:%M:%S')| default('N/A')) + '"' -%}
            {%- set json_string = json_string + order + mac + ip + state + name + type + make + model + first_seen + last_changed + '},' -%}
            {% set dev_list.devs = dev_list.devs + json_string  %}
            {% endfor %}
            {{ dev_list.devs }} 

It includes some more fixes too; changed trigger from state change to time_pattern and reformatting time of first_seen and last_changed to more readible format. Next step will be to reapplyall the fixes to your code.

[123 (Taras)]

This is prone to producing an error.

default('N/A')).split('.')[3]

Why? Because there’s no element 3 in the default string.

I’m not sure why you prefer to build a dictionary using strings to create JSON (complex mess of quotes and serial concatenations) instead of simply using dict.

You can use templates within a dict function therefore it can easily determine what to assign to model.

[mirekmal (Mirek Malinowski)]

@123

I’m not sure why you prefer to build a dictionary using strings to create JSON (complex mess of quotes and serial concatenations) instead of simply using dict.

I’m not jinja expert, by any means, just learning
Now, building on this exercise I modified the code you suggested and finally gopt what I wanted:

      - name: new_network_devices
        state: 'on'
        attributes:
          details: >
            {% set devices = state_attr('sensor.all_network_devices', 'devices') %}
            {% set ns = namespace(devices=[]) %}
            {% for d in devices %}
              {% set ns.devices = ns.devices + [dict(
                order=d.ip[0].split('.')[3]|int(0),
                mac=d.mac|string,
                ip=d.ip[0]|string,
                state=d.state|string,
                name=d.name|default("N/A")|string,
                type=d.type|default("N/A")|string,
                make=d.make|default("N/A")|string,
                model=d.model|default("N/A")|string,
                first_seen=as_timestamp(strptime((d.first_seen | replace('T', ' ') | replace('Z', '')).split('.')[0], '%Y-%m-%d %H:%M:%S')) | timestamp_custom('%d/%m/%y %H:%M:%S'),
                last_changed=as_timestamp(strptime((d.last_changed | replace('T', ' ') | replace('Z', '')).split('.')[0], '%Y-%m-%d %H:%M:%S')) | timestamp_custom('%d/%m/%y %H:%M:%S')
                )] %}
            {% endfor %}
            {{ ns.devices }}

Again, I’m not sure if thisw is the best wqay to deal with non existing or emty keys, but addind default and applying string filter resolved issues and now I get clean json. And abviously applying int filter to order did the trick with sorting devices properly on the list:

[123 (Taras)]

String conversion

It shouldn’t be necessary to use the string filter everywhere. All of the values (shown in your first post’s screenshot) are strings.

You can prove it to yourself with the following example.

{% set x =
  [{'ip': '192.168.52.91',
    'mac': '00:1C:42:48:45:20'},
   {'ip': '193.168.52.170',
    'mac': 'A0:A3:B3:21:C1:60'}] %}
{{ x[0].mac | default('N/A') }}
{{ x[0].mac | default('N/A') is string}}
{{ x[0].foo | default('N/A') }}
{{ x[0].foo | default('N/A') is string }}

Here are the results:

00:1C:42:48:45:20
True
N/A
True

Datetime formatting

The technique you used to reformat the datetime string can be simplified. You should be able to achieve the same result using as_timestamp with timestamp_custom. Besides being shorter, it also converts the time from UTC to your local timezone.

Compare the following two results.

{# The Z in this timestamp means Zulu time which is equivalent to UTC #}
{% set dt = '2024-11-22T09:08:17.224Z' %}
{{ as_timestamp(strptime((dt | replace('T', ' ') | replace('Z', '')).split('.')[0], '%Y-%m-%d %H:%M:%S')) | timestamp_custom('%d/%m/%y %H:%M:%S') }}
{{ dt | as_timestamp | timestamp_custom('%d/%m/%y %H:%M:%S') }}

Here’s what I get:

22/11/24 09:08:17
22/11/24 04:08:17

• Your template reports the original UTC time (09:08:17).
• My suggested template adjusts the time to my local timezone which is UTC-5.

If you do not want to convert the time to your local timezone, you can simply add local=false to timestamp_custom.

{% set dt = '2024-11-22T09:08:17.224Z' %}
{{ dt | as_timestamp
  | timestamp_custom('%d/%m/%y %H:%M:%S', local=false) }}

[mirekmal (Mirek Malinowski)]

@123

It shouldn’t be necessary to use the string filter everywhere. All of the values (shown in your first post’s screenshot) are strings.

Agree, though I put it there as sort of safety measure… For some of model and make values, even if at first these looked as strings I was getting json corruption without explicit convertion to string.

@123

The technique you used to reformat the datetime string can be simplified. You should be able to achieve the same result using as_timestamp with timestamp_custom. Besides being shorter, it also converts the time from UTC to your local timezone.

Good catch! I was so happy finding this method somewhere on the forum, so I applied it instantly, without really checking for result. I mean it looked ‘OK-ish’ and being readible, but indeed result is -1h away from reality

[123 (Taras)]

@mirekmal

For some of model and make values, even if at first these looked as strings I was getting json corruption without explicit convertion to string.

• Do you have the data that caused JSON corruption? I would like to test it.

• When the corruption occured, were you using your technique or dict to produce a dictionary?

[mirekmal (Mirek Malinowski)]

Well, issue is for both codes. In case of your I narrowed it down to type key. Here is your original template, striped down to this one code and corresponsing output:

Note highligted item, this one does not have type defined in original output, it i s empty.

Here is the same code with string filter added:

In this case it is proper json and warning is most likely related to highlighted device, but this time undefined is rendered as empty string.

Additionally in case of my template problem was withmodel of one of devices:

{
    "order": 126,
    "mac": "A0:78:17:ED:85:17",
    "ip": "192.168.52.126",
    "state": "UP",
    "name": "iMac-Dorota-WiFi",
    "type": "DESKTOP",
    "make": "Apple",
    "model": "iMac (24\") M1 8-Core 3.2/8-Core",
    "first_seen": "08/07/21 18:21:08",
    "last_changed": "18/02/25 13:35:28"
  },

In output above your template properly convert " in the name to escped with \ character, while my template was retaining just " and it was disrupting flow of data into json.

Bottom line I updated all of the keys with string filter to be on safe side, but in reality I’d expect that only name, type, make and model fields can be emty and create some issues.

UPDATE: it seems that this empty key is created as such by template, in reality source sensor does not contain such key:

- mac: 02:11:32:21:9E:76
  ip:
    - 192.168.52.86
  state: UP
  name: SynologyVM (VirtualDSM)
  first_seen: '2021-01-13T18:14:28.998Z'
  last_changed: '2025-02-09T09:21:10.260Z'

[123 (Taras)]

@mirekmal

this one does not have type defined in original output, it i s empty

In the original output, is the type key missing or if it exists what is its value?

The reason why I ask is because if you attempt to reference a non-existent key, it will produce an error even if you use the string filter.

{% set d = {'color': 'red'} %}
{{ d.foo | string }}

The correct way to handle a non-existent key is with default.

{% set d = {'color': 'red'} %}
{{ d.foo | default('NA') }}

On the other hand, if the type key exists but its value is “empty”, I need you to explain what you mean by “empty”.

For example, an empty string is "" and doesn’t require any special handling (like a string filter).

The other possibility is the value is a null object (none) in which case, once again, it can be handled correctly with default.

This reports None because the foo key is defined so it simply reports the key’s value.

{% set d = {'color': 'red'}, 'foo': none} %}
{{ d.foo | default('NA') }}

This reports NA because the foo key is defined but its value is none.

{% set d = {'color': 'red'}, 'foo': none} %}
{{ d.foo | default('NA', true) }}

[mirekmal (Mirek Malinowski)]

It it not empty "", it does not exist in source - this is what the last screenshot shows, this is the state of source sensor that does not contain type key.
BUT, if I apply string filter, I get warning (2nd screenshot in my last post), not error (at least in developer tools) and the key gets created in output. The key is created with empty "" value. So thats why using it seems to somehow overcome the problem of dealing with undefined key and I get for every single device full set of keys that I can use in markdown card output tabl;e in UI without errors.

[123 (Taras)]

@mirekmal

this is what the last screenshot shows, this is the state of source sensor that does not contain type key.

This is the last screenshot from your previous post.

Each dictionary element contains a type key. For the highlighted element, the type key’s value is an empty string.

Where’s the element that does not contain a type key?

@mirekmal

BUT, if I apply string filter, I get warning (2nd screenshot in my last post), not error (at least in developer tools) and the key gets created in output. The key is created with empty "" value.

Your template should produce no warnings or errors. Relying on results that are produced despite a warning may result in failure in a later version of Home Assistant.

[mirekmal (Mirek Malinowski)]

Sorry, that was not screenshot but copy/paste of source from dev tools/states:

- mac: 02:11:32:21:9E:76
  ip:
    - 192.168.52.86
  state: UP
  name: SynologyVM (VirtualDSM)
  first_seen: '2021-01-13T18:14:28.998Z'
  last_changed: '2025-02-09T09:21:10.260Z'

So it is missing…

@123

Relying on results that are produced despite a warning may result in failure in a later version of Home Assistant.

So yes, in the final template I’m using now, it is based on your sample and including the default handling for keys that might be missing from source:

{% set devices = state_attr('sensor.all_network_devices', 'devices') %}
{% set ns = namespace(devices=[]) %}
{% for d in devices %}
  {% set ns.devices = ns.devices + [dict(
    order=d.ip[0].split('.')[3]|int(0),
    mac=d.mac,
    ip=d.ip[0],
    state=d.state,
    name=d.name|default("N/A")|string,
    type=d.type|default("N/A")|string,
    make=d.make|default("N/A")|string,
    model=d.model|default("N/A")|string,
    first_seen=d.first_seen| as_timestamp | timestamp_custom('%d/%m/%y %H:%M:%S'),
    last_changed=d.last_changed | as_timestamp | timestamp_custom('%d/%m/%y %H:%M:%S')
    )] %}
{% endfor %}
{{ ns.devices }}

I’ve been posting previous versions as you wanted to know what is causing the corruption of output of the template…

[123 (Taras)]

@mirekmal

So it is missing…

Then, as per my previous explanation, use default to handle that situation.

@mirekmal

type=d.type|default("N/A")|string

It appears that, despite my best efforts, I still haven’t convinced you that the use of string isn’t needed.

The received values are already strings and need no conversion to string. If the entire key is missing then default will report N/A which is a string and doesn’t need conversion to string.

If you want to guard against the remote possibility that a key’s value is a null object, then use this version of default.

type=d.type|default("N/A", true)

[mirekmal (Mirek Malinowski)]

Well, you convinced me, but it is like between ‘blowing on the already cold’ and doing things quickly, between soup and potato… too fast - too easy to make mistake