RESTful sensor parse incoming data / array

Hi I have an energymanager (solar watt) that has provides the measurement as REST/JSON.
However the needed keys are stored in the JSON response object in a list/array (some kind of a list iteration for the connected devices)
My problem ist the Json keys I’m interested are not always in the same order, so not always on the same place in the Json list.
E.g. for PowerConsumedFromGrid:

 cat 1.json | jq -c 'paths | select(.[-1] == "PowerConsumedFromGrid")'
["result","items",8,"tagValues","PowerConsumedFromGrid"]

curl http://energymanager.local/rest/kiwigrid/wizard/devices |jq -c 'paths | select(.[-1] == "PowerConsumedFromGrid")'
["result","items",3,"tagValues","PowerConsumedFromGrid"]

So this Value is sometimes

value_template: "{{ value_json.result.items[8].tagValues.PowerConsumedFromGrid.value }}"

and sometimes:

value_template: "{{ value_json.result.items[8].tagValues.PowerConsumedFromGrid.value }}"

and maybe any other index in this litem ist.

The result json looks like:


    "result": {
        "items": [
            {
                "tagValues": {
                    "PowerConsumedFromGrid": {...}
                    },
                    "PowerOut": {...}
                    },
                    #  100 more

            },
            {
                "tagValues": {
                    "StateOfCharge": { ..}
                    },
            },
            # and so on    

Any hint how I can parse this with the restful integration?

There’s probably an elegant way to filter the list using selectattr() but I couldn’t figure it out. Here is a sledgehammer approach that just loops through every item in the list:

{% for item in value_json.result['items'] -%}
  {% if item.tagValues.PowerConsumedFromGrid is defined %}
    {{- item.tagValues.PowerConsumedFromGrid.value -}}
  {% endif %}
{%- endfor %}

You can use something like has(‘PowerConsumedFromGrid’) in JQ but as your data/result is half presented cannot help more

Please post full, valid JSON in future to save us having to manually recreate something. Assuming I’ve done it correctly:

{{ (value_json['result']['items']
    |selectattr('PowerConsumedFromGrid','defined')
    |map(attribute='PowerConsumedFromGrid')
    |first)['value'] }}

Hi thanks for the answers of far. And sorry not providing the full result json (its long and I didn’t want to spam here.)

So here is a (still shortened, full has 5k lines) sample (with the key I’m interested at the moment in item[5] & item[8]):

{
    "result": {
        "items": [
            {
                "tagValues": {
                    "Schedule": {
                        "tagName": "Schedule",
                        "guid": "urn:app:schedule:ERC05-000003672",
                        "value": []
                    },
                    "DegreeInclination": {
                        "tagName": "DegreeInclination",
                        "guid": "a53bcedb-1511-4ed2-afcc-e9e7563100f3",
                        "value": 5
                    },
                    "WorkAnnualYield": {
                        "tagName": "WorkAnnualYield",
                        "guid": "a53bcedb-1511-4ed2-afcc-e9e7563100f3",
                        "value": null
                    },
                    "PasswordLock": {
                        "tagName": "PasswordLock",
                        "guid": "a53bcedb-1511-4ed2-afcc-e9e7563100f3",
                        "value": null
                    }
                },
                "deviceModel": [
                    {
                        "deviceClass": "com.kiwigrid.devices.pvplant.PVPlant"
                    }
                ],
                "guid": "a53bcedb-1511-4ed2-afcc-e9e7563100f3"
            },
            {
                "tagValues": {
                    "PasswordLock": {
                        "tagName": "PasswordLock",
                        "guid": "urn:kiwigrid:gridflow:ERC05-000003672",
                        "value": null
                    },
                    "CurrentLimit": {
                        "tagName": "CurrentLimit",
                        "guid": "urn:kiwigrid:gridflow:ERC05-000003672",
                        "value": 50
                    },
                    "IdName": {
                        "tagName": "IdName",
                        "guid": "urn:kiwigrid:gridflow:ERC05-000003672",
                        "value": "GridFlow"
                    }
                },
                "deviceModel": [
                    {
                        "deviceClass": "com.kiwigrid.kiwiapp.gridflow.GridFlow"
                    }
                ],
                "guid": "urn:kiwigrid:gridflow:ERC05-000003672"
            },
            {
                "tagValues": {
                    "IdLabelSet": {
                        "tagName": "IdLabelSet",
                        "guid": "ERC05-000003672",
                        "value": null
                    },
                    "VersionExtensionsMap": {
                        "tagName": "VersionExtensionsMap",
                        "guid": "ERC05-000003672",
                        "value": {
                            "energymanager-rail": "7.23.0.0"
                        }
                    },
                    "URLCloud": {
                        "tagName": "URLCloud",
                        "guid": "ERC05-000003672",
                        "value": "auth.kiwigrid.com"
                    },
                    "StateVisibleIsSet": {
                        "tagName": "StateVisibleIsSet",
                        "guid": "ERC05-000003672",
                        "value": false
                    }
                },
                "deviceModel": [
                    {
                        "deviceClass": "com.kiwigrid.devices.em.EnergyManager"
                    }
                ],
                "guid": "ERC05-000003672"
            },
            {
                "tagValues": {
                    "IdProfileActive": {
                        "tagName": "IdProfileActive",
                        "guid": "urn:app:profile:ERC05-000003672",
                        "value": "1594299120866"
                    },
                    "IdModelCode": {
                        "tagName": "IdModelCode",
                        "guid": "urn:app:profile:ERC05-000003672",
                        "value": null
                    },
                    "IdPriorizedDeviceList": {
                        "tagName": "IdPriorizedDeviceList",
                        "guid": "urn:app:profile:ERC05-000003672",
                        "value": []
                    }
                },
                "deviceModel": [
                    {
                        "deviceClass": "com.kiwigrid.application.schedule.ProfileApp"
                    }
                ],
                "guid": "urn:app:profile:ERC05-000003672"
            },
            {
                "tagValues": {
                    "FactorForecastCAN": {
                        "tagName": "FactorForecastCAN",
                        "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7",
                        "value": "01 00 00 00 00 00 00 00 "
                    },
                    "VoltageBatteryCellMean": {
                        "tagName": "VoltageBatteryCellMean",
                        "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7",
                        "value": 4.005
                    },
                    "PowerACInMax": {
                        "tagName": "PowerACInMax",
                        "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7",
                        "value": null
                    },
                    "TimeEqualizingChargeRemaining": {
                        "tagName": "TimeEqualizingChargeRemaining",
                        "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7",
                        "value": null
                    },
                    "TemperatureBattery": {
                        "tagName": "TemperatureBattery",
                        "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7",
                        "value": 23
                    },
                    "VoltageBatteryString": {
                        "tagName": "VoltageBatteryString",
                        "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7",
                        "value": 96.09999999999997
                    },
                    "PowerACIn": {
                        "tagName": "PowerACIn",
                        "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7",
                        "value": 118.97179999999996
                    },
                    "StateOfHealth": {
                        "tagName": "StateOfHealth",
                        "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7",
                        "value": 96
                    },
                    "StateOfCharge": {
                        "tagName": "StateOfCharge",
                        "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7",
                        "value": 82.5
                    },
                    "TemperatureBatteryCellMax": {
                        "tagName": "TemperatureBatteryCellMax",
                        "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7",
                        "value": 36
                    },
                    "ModeConverter": {
                        "tagName": "ModeConverter",
                        "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7",
                        "value": "CHARGING"
                    },
                    "VoltageBatteryCellMax": {
                        "tagName": "VoltageBatteryCellMax",
                        "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7",
                        "value": 4.008
                    },
                    "CurrentBatteryOut": {
                        "tagName": "CurrentBatteryOut",
                        "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7",
                        "value": 0
                    }
                },
                "deviceModel": [
                    {
                        "deviceClass": "com.kiwigrid.devices.solarwatt.MyReserve"
                    }
                ],
                "guid": "urn:solarwatt:myreserve:bc:a30b000b4cc7"
            },
            {
                "tagValues": {
                    "PowerOut": {
                        "tagName": "PowerOut",
                        "guid": "4383bfde-a779-4a2b-a8da-b2c48ef3f393",
                        "value": 16
                    },
                    "DirectionMetering": {
                        "tagName": "DirectionMetering",
                        "guid": "4383bfde-a779-4a2b-a8da-b2c48ef3f393",
                        "value": "BIDIRECTIONAL"
                    },
                     "ActivePowerL1": {
                        "tagName": "ActivePowerL1",
                        "guid": "4383bfde-a779-4a2b-a8da-b2c48ef3f393",
                        "value": null
                    }
                },
                "deviceModel": [
                    {
                        "deviceClass": "com.kiwigrid.devices.solarwatt.MyReservePowermeter"
                    }
                ],
                "guid": "4383bfde-a779-4a2b-a8da-b2c48ef3f393"
            },
            {
                "tagValues": {
                    "StateDevice": {
                        "tagName": "StateDevice",
                        "guid": "urn:solarwatt:myreserve:pv:a30b000b4cc7",
                        "value": "OK"
                    },
                    "ACPower": {
                        "tagName": "ACPower",
                        "guid": "urn:solarwatt:myreserve:pv:a30b000b4cc7",
                        "value": null
                    },
                    "WorkACOut": {
                        "tagName": "WorkACOut",
                        "guid": "urn:solarwatt:myreserve:pv:a30b000b4cc7",
                        "value": 2.5216984823312838E7
                    },
                    "PowerACOut": {
                        "tagName": "PowerACOut",
                        "guid": "urn:solarwatt:myreserve:pv:a30b000b4cc7",
                        "value": 455.0804
                    }
                },
                "deviceModel": [
                    {
                        "deviceClass": "com.kiwigrid.devices.inverter.Inverter"
                    },
                    {
                        "deviceClass": "com.kiwigrid.devices.solarwatt.MyReserveInverter"
                    }
                ],
                "guid": "urn:solarwatt:myreserve:pv:a30b000b4cc7"
            },
            {
                "tagValues": {
                    "DatePowerConsumptionForecastStart": {
                        "tagName": "DatePowerConsumptionForecastStart",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 1698054300000
                    },
                    "PowerOut": {
                        "tagName": "PowerOut",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 16
                    },
                    "PowerBuffered": {
                        "tagName": "PowerBuffered",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 118.97179999999996
                    },
                    "PowerConsumedFromGrid": {
                        "tagName": "PowerConsumedFromGrid",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 0
                    },
                    "PowerConsumedFromProducers": {
                        "tagName": "PowerConsumedFromProducers",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 320.1086
                    },
                    "WorkOutFromStorage": {
                        "tagName": "WorkOutFromStorage",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 23071.903247226728
                    },
                    "PowerOutFromStorage": {
                        "tagName": "PowerOutFromStorage",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 0
                    },
                    "PowerConsumed": {
                        "tagName": "PowerConsumed",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 320.1086
                    },
                    "PowerIn": {
                        "tagName": "PowerIn",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 0
                    },
                    "PowerProduced": {
                        "tagName": "PowerProduced",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 455.0804
                    },
                    "WorkConsumedFromGrid": {
                        "tagName": "WorkConsumedFromGrid",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 7804835.357727677
                    },
                    "WorkConsumed": {
                        "tagName": "WorkConsumed",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 1.688301133289597E7
                    },
                    "WorkBuffered": {
                        "tagName": "WorkBuffered",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 3949011.6485447367
                    },
                    "PowerBufferedFromProducers": {
                        "tagName": "PowerBufferedFromProducers",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 118.97179999999996
                    },
                    "PowerConsumptionForecastNow": {
                        "tagName": "PowerConsumptionForecastNow",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": null
                    },
                    "WorkSelfSupplied": {
                        "tagName": "WorkSelfSupplied",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 9078175.975168308
                    },
                    "WorkConsumedFromStorage": {
                        "tagName": "WorkConsumedFromStorage",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 3506157.781874271
                    },
                    "PowerConsumedFromStorage": {
                        "tagName": "PowerConsumedFromStorage",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 0
                    },
                    "WorkOut": {
                        "tagName": "WorkOut",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 1.5980751716239212E7
                    },
                    "PowerOutFromProducers": {
                        "tagName": "PowerOutFromProducers",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 16.000000000000014
                    },
                    "PowerSelfSupplied": {
                        "tagName": "PowerSelfSupplied",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 320.1086
                    },
                    "WorkBufferedFromGrid": {
                        "tagName": "WorkBufferedFromGrid",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 125922.30329505456
                    },
                    "WorkOutFromProducers": {
                        "tagName": "WorkOutFromProducers",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 1.5821395509547807E7
                    },
                    "PowerBufferedFromGrid": {
                        "tagName": "PowerBufferedFromGrid",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 0
                    },
                    "WorkProduced": {
                        "tagName": "WorkProduced",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 2.5216503048092782E7
                    },
                    "PowerSelfConsumed": {
                        "tagName": "PowerSelfConsumed",
                        "guid": "31cea01c-8489-4a63-b58f-6d20c193728e",
                        "value": 439.0804
                    },
                },
                "deviceModel": [
                    {
                        "deviceClass": "com.kiwigrid.lib.device.Device"
                    },
                    {
                        "deviceClass": "com.kiwigrid.devices.location.Location"
                    }
                ],
                "guid": "31cea01c-8489-4a63-b58f-6d20c193728e"
            },
            {
                "tagValues": {
                    "IdModelCode": {
                        "tagName": "IdModelCode",
                        "guid": "urn:sunspec:fronius:inverter:30355065",
                        "value": "Symo 10.0-3-M"
                    },
                    "PowerACOut": {
                        "tagName": "PowerACOut",
                        "guid": "urn:sunspec:fronius:inverter:30355065",
                        "value": 324
                    },
                    "WorkACOut": {
                        "tagName": "WorkACOut",
                        "guid": "urn:sunspec:fronius:inverter:30355065",
                        "value": 2.9161991248472214E7
                    }
                },
                "deviceModel": [
                    {
                        "deviceClass": "com.kiwigrid.devices.sunspec.SunSpecInverter"
                    }
                ],
                "guid": "urn:sunspec:fronius:inverter:30355065"
            }
        ]
    }
}

I my try was first something like (in restful.yaml that I import to config)

 - resource: http://192.168.1.13/rest/kiwigrid/wizard/devices
   scan_interval: 120
   sensor:
     - name: "energymanager_erzeugung_pv"
       icon: mdi:solar-power-variant
       value_template: "{{ value_json.result.items[8].tagValues.PowerProduced.value | round(0)}}"
       state_class: measurement
       unit_of_measurement: "W"
       device_class: power
     - name: "energymanager_direktverbrauch"
       icon: mdi:home-lightning-bolt-outline
       value_template: "{{ value_json.result.items[8].tagValues.PowerConsumed.value | round(0)}}"
       state_class: measurement
       unit_of_measurement: "W"
       device_class: power

However, it seems that the device generates the result.items[]. list in a random order.

The interesting keys / values are:

“PowerProduced” , .result.items[8].tagValues.PowerProduced.value,
“PowerConsumed”, .result.items[8].tagValues.PowerConsumed.value,
“PowerOut”, .result.items[8].tagValues.PowerOut.value,
“PowerConsumedFromGrid”, .result.items[8].tagValues.PowerConsumedFromGrid.value,
“PowerBuffered”, .result.items[8].tagValues.PowerBuffered.value,
“PowerConsumedFromStorage”, .result.items[8].tagValues.PowerConsumedFromStorage.value,
“StateOfCharge”, .result.items[5].tagValues.StateOfCharge.value,
“ModeConverter”, .result.items[5].tagValues.ModeConverter.value’

I was thinking about something similar to the “brute force for loop” but doing it only once. (Since I need to create more entities with the single REST call)
So first iterate overItems[] and store all keys / values only once after the rest call and than just get the needed key in the sensor
(but I can’t get this to work)

{% set all_items = {} %}
{% for item in value_json.result['items'] %}
  {% for tag in item.tagValues  %}
    {# all_items[tag] = item.tagValues.[tag].value #}
    {{ tag }} ':' {{ item.tagValues.[tag] }}
  {% endfor %}
{% endfor %}
value_template: "{{ all_items['PowerConsumed']}}"

I need to study the selectattr('PowerConsumedFromGrid','defined') samples

Just to mention, in jq works :

jq -c '(
"PowerProduced", .result.items[].tagValues.PowerProduced.value,
"PowerConsumed", .result.items[].tagValues.PowerConsumed.value 
) | select(.!=null)'
     - name: "energymanager_direktverbrauch"
       icon: mdi:home-lightning-bolt-outline
       value_template: "{{ (value_json['result']['items']|selectattr('tagValues.PowerConsumed','defined')|list|first)['tagValues']['PowerConsumed']['value']|round(0) }}"
       state_class: measurement
       unit_of_measurement: "W"
       device_class: power

…and so on for the others. Just replace the two key names in each template. Same thing spaced for better readability:

     - name: "energymanager_direktverbrauch"
       icon: mdi:home-lightning-bolt-outline
       value_template: >
         {{ (value_json['result']['items']
             |selectattr('tagValues.PowerConsumed','defined')
             |list|first)
            ['tagValues']['PowerConsumed']['value'] |round(0) }}
       state_class: measurement
       unit_of_measurement: "W"
       device_class: power
2 Likes