Parsing json (I think...)

finity · July 2, 2019, 11:43am

I Think I need some help from the python gurus on here.

I’ve been trying for a couple of days now to figure out why what I’m trying to do isn’t working and I’m obviously missing something.

I would like to take the output of a custom component that creates an attribute and parse the result to extract certain keys.

I’ve copied the text of the attribute exactly as it is into the template editor and I can dig into it and get out exactly what I want. So that’s all good.

But when I use the same syntax to manipulate the attribute directly the results are not the same and I can’t figure out why.

Here is a screen shot of what is successful. For brevity I’ve paired down the text in the json for easier reading but it is structurally identical to the actual text in the attribute.

However, when I use the same syntax on the real sensor I get:

the reason that I get that error is because when I try to use the index notation for index 0 the only thing it picks up is the ‘[’ symbol at the beginning of the string.

And for reference here is the actual text of the attribute. The author of the component hasn’t done any formatting on it so it’s just a wall of text that I had to format to even know how to try to extract what I wanted. But like I said above, it’s syntactically equal to the formatted one above.

[{"id": "https://api.weather.gov/alerts/NWS-IDP-PROD-3677132-3183041", "type": "Feature", "geometry": null, "properties": {"@id": "https://api.weather.gov/alerts/NWS-IDP-PROD-3677132-3183041", "@type": "wx:Alert", "id": "NWS-IDP-PROD-3677132-3183041", "areaDesc": "Western Kenai Peninsula", "geocode": {"UGC": ["AKZ121"], "SAME": ["002122"]}, "affectedZones": ["https://api.weather.gov/zones/forecast/AKZ121"], "references": [{"@id": "https://api.weather.gov/alerts/NWS-IDP-PROD-3676260-3182579", "identifier": "NWS-IDP-PROD-3676260-3182579", "sender": "[email protected]", "sent": "2019-07-01T03:22:00-08:00"}], "sent": "2019-07-01T15:39:00-08:00", "effective": "2019-07-01T15:39:00-08:00", "onset": "2019-07-01T15:39:00-08:00", "expires": "2019-07-02T10:00:00-08:00", "ends": "2019-07-03T13:00:00-08:00", "status": "Actual", "messageType": "Update", "category": "Met", "severity": "Moderate", "certainty": "Likely", "urgency": "Expected", "event": "Dense Smoke Advisory", "sender": "[email protected]", "senderName": "NWS Anchorage, AK", "headline": "Dense Smoke Advisory issued July 1 at 3:39PM AKDT until July 3 at 1:00PM AKDT by NWS Anchorage, AK", "description": "* LOCATION...Interior Kenai Peninsula, including the Sterling\nHighway corridor from Sterling to Cooper Landing.\n\n* SMOKE...Smoke from the Swan Lake fire will reduce visibilities\nto one quarter mile or less at times. The worst conditions\nwill be late at night through the morning hours.\n\n* TIMING...Through Wednesday, though there is potential for\ndense smoke to persist through the remainder of the week.\n\n* IMPACTS...Travel may be difficult due to low visibility.\nPersons with respiratory problems may have difficulty\nbreathing when outside.", "instruction": "A dense smoke advisory means widespread fires will create smoke,\nlimiting visibilities. If driving, slow down, use your\nheadlights, and leave plenty of distance ahead of you in case a\nsudden stop is needed. For the latest road conditions call\n5 1 1 or visit 511.alaska.gov.", "response": "Execute", "parameters": {"NWSheadline": ["DENSE SMOKE ADVISORY REMAINS IN EFFECT UNTIL 1 PM AKDT WEDNESDAY FOR THE INTERIOR KENAI PENINSULA"], "VTEC": ["/O.CON.PAFC.SM.Y.0003.000000T0000Z-190703T2100Z/"], "PIL": ["AERNPWAER"], "BLOCKCHANNEL": ["CMAS", "EAS", "NWEM"], "eventEndingTime": ["2019-07-03T13:00:00-08:00"]}}}, {"id": "https://api.weather.gov/alerts/NWS-IDP-PROD-3676668", "type": "Feature", "geometry": null, "properties": {"@id": "https://api.weather.gov/alerts/NWS-IDP-PROD-3676668", "@type": "wx:Alert", "id": "NWS-IDP-PROD-3676668", "areaDesc": "Western Prince William Sound; Anchorage; Copper River Basin; Susitna Valley; Matanuska Valley; Western Kenai Peninsula", "geocode": {"UGC": ["AKZ125", "AKZ101", "AKZ141", "AKZ145", "AKZ111", "AKZ121"], "SAME": ["002261", "002122", "002020", "002240", "002170", "002068", "002050"]}, "affectedZones": ["https://api.weather.gov/zones/forecast/AKZ125", "https://api.weather.gov/zones/forecast/AKZ101", "https://api.weather.gov/zones/forecast/AKZ141", "https://api.weather.gov/zones/forecast/AKZ145", "https://api.weather.gov/zones/forecast/AKZ111", "https://api.weather.gov/zones/forecast/AKZ121"], "references": [], "sent": "2019-07-01T11:58:00-08:00", "effective": "2019-07-01T11:58:00-08:00", "onset": "2019-07-01T11:58:00-08:00", "expires": "2019-07-03T12:00:00-08:00", "ends": null, "status": "Actual", "messageType": "Alert", "category": "Met", "severity": "Moderate", "certainty": "Observed", "urgency": "Expected", "event": "Special Weather Statement", "sender": "[email protected]", "senderName": "NWS Anchorage, AK", "headline": "Special Weather Statement issued July 1 at 11:58AM AKDT by NWS Anchorage, AK", "description": "Elevated water conditions will persist for Eagle River. For those\nnear or attempting to cross Eagle River, be prepared for very\nhigh and swift water. Please see weather.gov/aprfc for the latest\nconditions and forecasts.\n\nGenerally across Southcentral, high river conditions are expected\nalong the Matanuska, Skwentna, Klutina, Sixmile, Talkeetna, and Kenai\nrivers. Water levels are expected to continue and/or rise into\nAction Stage and Bankfull conditions through the weekend. No\nsignificant impacts are expected along these rivers.\n\nA Flood Warning has been issued for the Yentna River at Fish\nCreek. Please see the Flood Warning for that area for further\ndetails.", "instruction": "", "response": "Execute", "parameters": {"EAS-ORG": ["WXR"], "PIL": ["AERSPSAER"], "BLOCKCHANNEL": ["CMAS", "EAS", "NWEM"]}}}]

I’m beginning to think the problem is that it’s because the text isn’t properly formatted json. But if that’s the case then why does it work when I manipulate it after I copy it directly to the template editor and set it as a “value”?

I’ve tried to do split(), replace() regex_findall_index() to try to slice and dice it up to get to what I want but there are too many layers of “{”, “[” and “id” to get it worked out.

Any suggestions?

123 · July 2, 2019, 12:06pm

In the following example, I’m assuming the attribute’s name is alerts_string. Does this version produce the same error?

{{ state_attr('sensor.noaaa_alerts_akz121', 'alerts_string')[0].properties.event }}

FWIW, that’s what I used to test the template.

finity · July 2, 2019, 12:07pm

yes,

and yes.

finity · July 2, 2019, 12:16pm

I just saw your edit…

That’s what I saw above also. It works if you paste the text into a template and set some variable to that. (you used “x”; I used “value”) then parse that out.

But if it’s contained in the actual sensor attribute it doesn’t work the same way for some reason.

petro · July 2, 2019, 12:50pm

I’m guessing that it’s a string and not json data.

petro · July 2, 2019, 12:51pm

~~There’s a way to convert it but I’m on my phone and can’t find it atm~~ All the jinja tojson solutions only work in specific scenarios.

pnbruckner · July 2, 2019, 12:52pm

I think your problem is because the attribute is a string, not JSON. You even say so yourself:

In fact, even the name of the attribute – alerts_string – is a dead give away.

I just did a test. First I used the States page to create a test sensor with an attribute defined as such:

{"alerts": [{"id": ... }]}

And then in the Template editor:

pnbruckner · July 2, 2019, 12:54pm

What custom component are you using? The easiest fix would be to change it to create the attribute as JSON instead of a string.

123 · July 2, 2019, 1:05pm

For testing purposes, I created an MQTT Sensor:

  - platform: mqtt
    name: "test"
    state_topic: "test/sensor1"
    value_template: "{{ value_json.state }}"
    json_attributes_topic: "test/sensor1/attr"
    json_attributes_template: '{"alerts_string": {{value}}}'

Then I published what you listed as “the actual text of the attribute” to test/sensor1/attr. The attribute was created and now I could experiment with it:

The template I posted above extracted the desired result:

If it’s not working for you then maybe something works a bit differently with RESTful Sensors (like the data loses its ‘JSONness’).

petro · July 2, 2019, 1:07pm

To be honest, your test solution would be a decent solution without having to do anything like adjusting a component.

pnbruckner · July 2, 2019, 1:11pm

Your test is flawed. You’re using the JSON, not the “JSON in a string”, which is what the original attribute is. E.g., in your first test you did this:

{% set x = [{"id": ... }] %}

But that’s not what the attribute is. To do an equivalent test, you’d have to do this:

{% set x = '[{"id": ... }]' %}

You can’t tell the difference when the attribute is output, because when it’s a string, the quoting characters aren’t printed. So a string which contains something that looks like JSON outputs the same way as the JSON itself.

EDIT: Actually, they don’t, and that’s another clue…

123 · July 2, 2019, 1:17pm

I guess the question is “Why does finity’s attribute contain stringified JSON?”

With MQTT Sensor, I imported the payload into an attribute as JSON. How (and why) is the RESTful sensor creating the attribute’s value as a string?

pnbruckner · July 2, 2019, 1:18pm

Yes, that’s the question. But he said he’s using a custom component, not a RESTful sensor. So, @finity…???

123 · July 2, 2019, 1:20pm

Oops! My mistake; I missed that important detail.

pnbruckner · July 2, 2019, 1:35pm

Actually, I need to be more careful about how I use the terminology. JSON actually is a text representation of data, so effectively is a string. The main point here is that the attribute contains the “raw JSON”, instead of a Python data type that is created from parsing the JSON.

finity · July 2, 2019, 1:35pm

I’m using the noaa weather alerts custom component.

Tbh, I’m really not sure what the use case is for the component author to present the attribute of the sensor in that manner. But I figured if it was there I would try to find a use for it.

I think I figured out eventually that I wasn’t dealing with a properly formatted json. As mentioned, it’s darn hard to tell the difference until it just wouldn’t work and I figured there had to be something wierd.

But i also couldn’t even figure out how to slice it up to make it useful as a regular string either.

pnbruckner · July 2, 2019, 1:41pm

I looked at the code. Why aren’t you using the alerts attribute instead of the alerts_string attribute? The latter is a JSON formatted representation of the former.

{{ state_attr('sensor.X', 'alerts')[0].properties.event }}

finity · July 2, 2019, 9:50pm

I’m not sure what you’re asking here.

Are you saying that the author of the component should use “alerts” in the component code instead of “alerts_string” to get a properly formatted json string?

Or are you saying that the component already provides an “alerts” attribute that is properly formatted json?

If it’s the latter then I’m not seeing that attribute anywhere.

If it’s the former then I’m not sure why the author wrote it the way they did. I haven’t really dug into the code to see what could be changed there to make it work. I was just trying to figure out if I was doing something wrong on my end.

Another related question tho is why would there be a difference in the results in the template editor between copying/pasting the entire contents of the attribute into the template editor and setting that to some variable as opposed to just directly using the attribute itself?

As you said, json is just a string. So where does the distinction come from between the json string and the raw string? I thought it had to do with the formatting of the string itself that allowed it to be parsed as json (just like a string of numbers & symbols can be interpreted as a datetime object as long as it’s correctly formatted)? But if that was the case then neither method should work or both methods should work.

I’m not grasping the difference.

pnbruckner · July 2, 2019, 10:02pm

The way I read the code, there should be both an alerts attribute and an alerts_string attribute. The former is a Python list/dict, and you should be able to get at the pieces you want as I showed. The latter is that passed through json.dumps, which takes the list/dict and outputs it in a single string which is a JSON representation of the list/dict. (That’s why when you did [0], you just got the first character of the string, namely '['.)

In Jinja you use . and [] operators to get at pieces of dict's and list's, respectively.

What does the entity look like in the States page? Or if you do {{ states.sensor.WHATEVER }} in the Template editor?

pnbruckner · July 2, 2019, 10:27pm

Because you copied it as if it was a list/dict, not as a string. As I said above, you’d have to do {% set x = '...' %} to be equivalent.

There is no difference, and sorry if I confused you earlier. The alerts_string attribute contains a string, which happens to have a list/dict formatted as JSON.