Parsing complex JSON schema error

Configuration
1

I’m trying to create a REST sensor to pull data about the river conditions near my home.

The URL I’m using is http://waterservices.usgs.gov/nwis/iv/?sites=13206305&parameterCd=00060&format=json

The data returned by that GET request is in the following format:

{
	"name": "ns1:timeSeriesResponseType",
	"declaredType": "org.cuahsi.waterml.TimeSeriesResponseType",
	"scope": "javax.xml.bind.JAXBElement$GlobalScope",
	"value": {
		"queryInfo": {
			"queryURL": "http://waterservices.usgs.gov/nwis/iv/sites=13206305&parameterCd=00060&format=json",
			"criteria": {
				"locationParam": "[ALL:13206305]",
				"variableParam": "[00060]",
				"parameter": []
			},
			"note": [{
				"value": "[ALL:13206305]",
				"title": "filter:sites"
			}, {
				"value": "[mode=LATEST, modifiedSince=null]",
				"title": "filter:timeRange"
			}, {
				"value": "methodIds=[ALL]",
				"title": "filter:methodId"
			}, {
				"value": "2016-12-03T07:00:13.669Z",
				"title": "requestDT"
			}, {
				"value": "237d2a40-b926-11e6-a4b5-6cae8b6642f6",
				"title": "requestId"
			}, {
				"value": "Provisional data are subject to revision. Go to http://waterdata.usgs.gov/nwis/help/?provisional for more information.",
				"title": "disclaimer"
			}, {
				"value": "caas01",
				"title": "server"
			}]
		},
		"timeSeries": [{
			"sourceInfo": {
				"siteName": "BOISE RIVER SOUTH CHANNEL AT EAGLE ID",
				"siteCode": [{
					"value": "13206305",
					"network": "NWIS",
					"agencyCode": "USGS"
				}],
				"timeZoneInfo": {
					"defaultTimeZone": {
						"zoneOffset": "-07:00",
						"zoneAbbreviation": "MST"
					},
					"daylightSavingsTimeZone": {
						"zoneOffset": "-06:00",
						"zoneAbbreviation": "MDT"
					},
					"siteUsesDaylightSavingsTime": true
				},
				"geoLocation": {
					"geogLocation": {
						"srs": "EPSG:4326",
						"latitude": 43.67527778,
						"longitude": -116.353611
					},
					"localSiteXY": []
				},
				"note": [],
				"siteType": [],
				"siteProperty": [{
					"value": "ST",
					"name": "siteTypeCd"
				}, {
					"value": "17050114",
					"name": "hucCd"
				}, {
					"value": "16",
					"name": "stateCd"
				}, {
					"value": "16001",
					"name": "countyCd"
				}]
			},
			"variable": {
				"variableCode": [{
					"value": "00060",
					"network": "NWIS",
					"vocabulary": "NWIS:UnitValues",
					"variableID": 45807197,
					"default": true
				}],
				"variableName": "Streamflow, ft³/s",
				"variableDescription": "Discharge, cubic feet per second",
				"valueType": "Derived Value",
				"unit": {
					"unitCode": "ft3/s"
				},
				"options": {
					"option": [{
						"name": "Statistic",
						"optionCode": "00000"
					}]
				},
				"note": [],
				"noDataValue": -999999.0,
				"variableProperty": [],
				"oid": "45807197"
			},
			"values": [{
				"value": [{
					"value": "204",
					"qualifiers": ["P"],
					"dateTime": "2016-12-02T23:30:00.000-07:00"
				}],
				"qualifier": [{
					"qualifierCode": "P",
					"qualifierDescription": "Provisional data subject to revision.",
					"qualifierID": 0,
					"network": "NWIS",
					"vocabulary": "uv_rmk_cd"
				}],
				"qualityControlLevel": [],
				"method": [{
					"methodDescription": "",
					"methodID": 47585
				}],
				"source": [],
				"offset": [],
				"sample": [],
				"censorCode": []
			}],
			"name": "USGS:13206305:00060:00000"
		}]
	},
	"nil": false,
	"globalScope": true,
	"typeSubstituted": false
}

I’m trying to get the value of 204 extracted out.

My sensor looks like this:

sensor river:
  - platform: rest
    name: River Flow
    resource: http://waterservices.usgs.gov/nwis/iv/?    sites=13206305&parameterCd=00060&format=json
    value_template: '{{ value_json.value.timeSeries.[0].values.[0].value.[0].value }}'
    unit_of_measurement: "CFS"

I can’t see an error with it, but it spits this out in the home-assistant.log file:

homeassistant.bootstrap: Invalid config for [sensor.rest]: invalid template (TemplateSyntaxError: expected name or number) for dictionary value @ data[‘value_template’]. Got ‘{{ value_json.value.timeSeries.[0].values.[0].value.[0].value }}’. (See ?:?). Please check the docs at https://home-assistant.io/components/sensor.rest/

Can anyone help me understand what I’ve got incorrect?

2

Your format seems off. Try something like '{{ value_json.value.timeSeries[0].values[0] ... }}' for the value_template.

3

Alas, that didn’t work. I tried:

sensor river:
  - platform: rest
    name: River Flow
    resource: http://waterservices.usgs.gov/nwis/iv/?sites=13206305&parameterCd=00060&format=json
    value_template: '{{ value_json.value.timeSeries[0].values[0].value[0].value }}'
    unit_of_measurement: "CFS"

And got this entry in the log:

homeassistant.helpers.template: Error parsing value: builtin_function_or_method object has no element 0 (value: {"name":"ns1:timeSeriesResponseType","declaredType":"org.cuahsi.waterml.TimeSeriesResponseType","scope":"javax.xml.bind.JAXBElement$GlobalScope","value":{"queryInfo":{"queryURL":"http://waterservices.usgs.gov/nwis/iv/sites=13206305&parameterCd=00060&format=json","criteria":{"locationParam":"[ALL:13206305]","variableParam":"[00060]","parameter":[]},"note":[{"value":"[ALL:13206305]","title":"filter:sites"},{"value":"[mode=LATEST, modifiedSince=null]","title":"filter:timeRange"},{"value":"methodIds=[ALL]","title":"filter:methodId"},{"value":"2016-12-03T17:37:30.867Z","title":"requestDT"},{"value":"2aa28d20-b97f-11e6-8de7-6cae8b6642ea","title":"requestId"},{"value":"Provisional data are subject to revision. Go to http://waterdata.usgs.gov/nwis/help/?provisional for more information.","title":"disclaimer"},{"value":"sdas01","title":"server"}]},"timeSeries":[{"sourceInfo":{"siteName":"BOISE RIVER SOUTH CHANNEL AT EAGLE ID","siteCode":[{"value":"13206305","network":"NWIS","agencyCode":"USGS"}],"timeZoneInfo":{"defaultTimeZone":{"zoneOffset":"-07:00","zoneAbbreviation":"MST"},"daylightSavingsTimeZone":{"zoneOffset":"-06:00","zoneAbbreviation":"MDT"},"siteUsesDaylightSavingsTime":true},"geoLocation":{"geogLocation":{"srs":"EPSG:4326","latitude":43.67527778,"longitude":-116.353611},"localSiteXY":[]},"note":[],"siteType":[],"siteProperty":[{"value":"ST","name":"siteTypeCd"},{"value":"17050114","name":"hucCd"},{"value":"16","name":"stateCd"},{"value":"16001","name":"countyCd"}]},"variable":{"variableCode":[{"value":"00060","network":"NWIS","vocabulary":"NWIS:UnitValues","variableID":45807197,"default":true}],"variableName":"Streamflow, ft³/s","variableDescription":"Discharge, cubic feet per second","valueType":"Derived Value","unit":{"unitCode":"ft3/s"},"options":{"option":[{"name":"Statistic","optionCode":"00000"}]},"note":[],"noDataValue":-999999.0,"variableProperty":[],"oid":"45807197"},"values":[{"value":[{"value":"189","qualifiers":["P"],"dateTime":"2016-12-03T10:30:00.000-07:00"}],"qualifier":[{"qualifierCode":"P","qualifierDescription":"Provisional data subject to revision.","qualifierID":0,"network":"NWIS","vocabulary":"uv_rmk_cd"}],"qualityControlLevel":[],"method":[{"methodDescription":"","methodID":47585}],"source":[],"offset":[],"sample":[],"censorCode":[]}],"name":"USGS:13206305:00060:00000"}]},"nil":false,"globalScope":true,"typeSubstituted":false}, template: {{ value_json.value.timeSeries[0].values[0].value[0].value }})

4

Well, I’m stumped. The error message in the log indicates there is no element 0 for at least one of the three arrays I’ve specified in my sensor. But each of timeSeries[0], values[0], and value[0] is definitely pointing to an array, as far as I can see.

Does anyone know of any resource available online to test this kind of query against sample data? Some tutorial on JSON formatting or jinja templating? Google hasn’t been a friend, here.

5

You are not alone. I arrived at the same problem pulling from another USGS river gage. Here is the URL I’m interested in:

http://waterservices.usgs.gov/nwis/dv/?format=json&sites=02312000&parameterCd=00065

Has anyone had any luck pulling current conditions from USGS sensors such as the current gage height?

6

I tried and failed to reply via email (maybe it will show up later), but is the problem that USGS uses the key “value” which looks to the parser like a “builtin_function_or_method”? The particular error message makes me suspicious:

Error parsing value: builtin_function_or_method object has no element 0

7

Have you tried this:

{{ value_json.value.timeSeries.values.value.value }}

Everything in that json file is a single dictionary in an array. For shits and giggles, try getting:

{{ value_json.value.queryInfo.queryURL }}

then

{{ value_json.value.queryInfo.note[0].value }}

Note looks like the only object that has more than 1 item in the array. I don’t know, just tossing ideas.

8

No dice:

Error parsing value: ‘list object’ has no attribute ‘values’

9

Sensor value is now “http://waterservices.usgs.gov/nwis/iv/format=json&sites=13206305&parameterCd=00060&siteStatus=all”

10

Sensor value is now
“[ALL:13206305]”

11

Based on the error messages I get as I try different things, it’s clear that the “values” item is the problem. If I leave the array [0] off of the timeSeries item, the error message is "Error parsing value: ‘list object’ has no attribute ‘values’ "

If I include the array [0] indicator for timeSeries, but leave it off of the “values” item, I get the error message
Error parsing value: ‘builtin_function_or_method object’ has no attribute ‘value’ "

If I include the array [0] indicator for timeSeries, and add it to the “values” item, I get the error message “Error parsing value: builtin_function_or_method object has no element 0”.

So the error message refers to timeSeries as ‘list object’ but refers to “values” as ‘builtin_function_or_method object’

I think the issue is the use of the term “values” in the USGS data. Is there a way to specify the JSON item I’m interested in other than using its name? Maybe what order it appears in the array, somehow?

12

You may need to use jinja to get the timeSeries object and then get all its members as a variable and use that variable to access the rest.

13

{% set n = value_json.value.timeSeries %}
{% for name in n.something something %}
etc…

Don’t know if that’s even possible, but it may be worth investigating.

14

also, you may want to try:

{{ value_json.value.timeSeries[0]['values'][0]['value'][0]['value'] }}

Again, no clue if that will work, but that’s a common way to treat dictionary items

EDIT: After revisiting this after ~5 years, the issue is that dictionaries have a built in method named values(). This returns all values in the dictionary as key valued pairs. The dictionary provided by the rest sensor also has a key named values. When using the dot notation, values() takes precedence over values the key. Hence the errors. Using ['values'] ensures that you’re getting the key, not the method.

With that being said, the following would also work.

{{ value_json.value.timeSeries[0]['values'][0].value[0].value }}
2 Likes
15

That’s the one! I was trying to figure out the bracket convention from the jinja documentation, but couldn’t work out how to get the arrays and dictionaries to play nice. Thanks for the help!

For future reference, the whole phrase had to go into double quotes, the single quotes it was in didn’t like the single quotes around ‘values’ and ‘value’

16

I know this is an old thread, but wanted to say Thank You to you both for the question and the solution! I was able to use this for my local USGS data as well and it worked perfectly! Thanks!

1 Like
17

Old thread bump! I was never able to get the template sensor working without errors for my USGS pull. However, I was able to get it via Node Red, and then push the value from Node Red to HA via MQTT. More steps for sure, but it works! Only thing HA lacks for USGS data is the proper unit of measurement for flow - I don’t think there is Cu. feet per second, which is what all the USGS data reports.

Here is the Node Red flow:

[{"id":"4df2865f.f597b8","type":"www-request","z":"f20f1b13.5e8608","name":"USGS Petersburg - FLOW","method":"GET","ret":"obj","url":"https://waterservices.usgs.gov/nwis/iv/?format=json&sites=01606500&parameterCd=00060","follow-redirects":true,"persistent-http":true,"tls":"","x":1120,"y":4580,"wires":[["d3e86a8f.1fe528"]]},{"id":"d5886142.001ae","type":"inject","z":"f20f1b13.5e8608","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"1800","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":870,"y":4600,"wires":[["4df2865f.f597b8","d8760784.9d40f8"]]},{"id":"d3e86a8f.1fe528","type":"function","z":"f20f1b13.5e8608","name":"","func":"msg.topic = \"homeassistant/usgs/petersburg/fps\"\nmsg.payload = msg.payload.value.timeSeries[0].values[0].value[0].value\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1360,"y":4580,"wires":[["d8c86073.9d333","e926bf2b.bf5ca"]]},{"id":"d8c86073.9d333","type":"mqtt out","z":"f20f1b13.5e8608","name":"homeassistant/usgs/petersburg/fps","topic":"","qos":"","retain":"","respTopic":"","contentType":"","userProps":"","correl":"","expiry":"","broker":"5e2c7825.f0b088","x":1620,"y":4580,"wires":[]},{"id":"d8760784.9d40f8","type":"www-request","z":"f20f1b13.5e8608","name":"USGS Petersburg - GAGE","method":"GET","ret":"obj","url":"https://waterservices.usgs.gov/nwis/iv/?format=json&sites=01606500&parameterCd=00065","follow-redirects":true,"persistent-http":true,"tls":"","x":1120,"y":4640,"wires":[["684fd105.c509d"]]},{"id":"684fd105.c509d","type":"function","z":"f20f1b13.5e8608","name":"","func":"msg.topic = \"homeassistant/usgs/petersburg/gage\"\nmsg.payload = msg.payload.value.timeSeries[0].values[0].value[0].value\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1360,"y":4640,"wires":[["b109596b.426128","dacaee31.d1a2"]]},{"id":"b109596b.426128","type":"mqtt out","z":"f20f1b13.5e8608","name":"homeassistant/usgs/petersburg/gage","topic":"","qos":"","retain":"","respTopic":"","contentType":"","userProps":"","correl":"","expiry":"","broker":"5e2c7825.f0b088","x":1630,"y":4640,"wires":[]},{"id":"5e2c7825.f0b088","type":"mqtt-broker","name":"MQTT","broker":"192.168.1.237","port":"1883","clientid":"node_red_mqqt_client","usetls":false,"protocolVersion":"4","keepalive":"60","cleansession":true,"birthTopic":"nodered_mqtt","birthQos":"0","birthRetain":"true","birthPayload":"node red MQTT birth msg","birthMsg":{},"closeTopic":"nodered_mqtt","closeQos":"0","closeRetain":"true","closePayload":"node red MQTT death msg","closeMsg":{},"willTopic":"nodered_mqtt","willQos":"0","willRetain":"true","willPayload":"node red MQTT unexpected death msg","willMsg":{},"sessionExpiry":""}]

18

Sooo… would someone mind pasting in everything from your configuration.yaml where you got this working? I’m trying to create a similar sensor, just for a different station… and I can’t get past the error messages that OP was getting.

19

After messing around awhile longer, I think I got it finally. I was getting tripped up on the sensor syntax…I feel like I haven’t fully grokked the REST sensor documentation but I was able to get what I wanted working, with a vague sense of unease that I’ve done it wrong.

I wanted a different value for stream data - OP was getting stream flow, I wanted gage height, so I slightly altered my query string changing the parameterCD from “00060” to “00065”. From there, here is my configuration.yaml entry:

sensor riverheight:
  - platform: rest
    ###resource: https://waterservices.usgs.gov/nwis/iv/?sites=13206305&parameterCd=00060&format=json
    resource: https://waterservices.usgs.gov/nwis/iv/?sites=13206305&indent=on&parameterCd=00065&format=json
    value_template: "{{ value_json.value.timeSeries[0]['values'][0]['value'][0]['value'] }}"
    name: riverheight
    unit_of_measurement: F
 
sensor:
  - platform: bhyve
  - platform: template
    sensors:
      officetemp:
        value_template: "{{ states('sensor.officespaceheater_temperature') }}"
        unit_of_measurement: '°F'
        friendly_name: "Office Space Heater Temp"
        icon_template: >-
          {% if is_state('switch.officespaceheater','on') %}
            mdi:radiator
          {% else %}
            mdi:radiator-off
          {% endif %}
1 Like