Problems to extract value from website of json format

Savanna_Chow · May 12, 2020, 10:12am

Hi,

Can some one help me to extract value from a website? This website represent local air-quality values in json format:

https://pm25.lass-net.org/data/last.php?device_id=74DA38F20824

I’m trying to get the particle matter value (which is “s_d0”) under “feeds” - “Aribox” of the website json results. So I add a sensor in the configuration.yaml file as below:

sensor:

platform: rest
resource: https://pm25.lass-net.org/data/last.php?device_id=74DA38F20824
name: PM2.5
value_template: ‘{{value_json.feeds[0].Airbox.s_d0}}’

However, I get no results fro the sensor, it just show up “unknown” from the sensor state. I check most of the discussion on foru, but still got no clues on how to get the correct value. No matter how I try I still get “unknown”

Would someone please help me with this?

Thanks~

sender · May 12, 2020, 10:19am

edit: if you test your url via https://jsonformatter.curiousconcept.com/

something seems wrong there.

amaximus · May 12, 2020, 10:30am

Not sure if it’s a typo or wrong rendering, but you should use ’ (ticks) on both sides. Now the starting one seems ` (back tick).

Savanna_Chow · May 12, 2020, 10:57am

Hmm…I checked, I use tick on both side, not back tick

I don’t knownwhy it looks like back tick when i post here, but in my configuration.yaml it’s tick.

Troon · May 12, 2020, 11:05am

The JSON from that URI is more than 255 characters long. See here for next steps.

amaximus · May 12, 2020, 11:51am

Try then:

feeds[0]['AirBox']['s_d0']

Savanna_Chow · May 13, 2020, 7:16am

I tired to use json_attributes, but still geo unavailable

The json website is here:

https://pm25.lass-net.org/data/last.php?device_id=74DA38F20824

{"source": "last by IIS-NRL", "feeds": [{"AirBox": {"gps_num": 9.0, "s_d0_hourly": 4.222222222222222, "app": "AirBox", "gps_alt": 2.0, "s_d2": 3.0, "c_d0_method": "BRR/30/6.84", "s_d0": 5.0, "s_d1": 5.0, "s_h0": 91.0, "addr": "[262]\u5b9c\u862d\u7e23\u7901\u6eaa\u9109\u8305\u57d4\u8def24\u865f", "area": "yilan", "SiteName": "\u5b9c\u862d\u7e23\u7e23\u7acb\u7389\u7530\u570b\u5c0f", "gps_fix": 1.0, "c_d0_source": "AS-IISNRL", "gps_lat": 24.795, "s_t0": 24.12, "timestamp": "2020-05-12T06:43:57Z", "gps_lon": 121.79, "SiteAddr": "[262]\u5b9c\u862d\u7e23\u7901\u6eaa\u9109\u8305\u57d4\u8def24\u865f", "date": "2020-05-12", "device_id": "74DA38F20824", "name": "\u7389\u7530\u570b\u5c0f", "c_d0": 2.63, "s_d0_hourly_num": 9, "time": "06:43:57"}}], "version": "2020-05-12T06:52:59Z", "num_of_records": 1, "device_id": "74DA38F20824"}

here is my configuration.yaml:

sensor:

  - platform: rest
    resource: https://pm25.lass-net.org/data/last.php?device_id=74DA38F20824
    name: yilan
    json_attributes:
      - "feeds"
    value_template: '{{ value_json["device_id"] }}'

  - platform: template
    sensors:
      pm25_yilan:
        value_template: '{{ state_attr("sensor.yilan", "feeds")[0]["Airbox"]["s_d0"] }}'
        entity_id: sensor.yilan

I also tried to use
value_template: '{{ state_attr("sensor.yilan", "feeds")[0]["Airbox"]["s_d0"] }}'
or
value_template: '{{ state_attr("sensor.yilan", "feeds")[0]["Airbox"].s_d0 }}'

but none of these works. I still can’t get correct value of s_d0

Can someone help me to solve this?

Savanna_Chow · May 13, 2020, 7:20am

I also found the attributes of entity “sensor.yilan” shows correct data retrieved form the website, but when I want to extract values like “s_d0” to [sensor.pm25_yilan] it just shows unavailable.

Troon · May 13, 2020, 7:44am

How about:

{{ (states("sensor.yilan")|from_json())["feeds"][0]["AirBox"]["s_d0"] }}

If that doesn’t work, I still think you are hitting the 255 character limit.

Savanna_Chow · May 13, 2020, 8:35am

no, it doesn’t work, but this time it shows “unknown” instead of unavailable

this json webstie seems to be very simple and should be easy to get data out, but I don’t understand why I still cannot figure out a solution, i already search for documentation and examples from the forum…

Troon · May 13, 2020, 9:18am

I’ve just set up this sensor in my own configuration, and this is from the logs:

homeassistant.exceptions.InvalidStateError: Invalid state encountered for entity id: sensor.yilan. State max length is 255 characters.

I don’t know a pure HA way around this problem yet, but I have solved a similar issue using command line sensors.

A quick one to extract just that s_d0 number is:

- platform: command_line
  command: "curl -s https://pm25.lass-net.org/data/last.php?device_id=74DA38F20824 | jq '.[\"feeds\"][0][\"AirBox\"][\"s_d0\"]'"
  name: yilan airbox s_d0

You’ll need to ensure you have curl and jq installed. If that works for you, you could use a more complex script to return a cut-down JSON that is under the 255-character limit.

Hellis81 · May 13, 2020, 10:29am

If you have somewhere to host a PHP file then I can write the code you need.
All you need then is a scrape sensor to get the value from the PHP file

Troon · May 13, 2020, 1:19pm

If you have somewhere to store a Python script (not sure how that varies by installation method), the script below will read in complex and long JSON, and give you just the bits you want. Save it as jsonfilter.py and call it in a command_line sensor thus:

/path/to/python /path/to/jsonfilter.py source filterspec1 [filterspec2…]

where source is the URL that returns the troublesome long JSON, and filterspec is a series of definitions like this:

"out=(JSON selector)"

You should specify the path to the same python executable as used by HomeAssistant.

For example, using this topic’s problem, you could use:

- platform: command_line
  name: Yilan sensors
  command: /path/to/python /path/to/jsonfilter.py "https://pm25.lass-net.org/data/last.php?device_id=74DA38F20824" "a=['feeds'][0]['AirBox']['s_d0']" "b=['feeds'][0]['AirBox']['s_d1']"

which would return:

and you can write template sensors from there on.

Here’s the Python script. It’s not pretty and will break without nice error messages if anything goes wrong.

import sys
import json
import requests

# usage: jsonfilter.py source_url "out1=['path']['to'][0]['in1']" ...

r = requests.get(sys.argv[1])
rj = r.json()
outd = {}

for x in range(2, len(sys.argv)):
    (outv, jsel) = sys.argv[x].split('=')
    outd[outv] = eval('rj%s' % jsel)

print(json.dumps(outd))

Savanna_Chow · May 14, 2020, 2:30am

Hi,

Thanks for your inspiration of using command line to get those json values. I finally use this method to get the correct value successfully:

curl -s https://pm25.lass-net.org/data/last.php?device_id=74DA38F20824 | python3 -c 'import json,sys;obj=json.load(sys.stdin);print(obj["'feeds'"][0]["'AirBox'"]["'s_d0'"])'

I found that HA doesn’t have jq and I don’t want to make things more complicated, so I tried another way to fetch the value I want by "python3 -c 'import json,sys;obj=json.load(sys.stdin);print(obj["'feeds'"][0]["'AirBox'"]["'s_d0'"])'"

However I still don’t understand why jason_attributes will hit the 255 characters limit and cannot output the value I want. I’ve checked documentations that jason_attributes can handle 255 characters, but it seems that this doesn’t work, at least I didn’t figure out the solution. Anyway, through command line I can still get the data I want.

Thank you very much for guys helping me

  - platform: command_line
    name: "PM2.5"
    scan_interval: 300
    command: curl -s https://pm25.lass-net.org/data/last.php?device_id=74DA38F20824 | python3 -c 'import json,sys;obj=json.load(sys.stdin);print(obj["'feeds'"][0]["'AirBox'"]["'s_d0'"])'
    value_template: '{{value}}'
    unit_of_measurement: 'μg/m3'