Creating sensors from json list

gregology · July 2, 2021, 4:10pm

I am running a local SkyAware instance and I would like to track aircraft in Home Assistant in order to correlate other sensor readings. Other use cases are identifying aircraft that we can see.

SkyAware provides a json blob of aircraft
http://<IP ADDRESS>:8080/data/aircraft.json

{ "now" : 1625240909.2,
  "messages" : 21637,
  "aircraft" : [
    {"hex":"3c71a1","flight":"BCS954  ","alt_baro":38000,"alt_geom":38800,"gs":495.2,"track":218.6,"baro_rate":0,"squawk":"6334","emergency":"none","category":"A5","nav_qnh":1012.8,"nav_altitude_mcp":38016,"lat":44.975006,"lon":-78.402100,"nic":8,"rc":186,"seen_pos":31.1,"version":2,"nic_baro":1,"nac_p":9,"nac_v":1,"sil":3,"sil_type":"perhour","gva":2,"sda":2,"mlat":[],"tisb":[],"messages":287,"seen":0.7,"rssi":-22.1},
    {"hex":"c01066","flight":"NDL654  ","alt_baro":25000,"alt_geom":25200,"gs":250.2,"track":57.1,"geom_rate":64,"squawk":"0552","emergency":"none","category":"A1","lat":44.258330,"lon":-77.722711,"nic":9,"rc":75,"seen_pos":0.9,"version":2,"nic_baro":0,"nac_p":10,"nac_v":2,"sil":3,"sil_type":"perhour","gva":2,"sda":2,"mlat":[],"tisb":[],"messages":56,"seen":0.9,"rssi":-20.3},
    {"hex":"ab7265","category":"A1","version":2,"sil_type":"perhour","mlat":[],"tisb":[],"messages":75,"seen":94.6,"rssi":-22.2},
    {"hex":"c02fe7","flight":"ACA607  ","alt_baro":30000,"alt_geom":30350,"gs":465.0,"track":237.9,"baro_rate":0,"category":"A3","nav_qnh":1013.6,"nav_altitude_mcp":24000,"nav_heading":251.7,"lat":44.676695,"lon":-77.419181,"nic":8,"rc":186,"seen_pos":8.7,"version":2,"nic_baro":1,"nac_p":10,"nac_v":2,"sil":3,"sil_type":"perhour","gva":2,"sda":2,"mlat":[],"tisb":[],"messages":308,"seen":8.6,"rssi":-16.6},
    {"hex":"c05a0e","flight":"NDL876  ","alt_baro":27000,"alt_geom":27275,"gs":254.8,"track":58.8,"geom_rate":-64,"squawk":"2217","emergency":"none","category":"A1","lat":44.522644,"lon":-77.270312,"nic":9,"rc":75,"seen_pos":0.5,"version":2,"nic_baro":1,"nac_p":10,"nac_v":2,"sil":3,"sil_type":"perhour","gva":2,"sda":2,"mlat":[],"tisb":[],"messages":1773,"seen":0.5,"rssi":-14.6},
    {"hex":"c035bb","alt_baro":31000,"alt_geom":31350,"gs":498.3,"track":69.9,"baro_rate":-64,"category":"A3","version":2,"nac_v":2,"sil_type":"perhour","mlat":[],"tisb":[],"messages":2199,"seen":52.6,"rssi":-22.5},
    {"hex":"c06c95","flight":"NSTAR2  ","alt_baro":1550,"alt_geom":1400,"gs":187.0,"track":232.8,"baro_rate":-256,"squawk":"0207","emergency":"none","category":"A1","lat":44.075938,"lon":-77.499782,"nic":9,"rc":75,"seen_pos":3.7,"version":2,"nic_baro":1,"nac_p":10,"nac_v":2,"sil":3,"sil_type":"perhour","gva":2,"sda":3,"mlat":[],"tisb":[],"messages":772,"seen":0.0,"rssi":-21.8},
    {"hex":"43c173","flight":"RRR6657 ","alt_baro":35000,"alt_geom":35650,"gs":464.2,"track":53.7,"geom_rate":0,"squawk":"5745","emergency":"none","category":"A5","nav_qnh":1013.6,"nav_altitude_mcp":35008,"nav_heading":51.3,"nav_modes":["althold","tcas"],"lat":45.447327,"lon":-76.389640,"nic":10,"rc":25,"seen_pos":15.7,"version":2,"nic_baro":1,"nac_p":10,"nac_v":2,"sil":3,"sil_type":"perhour","gva":2,"sda":2,"mlat":[],"tisb":[],"messages":1112,"seen":2.7,"rssi":-21.2},
    {"hex":"a37e05","category":"A2","version":2,"sil_type":"perhour","mlat":[],"tisb":[],"messages":2861,"seen":151.6,"rssi":-21.3},
    {"hex":"c0882f","category":"A1","version":2,"sil_type":"perhour","mlat":[],"tisb":[],"messages":2082,"seen":278.5,"rssi":-21.7}
  ]
}

New aircraft appear and disappear frequently. I have tried creating a single rest sensor

 - platform: rest
   name: SkyAware
   resource: http://10.0.0.109:8080/data/aircraft.json
   method: GET
   scan_interval: 60

But the json blob is too long

Invalid state encountered for entity ID: sensor.skyaware. State max length is 255 characters.

What are some other options and what is the best practice for storing information like this? Is there a way to create a new sensor for each aircraft that appears? Is this an anti pattern? My googling is failing so any direction would be useful.
Cheers Team!

nickrout · July 2, 2021, 11:12pm

Try this one RESTful - Home Assistant

gregology · July 3, 2021, 12:48am

Cheers @nickrout, I am using the REST integration. My question is how best to create individual sensors from the response.

smck83 · July 3, 2021, 1:08am

Hi Gregology I’ve done a similar thing for my brothers pool - see here Rest Sensor receiving JSON - Astralpool - connectmypool.com.au - #5 by smck83 where i use the rest sensor to update and then use sensor templates to update to a specific value. The only thing here, is I don’t have an enormous amount of data in the json blob. Also it looks like that dataset might change on incremental updates only.

This may work if you only want <10 flights, but if you added 100s or 1000s it could mean 100s or 1000s of individual sensors - ideally the resource would support passing a flight number or pagination to reduce the size of the response but I’m not familiar with what is storing that data. You could for example host a custom aircraft.php file that loads the large json file and only responds with the queries data - nodered might help here too and then have HA query /aircraft.php?flight=BCS954 or using pagination e.g. /aircraft.php?limit=50&page=1 - i think the former makes more sense though depending on what your goal is.

  - platform: rest
    name: SkyAware
    resource: http://192.168.8.182:8123/local/test/aircraft.json
    method: GET
    scan_interval: 60 
    value_template: 'OK'
    json_attributes:
    - now
    - messages
    - aircraft

and then use a template sensor for a particular result

  - platform: template
    sensors:
      skyaware_first_result:
        friendly_name: Skyaware First Result
        value_template: '{{ states.sensor.skyaware.attributes["aircraft"][0] }}'
      skyaware_second_result:
        friendly_name: Skyaware Second Result
        value_template: '{{ states.sensor.skyaware.attributes["aircraft"][1] }}'

Using the small dataset you provided - the above produces two sensors with attributes that could be used.

If the aircraft.json data isnt all flights but only the incremental updates i also wonder whether it might be more efficient to push it into MQTT (maybe with nodered?).

PS> Looking forward to our next hacking session

nickrout · July 3, 2021, 4:48am

No you are using the rest sensor, not the restful integration.

smck83 · July 3, 2021, 1:16pm

I wasnt familiar with the RESTful sensor integration @nickrout suggested but looks like it could be alot cleaner.

I would be interested if there is anyway to dynamically reference the array object as my example is hardcoding the array object id e.g. [0],[1],[2] and so on.

it would go in configuration.yaml under the heading:

rest:

or to make it tidy create rest.yaml in the folder and then include it in your configuration.yaml file:

rest: !include rest.yaml

then drop this as an example in the rest.yaml file

  resource: http://10.0.0.109:8080/data/aircraft.json
  method: GET
  scan_interval: 60 
  sensor:
    - name: "Skyaware 1"
      json_attributes_path: "$.aircraft[0]"
      value_template: "OK"
      json_attributes:
        - flight
        - alt_baro
        - alt_geom
        - emergency
    - name: "Skyaware 2"
      json_attributes_path: "$.aircraft[1]"
      value_template: "OK"
      json_attributes:
        - flight
        - alt_baro
        - alt_geom
        - gs
        - track
        - baro_rate
        - emergency
    - name: "Skyaware 3"
      json_attributes_path: "$.aircraft[2]"
      value_template: "OK"
      json_attributes:
        - flight
        - alt_baro
        - alt_geom
        - emergency
    - name: "Skyaware 4"
      json_attributes_path: "$.aircraft[3]"
      value_template: "OK"
      json_attributes:
        - flight
        - alt_baro
        - alt_geom
        - gs
        - track
        - baro_rate
        - emergency
    - name: "Skyaware 5"
      json_attributes_path: "$.aircraft[4]"
      value_template: "OK"
      json_attributes:
        - flight
        - alt_baro
        - alt_geom
        - gs
        - track
        - baro_rate
        - emergency

what you end up with is this (NOTE: skyware3 - "$.aircraft[2] is missing the attributes in the respone which is why its blank:

koying · July 3, 2021, 1:32pm

You can use more elaborate jsonpath filters, e.g.

$.aircraft[?(@.flight=="AA101")]

smck83 · July 3, 2021, 1:43pm

oh neat - would that mean you could reference a particular flight like this?

    - name: "Skyaware RRR6657"
      json_attributes_path: '$.aircraft[?(@.flight=="RRR6657")]'
      value_template: "OK"
      json_attributes:
        - flight
        - alt_baro
        - alt_geom
        - gs
        - track
        - baro_rate
        - emergency

koying · July 3, 2021, 1:45pm

Yes, something similar should work.

smck83 · July 3, 2021, 1:50pm

ok thanks - first attempt didnt work. Do you know if theres particular documentation on the syntax for this, e.g. specifically what @ or ? do and reference

EDIT: Got it working Thanks @koying
The Flight number in the data set had spaces after it

- name: "Skyaware RRR6657"
  json_attributes_path: '$.aircraft[?(@.flight == "RRR6657 ")]'
  value_template: "OK"
  json_attributes:
    - flight
    - alt_baro
    - alt_geom
    - gs
    - track
    - baro_rate
    - emergency

for json_attributes_path I stumbled across this which suggest ‘contains’ could possibly be used in place of == to avoid having to add the spaces but this didnt work in my quick test, reference:
https://support.smartbear.com/alertsite/docs/monitors/api/endpoint/jsonpath.html

koying · July 3, 2021, 3:15pm

JsonPath Expressions is probably safe to use.
I don’t know exactly what the library used by HA supports

smck83 · July 4, 2021, 11:52pm

Incase it helps anyone else - this was helpful for working out whether it would or wouldnt work

alexdelprete · August 30, 2021, 10:57pm

In the RESTful sensor docs, they reference Goessner’s site.