How can I import JSON data from a website to sensors?

[DaveTSG]

NASA’s ACE and NOAA’s DSCOVR spacecraft send measure the solar wind and data is made available every minute via these two JSON files: magnetic field parameters and solar wind parameters.

Each file contains many sensor read-outs. I want to import them all into Home Assistant.

From research, I understood the way to do this was using the Restful integration. My understanding is that I need to add this into configuration.yaml manually.

I added the following (using the example shown at the Restful integration page, and just changing what I knew needed changing):

# ACE & DSCOVR solar wind measurements
rest:
  - scan_interval: 60
    resource: https://noaa-swpc-pds.s3.amazonaws.com/json/rtsw/rtsw_wind_1m.json
    sensor:
      - name: "ACE/DSCOVR solar wind parameters"
        json_attributes_path: "$.response.system"
        value_template: "OK"
        json_attributes:
          - "time_tag"
          - "active"
          - "source"
          - "proton_speed"
          - "proton_temperature"
          - "proton_density"
          - "proton_vx_gse"
          - "proton_vy_gse"
          - "proton_vz_gse"
          - "proton_vx_gsm"
          - "proton_vy_gsm"
          - "proton_vz_gsm"
          - "proton_sample_size"
          - "alpha_speed"
          - "alpha_temperature"
          - "alpha_density"
          - "alpha_vx_gse"
          - "alpha_vy_gse"
          - "alpha_vz_gse"
          - "alpha_vx_gsm"
          - "alpha_vy_gsm"
          - "alpha_vz_gsm"
          - "alpha_sample_size"
          - "max_convergence_flag"
          - "max_data_flag"
          - "max_error_count_flag"
          - "max_processing_flag"
          - "max_range_flag"
          - "max_sample_count_flag"
          - "max_telemetry_flag"
          - "overall_quality"

# ACE & DSCOVR solar wind magnetic field measurements
rest:
  - scan_interval: 60
    resource: https://noaa-swpc-pds.s3.amazonaws.com/json/rtsw/rtsw_mag_1m.json
    sensor:
      - name: "ACE/DSCOVR solar wind magnetic field measurements"
        json_attributes_path: "$.response.system"
        value_template: "OK"
        json_attributes:
          - "time_tag"
          - "active"
          - "source"
          - "range"
          - "scale"
          - "sensitivity"
          - "manual_mode"
          - "sample_size"
          - "bt"
          - "bx_gse"
          - "by_gse"
          - "bz_gse"
          - "theta_gse"
          - "phi_gse"
          - "bx_gsm"
          - "by_gsm"
          - "bz_gsm"
          - "theta_gsm"
          - "phi_gsm"
          - "max_telemetry_flag"
          - "max_data_flag"
          - "overall_quality"

Unfortunately, I have little understanding of JSON and so realistically I have no idea what I’m doing.

With the above, all I get is an entity called “ACE/DSCOVR solar wind parameters” with a status “OK” and that’s it. I don’t get all the readings from each sensor into separate entities in Home Assistant.

Could someone please explain what I’m doing wrong?

Thanks.

[vingerha]

Not going to explain json, web has plenty of doc to read about that.
Your json has about 1800 blocks of text each with attributes you mention above, i.e. a block per minute so it seems.
The rest-integration can not handle this as attributes and importing the lot as a single attribute is way too much. What are you trying to achieve with this data? Do you need the attribs for all of the 1800 entries?

[DaveTSG]

@vingerha

What are you trying to achieve with this data? Do you need the attribs for all of the 1800 entries?

I want to view the data in HA.
I only need to import the most recent set of readings from each file, i.e. the first “time tag”. The file is updated every minute, so I was hoping to have HA get the latest data every minute.

[vingerha]

That makes it simpler. you only need to refer to the first one, i.e. 0
Try this ( I hope I without typos from my side

rest:
  - scan_interval: 60
    resource: https://noaa-swpc-pds.s3.amazonaws.com/json/rtsw/rtsw_wind_1m.json
    sensor:
      - name: "ACE/DSCOVR solar wind parameters"
        value_template: "{{ value_json.0.source }}"
        json_attributes_path: "$.0"
        json_attributes:
          - "time_tag"
          - "active"
          - "source"

The scan interval might not work, I trigger this via automation (reload)

EDIT, forgot the quotes…changed above

[DaveTSG]

Ah fantastic, thank you for your help with this so far.

I used your example and that worked - “time_tag”, “active”, and “source” came through as attributes of the one entity called “ACE/DSCOVR solar wind parameters”.

However, I am faced with two issues now:

1. I was hoping to have each read-out (for example “time_tag”) as its own entity, rather than having them all as attributes of the one entity (functionality is missing otherwise; for instance, I can’t select “time_tag” as an entity in the History page).
2. I need to modify your example to use the sensor readings I actually need, and I also need to take data from the second file as well.

Regarding #2:

I put the following into my configuration.yaml file:

#Import ACE and DSCOVR spacecraft data.
rest:
  - scan_interval: 60
    resource: https://noaa-swpc-pds.s3.amazonaws.com/json/rtsw/rtsw_wind_1m.json
    sensor:
      - name: "ACE/DSCOVR solar wind parameters"
        value_template: "{{ value_json.0.source }}"
        json_attributes_path: "$.0"
        json_attributes:
          - "proton_speed"
          - "proton_density"
  - scan_interval: 60
    resource: https://noaa-swpc-pds.s3.amazonaws.com/json/rtsw/rtsw_mag_1m.json
    sensor:
      - name: "ACE/DSCOVR solar wind magnetic field parameters"
        value_template: "{{ value_json.0.source }}"
        json_attributes_path: "$.0"
        json_attributes:
          - "bt"
          - "bx_gsm"
          - "by_gsm"
          - "bz_gsm"
          - "phi_gsm"
          - "theta_gsm"

This solves issue #2; the data for all of these sensors are coming through as attributes for the two sensors.

Regarding issue #1:

I did a little experimenting:

rest:
  - scan_interval: 60
    resource: https://services.swpc.noaa.gov/json/rtsw/rtsw_wind_1m.json
    sensor:
      - name: "Solar wind speed"
        value_template: "{{ value_json.0.source }}"
        json_attributes_path: "$.0"
        json_attributes:
          - "proton_speed"

Again, this worked to bring in the proton_speed into a separate entity called “Solar wind speed”, but of course, it came through as an attribute of that entity, rather being assigned to the entity itself.

Is there a way of assigning this to the entity itself, rather than as an attribute?

To give you a better example of what I mean (in case I am not using the correct terminology), I have attached two screenshots: one of a temperature sensor I have elsewhere in my HA setup, and one of the “Solar wind magnetic field parameters” entity with attributes shown.

Thank you again for your help.
Dave

[vingerha]

Try the other one, uses ONE resource to creatre multple sensors

RESTful - Home Assistant (home-assistant.io)

[vingerha]

rest:
  - scan_interval: 60
    resource: https://services.swpc.noaa.gov/json/rtsw/rtsw_wind_1m.json
    sensor:
      - name: "Solar wind speed"
        value_template: "{{ value_json.0.proton_speed}}"
...

[DaveTSG]

@vingerha

Try the other one, uses ONE resource to creatre multple sensors

RESTful - Home Assistant (home-assistant.io)

Ah gotcha. Thank you.

#Import ACE and DSCOVR spacecraft data.
rest:
  - scan_interval: 60
    resource: https://services.swpc.noaa.gov/json/rtsw/rtsw_wind_1m.json
    sensor:
      - name: "Solar wind speed"
        value_template: "{{ value_json.0.proton_speed }}"
      - name: "Solar wind density"
        value_template: "{{ value_json.0.proton_density }}"
  - scan_interval: 60
    resource: https://services.swpc.noaa.gov/json/rtsw/rtsw_mag_1m.json
    sensor:
      - name: "Solar wind Bt"
        value_template: "{{ value_json.0.bt }}"
      - name: "Solar wind Bx"
        value_template: "{{ value_json.0.bx_gsm }}"
      - name: "Solar wind By"
        value_template: "{{ value_json.0.by_gsm }}"
      - name: "Solar wind Bz"
        value_template: "{{ value_json.0.bz_gsm }}"
      - name: "Solar wind Phi angle"
        value_template: "{{ value_json.0.phi_gsm }}"
      - name: "Solar wind Theta angle"
        value_template: "{{ value_json.0.theta_gsm }}"

This creates eight entities, each with the sensor readings rather than as attributes. Perfect.

Thank you so much for your help!

Hopefully my last question, if you don’t mind: is there a way of getting the unit of measurements for the sensors?

I found this comment that suggested I need to specify “native_unit_of_measurement”. I also found this documentation that specifies what types there are, and that I need a device_class, and that - in order to save the data long-term for viewing in the history page - I need to specify state_class: measurement.

I also found this discussion that said I needed “unit_of_measurement” instead.

I entered the following:

rest:
  - scan_interval: 60
    resource: https://services.swpc.noaa.gov/json/rtsw/rtsw_wind_1m.json
    sensor:
      - name: "Solar wind speed"
        value_template: "{{ value_json.0.proton_speed }}"
        unit_of_measurement: km/s
        device_class: SensorDeviceClass.SPEED
        native_unit_of_measurement: km/s
        state_class: measurement

The data still loads into that sensor, but the km/s unit of measurement isn’t displayed anywhere in the entity.

Is this possible please?

Thank you again.

[vingerha]

This works (for me)

rest:
  - scan_interval: 60
    resource: https://noaa-swpc-pds.s3.amazonaws.com/json/rtsw/rtsw_wind_1m.json
    sensor:
      - name: "ACE/DSCOVR solar wind parameters"
        value_template: "{{ value_json.0.proton_speed }}"
        unit_of_measurement: km/s
        json_attributes_path: "$.0"
        json_attributes:
          - "time_tag"
          - "active"
          - "source"

[DaveTSG]

Got it, thank you.

Here’s the final configuration.yaml:

# Import ACE and DSCOVR spacecraft data.
rest:
  - scan_interval: 20
    resource: https://services.swpc.noaa.gov/json/rtsw/rtsw_wind_1m.json
    sensor:
      - name: "Solar wind speed"
        value_template: "{{ value_json.0.proton_speed }}"
        unit_of_measurement: km/s
      - name: "Solar wind density"
        value_template: "{{ value_json.0.proton_density }}"
        unit_of_measurement: p/cm^3
  - scan_interval: 20
    resource: https://services.swpc.noaa.gov/json/rtsw/rtsw_mag_1m.json
    sensor:
      - name: "Solar wind Bt"
        value_template: "{{ value_json.0.bt }}"
        unit_of_measurement: nT
      - name: "Solar wind Bx"
        value_template: "{{ value_json.0.bx_gsm }}"
        unit_of_measurement: nT
      - name: "Solar wind By"
        value_template: "{{ value_json.0.by_gsm }}"
        unit_of_measurement: nT
      - name: "Solar wind Bz"
        value_template: "{{ value_json.0.bz_gsm }}"
        unit_of_measurement: nT
      - name: "Solar wind Phi angle"
        value_template: "{{ value_json.0.phi_gsm }}"
      - name: "Solar wind Theta angle"
        value_template: "{{ value_json.0.theta_gsm }}"

Thank you very much for your help!

[vingerha]

NP, I however notice that the data runs infrequent behind ‘now’ meaning that the registration of the speed (example) and its sensor chage/update time are ‘off’ plus … if you updated them each minute … they will show sensor time changing but not the value which introduces another anomaly vs. the json data.
If you want it to be more exact, you could transform the json to a smaller set of data, say anything from now - 1hr and load that into one attribute. That you can then use for graphs…anyhow, if (!) you need time abd value to match that is

[DaveTSG]

Thank you, but at this point that’s unnecessary. The two spacecraft often have data blackouts (issues with communications) and also various other problems (e.g. when the spacecraft is rolling to desaturate control moment gyroscopes) so it’s not really worthwhile.

Thank you, though, for the suggestion and for taking the time to look into it further.

Side-note: the data are available in a graph on the NOAA SWPC website.