Help with JSON - Dexcom CGM Data

Hi there- I am fairly new to Hassio and I am trying to learn how to bring in outside information using a REST API from a JSON source. I’ve read over a number of the references here but I am still having trouble with exactly how the configuration should be setup. The big issue is that I’m new to working with JSON.

The first project I’m trying to tackle is pulling Continuous Glucose Monitor (CGM) data from a platform called Nightscout

I’ve been following this thread for reference: REST question

Here’s what the source data from Nightscout looks like:

[{"_id":"5dacc156dd5604c54a024f20","sgv":164,"date":1571602633000,"dateString":"2019-10-20T20:17:13.000Z","trend":4,"direction":"Flat","device":"share2","type":"sgv","utcOffset":0,"sysTime":"2019-10-20T20:17:13.000Z"}]

I have this working to pull out the sgv value from the JSON:

  - platform: rest
    name: Bryan_CGM
    resource: !secret nightscout_url
    value_template: '{{ value_json[0].sgv }}'

But, I am having trouble expanding this to pull out all of the info and to setup sgv as a sensor value so it will be graphed automatically.

  - platform: rest
    name: Bryan CGM Test
    json_attributes:
      - sgv
      - direction
    resource: !secret nightscout_url
    value_template: "OK"
  - platform: template
    sensors:
      sgv:
        friendly_name: 'Sugar'
        value_template: '{{ states.sensor.bryan_cgm_test.attributes["sgv"] }}'
        unit_of_measurement: "mg/dl"
      direction:
        friendly_name: 'Direction'
        value_template: '{{ states.sensor.bryan_cgm_test.attributes["direction"] }}'

Thanks for pointing me in the right direction.

Assuming your RESTful sensor is doing its job and storing sgv and direction as attributes, these ought to work:

        value_template: "{{ state_attr('sensor.bryan_cgm_test', 'sgv') }}"
        value_template: "{{ state_attr('sensor.bryan_cgm_test', 'direction') }}"

This is how I do it in my sensors.yaml:

  - platform: rest
    name: CGM sgv
    resource: https://my-g6.herokuapp.com/api/v1/entries/current.json
    value_template: '{{ value_json[0].sgv }}'

  - platform: rest
    name: CGM trend
    resource: https://my-g6.herokuapp.com/api/v1/entries/current.json
    value_template: '{{ value_json[0].trend }}'

  - platform: rest
    name: CGM direction
    resource: https://my-g6.herokuapp.com/api/v1/entries/current.json
    value_template: '{{ value_json[0].direction }}'
    
  - platform: rest
    name: CGM date
    resource: https://my-g6.herokuapp.com/api/v1/entries/current.json
    value_template: '{{ value_json[0].date }}'
    
  - platform: rest
    name: CGM dateString
    resource: https://my-g6.herokuapp.com/api/v1/entries/current.json
    value_template: '{{ value_json[0].dateString }}'
1 Like

Thanks @stevemann!

I was thinking about implementing it this way as well. But, isn’t possible to build this into one device with all of the attributes contained within? Not that it matters, but this would also be more efficient on the API.

I get the impression json_attributes isn’t extracting sgv and direction?

Correct, I am having issues getting it to work as json_attributes

However, I am a bit novice at this, so it could totally be user error. What @stevemann posted works, but isn’t there a better way to handle this type of data.

Yes, your idea is sound: create one sensor to retrieve all the data and, ideally, store it as attributes to be used by other sensors (one per attribute). This minimizes the number of times Home Assistant polls the remote data-source.

The challenge is parsing the received data’s structure (a list containing a dictionary). The RESTful sensor’s json_attributes option is rather limited. It expects the received data to be a dictionary and nothing but a dictionary.

Another way to do this is to retrieve all the data into the RESTful sensor’s state. As long as the data doesn’t exceed 255 characters in length, this is a viable option. Then you use template sensors to extract the portions you want from RESTful sensor’s state. However, you have to use regex_findall_index to perform the parsing because the data will no longer be handled as a JSON object once it’s stored in the sensor’s state.

I’ve helped others using this same approach so, if it interests you, let me know.


Give these a try:

  - platform: rest
    name: Bryan_CGM
    resource: !secret nightscout_url
    value_template: '{{ value_json[0] }}'

  - platform: template
    sensors:
      sgv:
        value_template: >
          {{ states('sensor.bryan_cgm') | regex_findall_index("sgv\":(\d+)") }}
      trend:
        value_template: >
          {{ states('sensor.bryan_cgm') | regex_findall_index("trend\":(\d+)") }}
      direction:
        value_template: >
          {{ states('sensor.bryan_cgm') | regex_findall_index("direction\":\"(\w+)\"") }}
1 Like

I only use the sgv and trend in my CGM monitor, (Trend is a numeric value for direction). I am a rank novice with JSON and I found it more straightforward to extract the data from the current.json REST API.

[{"_id":"5c64e08d935f1XXXXXXXXXXX","sgv":136,"date":1550114905000,"dateString":"2019-02-14T03:28:25.000Z","trend":4,"direction":"Flat","device":"share2","type":"sgv"}]

So, sgv and trend are easy to extract using
value_template: ‘{{ value_json[0].sgv }}’
value_template: ‘{{ value_json[0].trend }}’

I never could get the time value to make sense, and decided instead to send to my appliance the time from Node Red every ten seconds.

1 Like

The implementation of json_attributes missed the mark. It doesn’t work when the json object is a list of dictionaries. It only works when it’s a dictionary. Very poor implementation if you ask me. Rest and Commandline sensor needs json_attributes_template that MQTT has. Then people can properly extract the info they want in ANY json object.

2 Likes

Agree 100% and it echoes what I said in a previous post about ‘fit and finish’ issues, namely inconsistencies in behavior or operation (or outright deficiencies) that need to be addressed before 1.0. I see it as an initiative that ought to be driven by the project lead because of its comprehensive scope.

1 Like

Here’s a slightly improved version of my previous example. To mitigate the possibility that the received data may exceed 255 characters, the RESTful sensor’s template extracts and stores only what’s needed (and discards the balance).

In other words, if it receives this data:

[{"_id":"5dacc156dd5604c54a024f20","sgv":164,"date":1571602633000,"dateString":"2019-10-20T20:17:13.000Z","trend":4,"direction":"Flat","device":"share2","type":"sgv","utcOffset":0,"sysTime":"2019-10-20T20:17:13.000Z"}]

it will extract and store it in this, more compact, format:

sgv:164, trend:4, direction:Flat

The Template Sensors need a minor tweak in their regex pattern to accommodate the simpler format.

Here’s the code:

  - platform: rest
    name: Bryan_CGM
    resource: !secret nightscout_url
    value_template: >
      sgv:{{value_json[0].sgv}}, trend:{{value_json[0].trend}}, direction:{{value_json[0].direction}}

  - platform: template
    sensors:
      sgv:
        value_template: >
          {{ states('sensor.bryan_cgm') | regex_findall_index("sgv:(\d+)") }}
      trend:
        value_template: >
          {{ states('sensor.bryan_cgm') | regex_findall_index("trend:(\d+)") }}
      direction:
        value_template: >
          {{ states('sensor.bryan_cgm') | regex_findall_index("direction:(\w+)") }}
1 Like

Thanks, @123 & @stevemann, these are great examples.

I’ve implemented them both, and they are working great. But, it seems like using separate API calls (what @stevemann is doing) for each element might make more sense (to this novice user).

For simple actions like displaying the values in Lovelace, the separate sensors work better.

The first column above is using separate sensors, while the second column is using the combined method.

Purely for my education, other than reducing the number of API queries, is there any other benefit to using your method @123?

I get the feeling there may be a misunderstanding because the example I provided does, in fact, use a separate sensor to display each parameter.

sensor.bryan_cgm is exclusively for data-collection. The data it collects is consumed by the other three sensors. Don’t even bother displaying sensor.bryan_cgm in the UI.

These three Template Sensors get their data from sensor.bryan_cgm.

  1. sensor.sgv
  2. sensor.trend
  3. sensor.direction

They are suitable for display in the UI (i.e. graphing with an appropriate card).

The approach I’ve presented reduces the number of requests to the REST data-source (Nightscout). One RESTful sensor collects the data and the three Template Sensors consume it. This is more efficient than having three RESTful sensors where each one independently gets the same data. That’s three requests for the same data every 30 seconds instead of just one request.

Oh, got it, I did misunderstand how the template sensors work and I didn’t notice the three separate new sensors. Thank you, makes perfect sense now!

For anyone else following, here’s the code I ended up with:

########################################################################
# Dexcom CGM Interface
# https://community.home-assistant.io/t/rest-question/98222/19
# https://community.home-assistant.io/t/help-with-json-dexcom-cgm-data/143564
########################################################################
  - platform: rest
    name: bryan_cgm
    resource: !secret nightscout_url
    value_template: >
      sgv:{{value_json[0].sgv}}, trend:{{value_json[0].trend}}, direction:{{value_json[0].direction}}

  - platform: template
    sensors:
      bryan_cgm_sgv:
        value_template: >
          {{ states('sensor.bryan_cgm') | regex_findall_index("sgv:(\d+)") }}
        unit_of_measurement: "mg/dl"
      bryan_cgm_trend:
        value_template: >
          {{ states('sensor.bryan_cgm') | regex_findall_index("trend:(\d+)") }}
      bryan_cgm_direction:
        value_template: >
          {{ states('sensor.bryan_cgm') | regex_findall_index("direction:(\w+)") }}

Just one slight change from the code @123 provided. I added unit_of_measurement: "mg/dl" to the sgv sensor so lovelace can display a graph in the card.

1 Like

Great work! I may swap to this method to cut down on api polls.

I use the below method to display how many minutes ago the sensor was read.

How would it be added to your template?

  - platform: rest
    name: bg_dateString
    resource: !secret eli_nightscout_bg
    value_template: "{{ ( (as_timestamp(now()) - as_timestamp(value_json[0].dateString) ) / 60) | round(0) }}"

I’m not exactly sure, @123, do you have any input on this one?

For those following this, I wanted to share a new thread I started around using the CGM data.

Check it out and post your GGM data use examples:

Append it to the first sensor’s value_template:

    value_template: >
      sgv:{{value_json[0].sgv}}, trend:{{value_json[0].trend}}, direction:{{value_json[0].direction}}, time:{{ ( (as_timestamp(now()) - as_timestamp(value_json[0].dateString) ) / 60) | round(0) }}

If I test that in the Template Editor, using your original data, it reports the appended time parameter (as an integer value representing the difference in timestamps between now and the reading … if I’ve understood the supplied template):

Lastly, a fourth Template Sensor is needed to extract and report the time value.

  - platform: template
    sensors:
      cgm_time:
        value_template: >
          {{ states('sensor.cgm') | regex_findall_index("time:(\d+)") }}
1 Like

Thanks, @123, that worked great! I added it to my setup.

My updated code:

########################################################################
# Dexcom CGM Interface
# https://community.home-assistant.io/t/rest-question/98222/19
# https://community.home-assistant.io/t/help-with-json-dexcom-cgm-data/143564
# https://community.home-assistant.io/t/using-continuous-glucose-monitor-cgm-data/144624/2
########################################################################
  - platform: rest
    name: bryan_cgm
    resource: !secret nightscout_url
    value_template: >
      sgv:{{value_json[0].sgv}}, trend:{{value_json[0].trend}}, direction:{{value_json[0].direction}}, time:{{ ( (as_timestamp(now()) - as_timestamp(value_json[0].dateString) ) / 60) | round(0) }}

  - platform: template
    sensors:
      bryan_cgm_sgv:
        value_template: >
          {{ states('sensor.bryan_cgm') | regex_findall_index("sgv:(\d+)") }}
        unit_of_measurement: "mg/dl"
      bryan_cgm_trend:
        value_template: >
          {{ states('sensor.bryan_cgm') | regex_findall_index("trend:(\d+)") }}
      bryan_cgm_direction:
        value_template: >
          {{ states('sensor.bryan_cgm') | regex_findall_index("direction:(\w+)") }}
      bryan_cgm_time:
        value_template: >
          {{ states('sensor.bryan_cgm') | regex_findall_index("time:(\d+)") }}

Code for the mini-graph-card:

aggregate_func: avg
color_thresholds:
  - color: '#039BE5'
    value: 2
  - color: '#0da035'
    value: 4
  - color: '#e38f09'
    value: 6
entities:
  - entity: sensor.bryan_cgm_time
hours_to_show: 1
name: Min Since Last Poll
points_per_hour: 60
show:
  graph: line
  labels: true
type: 'custom:mini-graph-card'

Which results in:
51%20PM

1 Like