Parsing out large JSON from CLI sensor

[danodemano]

I’m working on controlling my Anova cooker using this wrapper: https://github.com/bmedicke/anova.py and a custom py script. When the cooker is not running I have a manageable chunk of JSON:

{
    "status": {
        "cooker_id": "anova xxxxxxxxxxxxxxxx",
        "current_temp": 72.4,
        "firmware_version": "ver 6.1.1",
        "is_running": false,
        "is_timer_running": false,
        "speaker_mode": true,
        "target_temp": 72,
        "temp_unit": "f",
        "timer_length": 240
    }
}

However when the cooker IS running I quickly overflow the max 255 characters:

{
    "status": {
        "cooker_id": "anova xxxxxxxxxxxxxxxx",
        "current_job": {
            "is_running": true,
            "job_id": "fee68923-6cf2-46fa-9202-8f452e25a082",
            "job_info": {
                "display_item_identifier": "",
                "duration": 300,
                "source": "user_defined",
                "source_identifier": "",
                "temperature": 72,
                "temperature_unit": "Fahrenheit"
            },
            "job_stage": "cooking",
            "job_start_time": "2019-02-19T00:22:47.846004Z",
            "job_type": "manual_cook",
            "job_update_time": "2019-02-19T00:22:48.846006Z",
            "max_circulation_interval": 300,
            "target_temp": 72,
            "temp_unit": "f",
            "threshold_temp": 40,
            "timer_length": 300
        },
        "current_job_id": "fee68923-6cf2-46fa-9202-8f452e25a082",
        "current_temp": 72.2,
        "firmware_version": "ver 6.1.1",
        "is_running": true,
        "is_timer_running": true,
        "speaker_mode": true,
        "target_temp": 72,
        "temp_unit": "f",
        "timer_length": 300
    }
}

My current config is this which I intended to parse out:

- platform: command_line
  command: python3 /scripts/anova_control.py -m status
  name: Anova RAW Status
  value_template: '{{value_json}}'
  scan_interval: 60

However that obviously doesn’t work given the length. So my question is, how can I parse out that data without making a bunch of different calls? Each run of that script is an API hit to their (Anova’s) site and given the undocumented nature of the API I would prefer not to be hitting it 20+ times a minute.

I saw some stuff around a rest sensor but that doesn’t help me here. Any help would be greatly appreciated!

[Jardiamj (Jardi Martínez Jordan)]

You could pass that whole bunch of information to the command_line sensor as json_attributes, and then create template sensors for any values in the attributes that you want in your HA UI.

Jardi.

[danodemano]

@Jardiamj - I guess I’m not sure I follow. It looks like that works for RESTful or MQTT sensors. I don’t see anything about using json_attributes for a CLI sensor. Or maybe I’m not understanding correctly what you are proposing?

EDIT: I think I’m just being daft. When I Google searched for it I got nothing but I can see that as an option in the CLi sensor docs. I will play around with that, thanks!

[danodemano]

Alright, I’ve tried but nothing is parsed out. Config:

- platform: command_line
  command: /scripts/anova_control.py -m status
  name: Anova Status
  json_attributes:
    - current_temp
    - is_running
    - is_timer_running
    - speaker_mode
    - target_temp
    - temp_unit
    - timer_length
    - current_job.job_stage
    - current_job.job_info.duration
    - current_job.job_start_time
  value_template: '{{value_json.status}}'
  scan_interval: 60

But I get nothing from the sensor except the JSON:

[danodemano]

Ok, I’m pulling my hair out here. I’ve tried all sorts of ways to parse out the JSON:

- platform: command_line
  command: /scripts/anova_control.py -m status
  name: Anova Status
  json_attributes:
    - '{{ value_json[0].status.current_temp }}'
    - value_json[0].status.current_temp
    - '{{ value_json.status.current_temp }}'
    - value_json.status.current_temp
    - status.current_temp
    - is_running
    - is_timer_running
    - speaker_mode
    - target_temp
    - temp_unit
    - timer_length
    #- current_job.job_stage
    #- current_job.job_info.duration
    #- current_job.job_start_time
  value_template: '{{value_json}}'
  scan_interval: 60

None of that works. HOWEVER, if I copy the JSON to the template window in the UI I have no issues accessing the individual elements:

{% set my_test_json = [
{
    "status": {
        "cooker_id": "anova xxxxxxxxxxxxx",
        "current_temp": 72.4,
        "firmware_version": "ver 6.1.1",
        "is_running": false,
        "is_timer_running": false,
        "speaker_mode": true,
        "target_temp": 72,
        "temp_unit": "f",
        "timer_length": 240
    }
}

]
 %}

The temperature is {{ my_test_json[0].status.current_temp }} 

^ That works as expected and gives me the current temp. What am I missing here?

[gpbenton (Graham)]

There have been some changes recently, so I am not sure if this is still valid, but the json_attributes used to have a limitiation that it could only extract the top level components. In your case that is just “status” which doesn’t help you much.

[danodemano]

Oh, well that would explain my issue if that’s still the case. And I’ve been fighting a ghost it would seem.

[Jardiamj (Jardi Martínez Jordan)]

You need to pass a list of key:value attributes to the json_attributes and I believe it has to be in the top level of the json string.

For example, you could do:

- platform: command_line
  command: /scripts/anova_control.py -m status
  name: Anova Status
  json_attributes:
    - status
  value_template: '{{value_json.status.current_temp}}'
  scan_interval: 60

It seems like a dumb example, but it’s the only thing I see matching you json string. That will create a list of attributes with the elements inside the status node.

If you have any control on how your script generates the json data I suggest you structure it in a more HA friendly way.

I am putting current_temp in the value_template just as an example for the sensor, that way you don’t get the whole string in there. You can out any value you want to see as the state of the sensor.

[lmamakos (Louis Mamakos)]

It’s sort of funny how this issue comes up so frequently. Just a few days ago, this was addressed in this other thread, that might be useful: Need help storing URL for use in generic IP camera (over 255 chars)

[Jardiamj (Jardi Martínez Jordan)]

@lmamakos, also I think since status is the only top level node in his json string, his script should print the block under status. It would look like this:

 {
        "cooker_id": "anova xxxxxxxxxxxxxxxx",
        "current_job": {
            "is_running": true,
            "job_id": "fee68923-6cf2-46fa-9202-8f452e25a082",
            "job_info": {
                "display_item_identifier": "",
                "duration": 300,
                "source": "user_defined",
                "source_identifier": "",
                "temperature": 72,
                "temperature_unit": "Fahrenheit"
            },
            "job_stage": "cooking",
            "job_start_time": "2019-02-19T00:22:47.846004Z",
            "job_type": "manual_cook",
            "job_update_time": "2019-02-19T00:22:48.846006Z",
            "max_circulation_interval": 300,
            "target_temp": 72,
            "temp_unit": "f",
            "threshold_temp": 40,
            "timer_length": 300
        },
        "current_job_id": "fee68923-6cf2-46fa-9202-8f452e25a082",
        "current_temp": 72.2,
        "firmware_version": "ver 6.1.1",
        "is_running": true,
        "is_timer_running": true,
        "speaker_mode": true,
        "target_temp": 72,
        "temp_unit": "f",
        "timer_length": 300
    }

That way he could do something like this:

- platform: command_line
  command: /scripts/anova_control.py -m status
  name: Anova Status
  json_attributes:
    - cooker_id
    - current_jod
    - current_job_id
    - target_temp
    - firmware_version
    - is_running
    - is_timer_running
    - speaker_mode
    - target_temp
    - temp_unit
    - timer_length
  value_template: '{{value_json.current_temp}}'
  scan_interval: 60

I think that should do the trick. But I haven’t tested it.

[danodemano]

@Jardiamj - I will give that a shot tonight!

I was actually considering re-structuring the JSON. That’s how it comes from the API but the Python script is mine so I should be able to manipulate it into something more manageable. Just to make sure I understand, everything I want needs to be a top-level node correct? You’ll notice some of what I’m trying to grab is also in those current_job and job_info blocks. I’m thinking just restructuring the JSON to have those data points I need in a top-level node makes sense if HA can deal with that.

[Jardiamj (Jardi Martínez Jordan)]

You can have a whole json string in a parameter and then access it when you create the template sensors. In fact, I am doing that exact same thing to integrate my Personal Weather Station with HA. In my main sensor I have an attribute called current that gets a json string with all the current data from the station, then I create a template sensor for let’s say Outside Temperature in the following way:

- platform: template
  sensors:
    outside_temperature:
      friendly_name: Outside Temperature
      entity_id:
        - sensor.weather_station
      value_template: "{{ states.sensor.weather_station.attributes.current['outTemp'] }}"
      unit_of_measurement: '°F'

[danodemano]

Can you elaborate a bit more on your config and the JSON you are parsing please? I’m still a little hazy as to how I need to be formatting my JSON out of the Python script and what to do with it in HA.

[Jardiamj (Jardi Martínez Jordan)]

I posted what I did with my Weather Station here:

Weather Station Data

I finally got the time to work on getting my weather station data from weeWX to Home Assistant. I didn’t want to mess up with MQTT so I created a json template for the skin that I am using, seasons (which is the default skin in the last release). Since my weewx is running in a Debian machine, installed using apt, I created the file: /etc/weewx/skins/Seasons/daily.json.tmpl with the following template: #encoding UTF-8 {"title":"Current Values","location":"$station.location","time":"$current.date…

This is what the generated json string looks like:

{"title":"Current Values",
  "location":"City, WA",
  "time":"02/19/2019 10:00:00 AM",
  "lat":"XX° XX' N",
  "lon":"XX° XX' W",
  "alt":"407 feet",
  "hardware":"Weather Monitor II",
  "uptime":"2 days, 13 hours, 58 minutes",
  "serverUptime":"2 days, 14 hours, 0 minutes",
  "weewxVersion":"3.9.1",
  "current": {"outTemp":"38.1","windchill":"38.1","heatIndex":"38.1","dewpoint":"36.2","humidity":"93","insideHumidity":"29","barometer":"30.274","barometerTrendDelta":"3 hours","barometerTrendData":"-0.044","windSpeed":"0","windDir":"   N/A","windDirText":"N/A","windGust":"0","windGustDir":"   N/A","rainRate":"0.00","rainToday":"0.01","insideTemp":"75.8"}
}

sensor.weather_station will have an attribute called “current” with this as the value:

{
    "outTemp":"38.1",
    "windchill":"38.1",
    "heatIndex":"38.1",
    "dewpoint":"36.2",
    "humidity":"93",
    "insideHumidity":"29",
    "barometer":"30.274",
    "barometerTrendDelta":"3 hours",
    "barometerTrendData":"-0.044",
    "windSpeed":"0",
    "windDir":"   N/A",
    "windDirText":"N/A",
    "windGust":"0",
    "windGustDir":"   N/A",
    "rainRate":"0.00",
    "rainToday":"0.01",
    "insideTemp":"75.8"
}

And you can access any value in that json string this way:

value_template: "{{ states.sensor.weather_station.attributes.current['outTemp'] }}"

I hope that makes sense.

[danodemano]

I think that’s exactly what I needed, thanks so much! Will have to do some testing tonight and see if I can get this working. Really appreciate it!!!

[123 (Taras)]

Perhaps not of immediate use to you but for future reference, Node-Red handles the lengthy Anova data. In this example, I simply fed it the JSON data and asked it to display it as a JSON object. The result is in the debug window on the right.

One could proceed to select a subset of the desired elements and supply it to Home Assistant. Anyway, food for thought should you ever add Node-red to your home automation system.

[danodemano]

@123 - Thanks, I’m using Node-RED currently and considered that path. However, my hope was to have Home Assistant view/control directly without hopping through NR to make that happen. However, once I actually get to the control piece using NR may be inevitable. Thanks for the info!

[123 (Taras)]

You’re welcome!

FWIW, I’ve found Node-Red to be very handy for “gluing” systems together and/or reformatting data into something more digestible (like MQTT).

A simple example:

• Using MacroDroid, my phone sends its battery and powerstate via UDP to Node-red.
• Node-red receives the UDP packets and publishes them as MQTT topics (phone/battery and phone/power).
• MQTT Sensors in Home Assistant are subscribed to the the two topics.

Similarly, our home phone uses a VOIP adapter that can report its activities to a syslog server (via UDP). I directed this stream of syslog messages to Node-red (using the stock syslog node), parse out the info I want, and publish it to several MQTT topics. Now anything in my home that understands MQTT can be apprised of the home phone’s status.

[danodemano]

@Jardiamj - Got i!

I simplified the JSON way down to just what I needed with nothing nested:

{"is_timer_running": false, "job_stage": null, "timer_length": 240, "duration": null, "target_temp": 72, "speaker_mode": true, "job_start_time": null, "alarm_active": null, "current_temp": 58.4, "temp_unit": "f", "is_running": false}

Then the config is stupid easy:

- platform: command_line
  command: /scripts/anova_control.py -m status -o h
  name: Anova Status
  json_attributes:
    - current_temp
    - is_running
    - is_timer_running
    - speaker_mode
    - target_temp
    - temp_unit
    - timer_length
    - job_stage
    - job_info.duration
    - job_start_time
  value_template: '{{value_json.current_temp}}'
  scan_interval: 60

Thanks again! On to the next bit, control!!