Creating Sensors and filtering specific JSON Value from Curb Energy API

luisgarcia87 · May 6, 2023, 3:24am

Hi everyone I need some help creating template sensor from this JSON String that Curb’s Energy Monitor API provides:

{
  "timestamp": 1683341100,
  "locationId": "secret_id",
  "consumption": 2210,
  "production": 0,
  "storage": 0,
  "net": 2210,
  "circuits": [
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "Fridge and Freezer",
      "id": "d7d5d888-aad0-41f6-9998-a7a7259999b4",
      "w": 194,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "Office Bathroom and Old Kitchen Plug",
      "id": "bf2ad1cb-24fe-4c4b-aac1-5a8a492b148f",
      "w": 18,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "AC Master Bedroom",
      "id": "bbdb13a0-67c8-4268-b1d7-ee586cc0331d",
      "w": 4,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "AC Living Room",
      "id": "09817feb-fd6a-4865-882a-d7440eded30d",
      "w": 4,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "Washer, Dryer and Master TV",
      "id": "f1877bcb-e9fd-483b-b834-1c799af59467",
      "w": 83,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "Kitchen Stove, Mini Bar, Wine Cooler",
      "id": "57e8efea-182c-4cc2-9524-27bf2601d95b",
      "w": 43,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "Water Pump and Deck",
      "id": "2a559310-48b1-4a22-b882-aa143c89a070",
      "w": 18,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "AC Baby Room",
      "id": "17ee63fb-0773-4cbd-985f-c29b305989fa",
      "w": 4,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "Baby Room and Guest Bathroom",
      "id": "9c62c562-9be7-4bbe-8f1b-91dce4d508b2",
      "w": 11,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "Dishwasher and Disposal",
      "id": "da1dba20-89ae-4770-80d3-89b5f491c22f",
      "w": 893,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "Office and PC Equipment",
      "id": "03b725c6-c282-4724-bf1e-aa9ec232d7ae",
      "w": 130,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "Living, Dining, Garage Lights",
      "id": "95334270-705b-449e-8b88-37b3fa9eb29e",
      "w": 353,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "AC Office",
      "id": "1ab4d7f3-f844-41d5-8684-70360f59c273",
      "w": 4,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "Master Bathroom and Wall behind Bed",
      "id": "272f57f4-40a0-4a9c-a7e3-6a66f73758b0",
      "w": 86,
      "main": false
    },
    {
      "grid": false,
      "production": false,
      "battery": false,
      "circuit_type": "consumption",
      "label": "Garage Door",
      "id": "2037cb9d-1203-421b-a874-3e618f04d5d0",
      "w": 7,
      "main": false
    }
  ]
}

I want to have a separate sensor from each circuit. But due to my lack of knowledge in selecting JSON specific values I am stuck.

Any help is appreciated

Thank you

tom_l · May 6, 2023, 3:34am

This will help: https://jsonpathfinder.com/

bmclapham1973 · May 9, 2023, 3:41pm

Can you share how you are retrieving this information from the Club Energy device? I’d be happy to collaborate on parsing the JSON.

luisgarcia87 · May 9, 2023, 5:50pm

Sure can.

First things first. After reaching out for more than 8 months to Curb’s customer support, They point me to a github page that helped to set up the integration. (I don’t know if this was available before. As I was searching for a solution for a long time)

Curb v2 Third Party App Integration

You need to send a curl POST command with this information:

curl -X POST \
  https://energycurb.auth0.com/oauth/token \
  -H 'Cache-Control: no-cache' \
  -H 'Content-Type: application/json' \
  -d '{
	"grant_type": "password",
	"audience": "app.energycurb.com/api",
	"username": "{{USER_EMAIL}}",
	"password": "{{USER_PASSWORD}}",
	"client_id": "iKAoRkr3qyFSnJSr3bodZRZZ6Hm3GqC3",
	"client_secret": "dSoqbfwujF72a1DhwmjnqP4VAiBTqFt3WLLUtnpGDmCf6_CMQms3WEy8DxCQR3KY"
}'

Then add to the previous code your username and password that you use to login to Curb’s Montioring Website

Note that the client_id and client_secret are taken from Curb’s GitHub:

We provide an example pair of client credentials that you can use to get up and running with our API quickly and easily.

Client ID : iKAoRkr3qyFSnJSr3bodZRZZ6Hm3GqC3
Client Secret : dSoqbfwujF72a1DhwmjnqP4VAiBTqFt3WLLUtnpGDmCf6_CMQms3WEy8DxCQR3KY"

After you filled that up send the command and you should receive the following response:

With the received access_token you can call Curb’s API, as per the documentation to get the parameters required to call the circuit data:

GET /api/v3/locations - Get all of the users’s locations.

curl -X GET https://app.energycurb.com/api/v3/locations -H 'authorization: Bearer {{access_token}}'

Replace {{access_token}} with the access token received on the previous command, and don’t forget the ' at the end of the curl command.

You should receive a response with other details, but what you are interested in, is the “location_id” or just “id”) section:

    "address": "1524 S IH 35",
    "country": "USA",`Preformatted text`
    "geocode": "30.24306,-97.73605500000001",
    "name": "CURB Office",
    "id": "32d23ga8-re33-4294-a382-7fga34m4ff28",
    "postcode": "78704",
    "hasProduction": true

Copy this “location_id” or just “id” into the next section of the code to get the circuit data:

GET /latest/:locationId - Get the latest averaged minute sample snapshot of all circuits, plus the consumption, production, and net values.

curl -X GET https://app.energycurb.com/api/v3/latest/{{location_id}} -H 'authorization: Bearer {{acces_token}}'

Replace {{location_id}} with the id received on the previous command and also replace {{access_token}} with the access token, and don’t forget the ' at the end of the curl command.

Received information:

{
	"timestamp": 1683652080,
	"locationId": "32d23ga8-re33-4294-a382-7fga34m4ff28",
	"consumption": 1624,
	"production": 0,
	"storage": 0,
	"net": 1624,
	"circuits": [{
		"grid": false,
		"production": false,
		"battery": false,
		"circuit_type": "consumption",
		"label": "Fridge and Freezer",
		"id": "a7e5f658-bbd0-42f6-9778-a7a2259899b4",
		"w": 144,
		"main": false

With this information I went to https://jsonpathfinder.com/ as suggested by @tom_l and I was able to find the path to my circuits on the JSON string and create a rest api and a template sensor for each of the circuits on my data.

This following information is stored under my sensor.yaml (As I have separted it from the configuration.yaml to avoid many lines on that configuration.yaml file)

  - platform: rest
    name: curb_fridge
    resource: https://app.energycurb.com/api/v3/latest/32d29fe8-be29-4184-a159-7fde34a4ff34
    headers:
      Authorization: !secret curb_secret_token
      Content-Type: application/json
    json_attributes_path: "$.circuits.[0]"
    json_attributes:
      - w
    scan_interval: 50
    value_template: "{{value_json.curb_fridge_watts}}"
    method: GET
  - platform: template
    sensors:
      curb_fridge_consumption:
        value_template: "{{ state_attr('sensor.curb_fridge', 'w')}}"

Note: To capture the first circuit that shows up on your data the “json_attributes_path” would be “$.circuits.[0]”, the second circuit would be “$.circuits.[1]” and so on…

The authorization on this rest api sensor includes a line on the secrets.yaml file as this:

curb_secret_token: Bearer {{access_token}} #(Replace with your own access token taken before)

Rest Api Sensor with the W (watts) attributes of the specific breaker

Template sensor of the Watts taken from the Rest Api Sensor.

I am sure there’s better way to create the sensors, I am open to ideas as I am not an expert at all on this.

On the down side of this integration:

acces_token get’s revoked every 24 hours (I am trying to reach out to Curb’s Customer Support to see if it can be changed to something more permanent)
Circuit Data information is not “live”, meaning when you input the cURL * GET ‘/latest/:locationId’ - You get the latest averaged minute sample snapshot of all circuits. (It updates every minute not every second)

I am hoping to get real live data on Home Assistant via the simple-browser-client from Curb’s but so far I am not able to web-scrape the data from there due to my limited knowledge (it seems as if the server first need to call a javascript to sign in and then access the data. If there’s anyone with some insight for this please jump in.

Thanks

bmclapham1973 · May 10, 2023, 3:55pm

Wow! What a detailed and thoughtful reply. This exactly what I was hoping to do. I will let you know once I have mine up and running.

tom_l · May 10, 2023, 4:07pm

If you use the rest integration ( instead of the rest sensor platform) you can retrieve all your sensors at once from one call to the resource and no need for template sensors (use the value_template for each).

luisgarcia87 · May 10, 2023, 8:44pm

Thank you @tom_l for that suggestion, now with 1 api call I can update all my sensors.

Here’s the yaml for anyone that might need it.

rest:
  - resource: https://app.energycurb.com/api/latest/{{location_id}}
    scan_interval: 10
    method: GET
    headers:
      Authorization: !secret curb_secret_token
      Content-Type: application/json
    sensor:
      - name: "Curb Fridge Freezer"
        value_template: "{{ state_attr('sensor.curb_fridge_freezer', 'w')}}"
        json_attributes_path: "$.circuits.[0]"
        json_attributes:
          - w
        unique_id: curb-0
      - name: "Curb Office Bath and Old Kitchen"
        value_template: "{{ state_attr('sensor.curb_office_bath_and_old_kitchen', 'w')}}"
        json_attributes_path: "$.circuits.[1]"
        json_attributes:        
          - w
        unique_id: curb-1
      - name: "Curb AC Master"
        value_template: "{{ state_attr('sensor.curb_ac_master', 'w')}}"
        json_attributes_path: "$.circuits.[2]"
        json_attributes:
          - w
        unique_id: curb-2
      - name: "Curb AC Living Room"
        value_template: "{{ state_attr('sensor.curb_ac_living_room', 'w')}}"
        json_attributes_path: "$.circuits.[3]"
        json_attributes:
          - w 
        unique_id: curb-3
      - name: "Curb Washer and Dryer"
        value_template: "{{ state_attr('sensor.curb_washer_and_dryer', 'w')}}"
        json_attributes_path: "$.circuits.[4]"
        json_attributes:
          - w
        unique_id: curb-4
      - name: "Curb Kitchen and Mini Bar"
        value_template: "{{ state_attr('sensor.curb_kitchen_and_mini_bar', 'w')}}"
        json_attributes_path: "$.circuits.[5]"
        json_attributes:
          - w
        unique_id: curb-5
      - name: "Curb Water Pump and Deck"
        value_template: "{{ state_attr('sensor.curb_water_pump_and_deck', 'w')}}"
        json_attributes_path: "$.circuits.[6]"
        json_attributes:
          - w
        unique_id: curb-6
      - name: "Curb AC Baby Room"
        value_template: "{{ state_attr('sensor.curb_ac_baby_room', 'w')}}"
        json_attributes_path: "$.circuits.[7]"
        json_attributes:
          - w      
        unique_id: curb-7
      - name: "Curb Baby Room and Bathroom"
        value_template: "{{ state_attr('sensor.curb_baby_room_and_bathroom', 'w')}}"
        json_attributes_path: "$.circuits.[8]"
        json_attributes:
          - w
        unique_id: curb-8
      - name: "Curb Dishwasher"
        value_template: "{{ state_attr('sensor.curb_dishwasher', 'w')}}"
        json_attributes_path: "$.circuits.[9]"
        json_attributes:
          - w
        unique_id: curb-9
      - name: "Curb Office Equipment"
        value_template: "{{ state_attr('sensor.curb_office_equipment', 'w')}}"
        json_attributes_path: "$.circuits.[10]"
        json_attributes:
          - w
        unique_id: curb-10
      - name: "Curb Living and Dining"
        value_template: "{{ state_attr('sensor.curb_living_and_dining', 'w')}}"
        json_attributes_path: "$.circuits.[11]"
        json_attributes:
          - w 
        unique_id: curb-11
      - name: "Curb AC Office"
        value_template: "{{ state_attr('sensor.curb_ac_office', 'w')}}"
        json_attributes_path: "$.circuits.[12]"
        json_attributes:
          - w
        unique_id: curb-12
      - name: "Curb Master Bathroom"
        value_template: "{{ state_attr('sensor.curb_master_bathroom', 'w')}}"
        json_attributes_path: "$.circuits.[13]"
        json_attributes:
          - w
        unique_id: curb-13
      - name: "Curb Garage Door"
        value_template: "{{ state_attr('sensor.curb_garage_door', 'w')}}"
        json_attributes_path: "$.circuits.[14]"
        json_attributes:
          - w
        unique_id: curb-14

Remember to add unique_id to each sensor otherwise when you reload the YAML you’ll end up like me trying to delete and purge over 100 duplicated entities created by each restart.

tom_l · May 11, 2023, 2:28am

You can simplify it like this:

rest:
  - resource: https://app.energycurb.com/api/latest/{{location_id}}
    scan_interval: 10
    method: GET
    headers:
      Authorization : !secret curb_secret_token
      Content-Type: application/json
    sensor:
      - name: "Curb Fridge Freezer"
        value_template: "{{ value_json.circuits[0].w }}"
        unique_id: curb-0
      - name: "Curb Office Bath and Old Kitchen"
        value_template: "{{ value_json.circuits[1].w }}"
        unique_id: curb-1
      - name: etc...

luisgarcia87 · May 11, 2023, 6:42pm

Thanks again, I just did the changes and it’s indeed less lines on it.

  - resource: https://app.energycurb.com/api/latest/32d29fe8-be29-4184-a159-7fde34a4ff34
    scan_interval: 10
    method: GET
    headers:
      Authorization : !secret curb_secret_token
      Content-Type: application/json
    sensor:
      - name: "Curb Fridge Freezer"
        device_class: power
        unit_of_measurement: w
        value_template: "{{ value_json.circuits[0].w }}"
        unique_id: curb-0

I added the device_class and the unit_of_measurment so I can create another sensor to be able to display it on the Energy Dashboard:

I’ve converted them from a power (W) sensor to a energy (KwH) sensor using the Reimann Integartion

  - platform: integration
    source: sensor.curb_fridge_freezer
    name: energy_spent_curb_fridge_freezer
    unit_prefix: k
    round: 2

Unfortunately the newly created sensor does not show on the energy dashboard. But it does show up as a sensor on HA:

As you can guess the follow up question is: How can I make the new energy sensors show up in my energy dashboard?

tom_l · May 12, 2023, 12:38am

Change the unit of the rest sensors to a capital “W”. To be used in the energy dashboard the Riemann sensor must have a unit of Wh or kWh, the capitalisation is important.

luisgarcia87 · May 12, 2023, 1:36am

Ok i changed all rest sensors to a capital “W”

    sensor:
      - name: "Curb Fridge Freezer"
        device_class: power
        unit_of_measurement: "W"
        value_template: "{{ value_json.circuits[0].w }}"
        unique_id: curb-0

And the Riemann Integration works because the new energy sensor shows up as a kwh sensor:

But I am not able to add that Riemann sensor to the Energy Dashboard under the Individual Devices section, because it does not show up.

luisgarcia87 · May 12, 2023, 1:51am

*** UPDATE FOR AUTO-RENEW ACCESS TOKEN ****

As you know the token for the API lasts only 24 hours, and then you have to make another curl POST to get the new access_token…

Well I figure it out how to auto-refresh that token automatically so the sensors would get the updated token every time.

Here’s what I did:

First I created a rest sensor that would call the API and retrieve the access_token in json format:

### CURB ACCESS_TOKEN REFRESH ###
  - platform: rest
    name: curb_token_request
    resource: https://energycurb.auth0.com/oauth/token
    method: POST
    headers:
      Content-Type: application/json
      Cache-Control: no-cache
    payload: '{"grant_type": "password", "audience": "app.energycurb.com/api", "username": "{{USERNAME}}", "password": "{{PASSWORD}}", "client_id": "iKAoRkr3qyFSnJSr3bodZRZZ6Hm3GqC3", "client_secret": "dSoqbfwujF72a1DhwmjnqP4VAiBTqFt3WLLUtnpGDmCf6_CMQms3WEy8DxCQR3KY"}'  
    value_template: 'OK'
    scan_interval: 43200
    json_attributes:
      - access_token

Replace {{USERNAME}} and {{PASSWORD}} with your own credentials.

You can’t have the access_token on the value_template section because it exceeds 255 characters limited by HA. But we CAN use the sensor’s attribute to call it from the rest integration. Also I set a scan_interval of 43200 seconds (12 hours) even tough the access_token last for 24 hours, just to play it safe.

To do this I modified the rest integration (not the rest sensor platform) to use the attribute on the sensor.curb_token_request on the Authorization parameter.

rest:
  - resource: https://app.energycurb.com/api/latest/32d29fe8-be29-4184-a159-7fde34a4ff34
    scan_interval: 10
    method: GET
    headers:
      Authorization: >
        Bearer {{ state_attr('sensor.curb_token_request', 'access_token') }}
      Content-Type: application/json

Now the integration does not need us to refresh the access_token every 24 hours. It will auto renew.

tom_l · May 12, 2023, 2:01am

Go to Developer Tools → Statistics. Are there errors there to fix?

luisgarcia87 · May 12, 2023, 12:11pm

You are correct!

There were errors on the statistics.

After I fixed the issue the sensor showed up on the energy dashboard. Thanks

Pub · August 3, 2023, 7:11pm

First of all let me say thank you. I have been trying to find a way to integrate Curb into HA for over a year.

I got everything working up until I tried to automate the token refresh. When I view the sensor.curb_token_request it shows an access_token but the sensors no longer are giving data. I tested the token it gave me using it in a CURL request and it allows me to pull data but for some reason it will not work within HA.
I am using this in my sensors.yaml

### CURB ACCESS_TOKEN REFRESH ###
  - platform: rest
    name: curb_token_request
    resource: https://energycurb.auth0.com/oauth/token
    method: POST
    headers:
      Content-Type: application/json
      Cache-Control: no-cache
    payload: '{"grant_type": "password", "audience": "app.energycurb.com/api", "username": "myemail.com", "password": "mypassword", "client_id": "iKAoRkr3qyFSnJSr3bodZRZZ6Hm3GqC3", "client_secret": "dSoqbfwujF72a1DhwmjnqP4VAiBTqFt3WLLUtnpGDmCf6_CMQms3WEy8DxCQR3KY"}'  
    value_template: 'OK'
    scan_interval: 43200
    json_attributes:
      - access_token

Followed by this in my configuration.yaml

rest:
  - resource: https://app.energycurb.com/api/latest/045ae1b3-e47b-479d-ad65-6904e02c9c37
    scan_interval: 10
    method: GET
    headers:
      Authorization: >
        Bearer "{{ state_attr('sensor.curb_token_request', 'access_token') }}"
      Content-Type: application/json
    sensor:
      - name: "Living Room outlets"
        value_template: "{{ value_json.circuits[0].w }}"
        unique_id: curb-0

If I remove the automated token update and just use my secrets.yaml with the token it works with no issues.

Pub · August 3, 2023, 7:24pm

After reviewing the code it looks like removing the “” from this part of the code fixed it.
Your code:

 Authorization: >
        Bearer "{{ state_attr('sensor.curb_token_request', 'access_token') }}"

Working code for me:

Authorization: >
        Bearer {{ state_attr('sensor.curb_token_request', 'access_token') }}

luisgarcia87 · August 3, 2023, 7:40pm

You are right I just checked my working code and it does not have the “ on it. I didn’t copy the updated code when doing my post. Glad you got it working.

If you find a way to get webhooks connection to curb’s working on home assistant to get live data and not 1 min refresh data, let me know, but for now this will work.

Pub · August 3, 2023, 10:28pm

My coding knowledge is based off chatGPT and copying other peoples code

I made a new thread with an issue I can’t figure out. One of my sensors from curb I am using to see how much power I am returning to grid (NET) which outputs a negative value. Using the sensor for the energy dashboard gives improper data because of the negative value when I am returning power to the grid. I cannot figure out how to properly utilize this sensor so I can have data on energy returned to the grid.

Heres my thread Energy Dashboard Sensor Help

Let me buy you a beer for at least figuring out how to get the sensors into HA.

luisgarcia87 · August 4, 2023, 1:32am

I just replied to your new post regarding negative values. I’m always up for a beer!

Pub · August 5, 2023, 9:28am

Bumping this thread again because I updated to the newest HA and it seems to have broke everything. What version are you running?

The system cannot restart because the configuration is not valid: Invalid config for [rest]: expected SensorStateClass or one of ‘measurement’, ‘total’, ‘total_increasing’ for dictionary value @ data[‘rest’][0][‘sensor’][12][‘state_class’]. Got None. (See /config/configuration.yaml, line 201).

Line 201

rest:
  - resource: https://app.energycurb.com/api/latest/045ae1b3-e47b-479d-ad65-6904e02c9c37
    scan_interval: 10
    method: GET
    headers:
      Authorization: >
        Bearer {{ state_attr('sensor.curb_token_request', 'access_token') }}
      Content-Type: application/json