Template in payload for Tibber rest sensor not behaving as expected

Mario2 · December 7, 2021, 11:52am

Hi,

I’m struggling with making a rest sensor with a template payload. I need the POST to fetch the power consumption for the current month, and not the last 31 days.

The query works fine with a fixed parameter of 31, and returns power consumption for the last 31 days.

sensor:
  - platform: rest
    name: Tibber This Month
    resource: https://api.tibber.com/v1-beta/gql
    method: POST
    payload: '{ "query": "{ viewer { homes { consumption(resolution: DAILY, last: 31) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }'
    json_attributes_path: "$.data.viewer.homes[0].consumption"
    json_attributes:
      - nodes
    value_template: 'Ok'
    headers:
      Authorization: !secret tibber_token
      Content-Type: application/json
      User-Agent: REST

It returns:

{
   "data":{
      "viewer":{
         "homes":[
            {
               "consumption":{
                  "nodes":[
                     {
                        "from":"2021-11-06T00:00:00.000+01:00",
                        "to":"2021-11-07T00:00:00.000+01:00",
                        "cost":68.4396865,
                        "unitPrice":0.98016,
                        "unitPriceVAT":0.196032,
                        "consumption":69.825,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-07T00:00:00.000+01:00",
                        "to":"2021-11-08T00:00:00.000+01:00",
                        "cost":45.1807827875,
                        "unitPrice":0.881164,
                        "unitPriceVAT":0.176233,
                        "consumption":51.274,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-08T00:00:00.000+01:00",
                        "to":"2021-11-09T00:00:00.000+01:00",
                        "cost":78.4728869125,
                        "unitPrice":1.710059,
                        "unitPriceVAT":0.342012,
                        "consumption":45.889,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-09T00:00:00.000+01:00",
                        "to":"2021-11-10T00:00:00.000+01:00",
                        "cost":77.3141810875,
                        "unitPrice":1.047816,
                        "unitPriceVAT":0.209563,
                        "consumption":73.786,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-10T00:00:00.000+01:00",
                        "to":"2021-11-11T00:00:00.000+01:00",
                        "cost":43.9943054125,
                        "unitPrice":1.096841,
                        "unitPriceVAT":0.219368,
                        "consumption":40.11,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-11T00:00:00.000+01:00",
                        "to":"2021-11-12T00:00:00.000+01:00",
                        "cost":54.1913124625,
                        "unitPrice":1.122973,
                        "unitPriceVAT":0.224595,
                        "consumption":48.257,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-12T00:00:00.000+01:00",
                        "to":"2021-11-13T00:00:00.000+01:00",
                        "cost":57.560875225,
                        "unitPrice":1.160127,
                        "unitPriceVAT":0.232025,
                        "consumption":49.616,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-13T00:00:00.000+01:00",
                        "to":"2021-11-14T00:00:00.000+01:00",
                        "cost":71.8537981375,
                        "unitPrice":1.158316,
                        "unitPriceVAT":0.231663,
                        "consumption":62.033,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-14T00:00:00.000+01:00",
                        "to":"2021-11-15T00:00:00.000+01:00",
                        "cost":68.3418974375,
                        "unitPrice":1.221744,
                        "unitPriceVAT":0.244349,
                        "consumption":55.938,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-15T00:00:00.000+01:00",
                        "to":"2021-11-16T00:00:00.000+01:00",
                        "cost":77.52910155,
                        "unitPrice":1.232166,
                        "unitPriceVAT":0.246433,
                        "consumption":62.921,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-16T00:00:00.000+01:00",
                        "to":"2021-11-17T00:00:00.000+01:00",
                        "cost":74.0528257375,
                        "unitPrice":1.205346,
                        "unitPriceVAT":0.241069,
                        "consumption":61.437,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-17T00:00:00.000+01:00",
                        "to":"2021-11-18T00:00:00.000+01:00",
                        "cost":63.9960246625,
                        "unitPrice":1.125383,
                        "unitPriceVAT":0.225077,
                        "consumption":56.866,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-18T00:00:00.000+01:00",
                        "to":"2021-11-19T00:00:00.000+01:00",
                        "cost":45.6506907125,
                        "unitPrice":1.128766,
                        "unitPriceVAT":0.225753,
                        "consumption":40.443,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-19T00:00:00.000+01:00",
                        "to":"2021-11-20T00:00:00.000+01:00",
                        "cost":43.0527578625,
                        "unitPrice":1.060935,
                        "unitPriceVAT":0.212187,
                        "consumption":40.58,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-20T00:00:00.000+01:00",
                        "to":"2021-11-21T00:00:00.000+01:00",
                        "cost":53.608823525,
                        "unitPrice":1.130392,
                        "unitPriceVAT":0.226078,
                        "consumption":47.425,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-21T00:00:00.000+01:00",
                        "to":"2021-11-22T00:00:00.000+01:00",
                        "cost":68.1558335125,
                        "unitPrice":1.229495,
                        "unitPriceVAT":0.245899,
                        "consumption":55.434,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-22T00:00:00.000+01:00",
                        "to":"2021-11-23T00:00:00.000+01:00",
                        "cost":86.55705785,
                        "unitPrice":1.448629,
                        "unitPriceVAT":0.289726,
                        "consumption":59.751,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-23T00:00:00.000+01:00",
                        "to":"2021-11-24T00:00:00.000+01:00",
                        "cost":87.7003419125,
                        "unitPrice":1.304017,
                        "unitPriceVAT":0.260803,
                        "consumption":67.254,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-24T00:00:00.000+01:00",
                        "to":"2021-11-25T00:00:00.000+01:00",
                        "cost":72.8653619875,
                        "unitPrice":1.374325,
                        "unitPriceVAT":0.274865,
                        "consumption":53.019,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-25T00:00:00.000+01:00",
                        "to":"2021-11-26T00:00:00.000+01:00",
                        "cost":82.81616345,
                        "unitPrice":1.434121,
                        "unitPriceVAT":0.286824,
                        "consumption":57.747,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-26T00:00:00.000+01:00",
                        "to":"2021-11-27T00:00:00.000+01:00",
                        "cost":74.0476812,
                        "unitPrice":1.488007,
                        "unitPriceVAT":0.297601,
                        "consumption":49.763,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-27T00:00:00.000+01:00",
                        "to":"2021-11-28T00:00:00.000+01:00",
                        "cost":99.7744653,
                        "unitPrice":1.950206,
                        "unitPriceVAT":0.390041,
                        "consumption":51.161,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-28T00:00:00.000+01:00",
                        "to":"2021-11-29T00:00:00.000+01:00",
                        "cost":106.976069475,
                        "unitPrice":2.395398,
                        "unitPriceVAT":0.47908,
                        "consumption":44.659,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-29T00:00:00.000+01:00",
                        "to":"2021-11-30T00:00:00.000+01:00",
                        "cost":143.3477653375,
                        "unitPrice":2.984174,
                        "unitPriceVAT":0.596835,
                        "consumption":48.036,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-11-30T00:00:00.000+01:00",
                        "to":"2021-12-01T00:00:00.000+01:00",
                        "cost":114.7596663375,
                        "unitPrice":1.655004,
                        "unitPriceVAT":0.331001,
                        "consumption":69.341,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-12-01T00:00:00.000+01:00",
                        "to":"2021-12-02T00:00:00.000+01:00",
                        "cost":97.4594623875,
                        "unitPrice":1.484802,
                        "unitPriceVAT":0.29696,
                        "consumption":65.638,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-12-02T00:00:00.000+01:00",
                        "to":"2021-12-03T00:00:00.000+01:00",
                        "cost":110.122295375,
                        "unitPrice":2.129037,
                        "unitPriceVAT":0.425807,
                        "consumption":51.724,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-12-03T00:00:00.000+01:00",
                        "to":"2021-12-04T00:00:00.000+01:00",
                        "cost":150.9669601,
                        "unitPrice":1.854768,
                        "unitPriceVAT":0.370954,
                        "consumption":81.394,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-12-04T00:00:00.000+01:00",
                        "to":"2021-12-05T00:00:00.000+01:00",
                        "cost":108.9824185625,
                        "unitPrice":1.722116,
                        "unitPriceVAT":0.344423,
                        "consumption":63.284,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-12-05T00:00:00.000+01:00",
                        "to":"2021-12-06T00:00:00.000+01:00",
                        "cost":110.660477,
                        "unitPrice":1.59359,
                        "unitPriceVAT":0.318718,
                        "consumption":69.441,
                        "consumptionUnit":"kWh"
                     },
                     {
                        "from":"2021-12-06T00:00:00.000+01:00",
                        "to":"2021-12-07T00:00:00.000+01:00",
                        "cost":110.5067917125,
                        "unitPrice":2.064351,
                        "unitPriceVAT":0.41287,
                        "consumption":53.531,
                        "consumptionUnit":"kWh"
                     }
                  ]
               }
            }
         ]
      }
   }
}

But when i try to get data for the current month, it fails:

sensor:
  - platform: rest
    name: Tibber This Month
    resource: https://api.tibber.com/v1-beta/gql
    method: POST
    payload: '{ "query": "{ viewer { homes { consumption(resolution: DAILY, last: {{ (now().day-1) }}) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }'
    json_attributes_path: "$.data.viewer.homes[0].consumption"
    json_attributes:
      - nodes
    value_template: 'Ok'
    headers:
      Authorization: !secret tibber_token
      Content-Type: application/json
      User-Agent: REST

This is shown in the log:

> 2021-12-07 12:20:42 DEBUG (MainThread) [homeassistant.components.rest.sensor] Data fetched from resource: {"error":"syntax error"}

After reading several threads on the forum it seems like it should be possible to use templates in the payload. After testing with several template values, it seems like they are working, but that I’m probably have a fault in my template.

According to the Tibber documentation the last parameter should be an int. When I try:

    payload: '{ "query": "{ viewer { homes { consumption(resolution: DAILY, last: {{ (now().day-1)|int }}) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }'

I get a different reply:

> 2021-12-07 12:26:00 DEBUG (MainThread) [homeassistant.components.rest.sensor] Data fetched 
 from resource: {"errors":[{"message":"Syntax Error: Expected Name, found {","locations":[{"line":1,"column":58}],"extensions":{"code":"GRAPHQL_PARSE_FAILED"}}]}

For another forum post it was recomended to use double quotes,

    payload: '{ "query": "{ viewer { homes { consumption(resolution: DAILY, last: "{{ (now().day-1)|int }}") { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }'

but then I get:

> 2021-12-07 12:36:01 DEBUG (MainThread) [homeassistant.components.rest.sensor] Data fetched from resource: {"error":"syntax error"}

I have also tried to slice the reply into the attributes without sucsess, it seems like the json_attributes doesn’t accept templates?

    json_attributes: '{{ value_json.data.viewer.homes[0].consumption.nodes[-(now().day-1)]}}'

Not sure if this is a Tibber API or rest sensor/template question. Do anyone see what I’m doing wrong? Any hints on how I can proceed to get this working?

tom_l · December 7, 2021, 12:07pm

Pretty sure payload does not support templates:

Screenshot 2021-12-07 at 23-05-22 RESTful Sensor

Compare:

Mario2 · December 7, 2021, 12:17pm

Thanks for the quick reply.

Hmm, I was afraid that was the case. I saw that the rest command documentation had Template as type, but the rest sensor didn’t.

I did play a lot around with the resource_template, instead of resource, but i wasn’t able to achieve anything else than fault messages. (I’m quite new to REST API’s)

      resource_template: 'https://api.tibber.com/v1-beta/gql/{"query":"{viewer{homes{consumption(resolution: DAILY, last: {{ (now().day-1)|int }}){nodes{from to cost unitPrice unitPriceVAT consumption consumptionUnit}}}}}"}'

      resource_template: 'https://api.tibber.com/v1-beta/gql?{"query":"{viewer{homes{consumption(resolution: DAILY, last: {{ (now().day-1)|int }}){nodes{from to cost unitPrice unitPriceVAT consumption consumptionUnit}}}}}"}'

      resource_template: 'https://api.tibber.com/v1-beta/gql?data={"query":"{viewer{homes{consumption(resolution: DAILY, last: {{ (now().day-1)|int }}){nodes{from to cost unitPrice unitPriceVAT consumption consumptionUnit}}}}}"}'

Any hint on how the payload should be included in the resource_template string?

tom_l · December 7, 2021, 12:25pm

There was a big thread on templating inside a json payload I’ll see if I can find it.

Meanwhile, you probably don’t need to do this:

{{ (now().day-1)|int }}

This will return an integer:

{{ now().day-1 }}

EDIT: Have a read of this: https://community.home-assistant.io/t/problem-with-creating-a-json-like-string-message-in-notify-automation/297134

Mario2 · December 7, 2021, 2:05pm

Thanks again, that was interesting reading.

Just realized that my JSON consists of only one key/value pair, where the key is “query” and the value is the graphQL string.

Made some tests in the Dev->Template:

{% set original = '{ "query": "{ viewer { homes { consumption(resolution: DAILY, last: 31) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }'%}
{% set with_replace = '{ "query": "{ viewer { homes { consumption(resolution: DAILY, last: now().day) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }' %}

original:
{{ original }}

with_replace:
{{ with_replace|replace("now().day", now().day-1) }}

As a string:
{ "query": "{ viewer { homes { consumption(resolution: DAILY, last: 29) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }

and it outputs:

original:
{ "query": "{ viewer { homes { consumption(resolution: DAILY, last: 31) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }

with_replace:
{ "query": "{ viewer { homes { consumption(resolution: DAILY, last: 6) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }

As a string:
{ "query": "{ viewer { homes { consumption(resolution: DAILY, last: 29) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }

They look quite equal to me (except for the last: x ). Still, only the last one works with the rest sensor.

I’v also tested to cut n’ paste the template result into Tibbers API explorer, and the work nicely.

Is it possible something is done to the string when it is passed as a template? Is there anyway to get some debug logs from the POST that is made by the rest sensor?

tom_l · December 7, 2021, 7:33pm

Delete this )

{% set with_replace = '{ "query": "{ viewer { homes { consumption(resolution: DAILY, last: now().day) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }' %}
                                                                                                    ^
                                                                                                    |

Mario2 · December 7, 2021, 9:31pm

Hmm, that didn’t help, gives me the same reply:

2021-12-07 22:00:44 DEBUG (MainThread) [homeassistant.components.rest.sensor] Data fetched from resource: {"error":"syntax error"}

I belive it is supposed to be there, its the closing bracket for the “consumption” parameters. The string I’m trying to build:

{"query" :
	"{
  	viewer {
      homes {
        consumption(resolution: DAILY, last: 6) {
          nodes {
            from
            to
            cost
            unitPrice
            unitPriceVAT
            consumption
            consumptionUnit
          }
        }
      }
    }"
  }

So when I put the same template string for value_template

sensor:
  - platform: rest
    value_template: >
      { "query": "{ viewer { homes { consumption(resolution: DAILY, last: {{ now().day-1 }}) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }

the state of the sensor is set to:

{ "query": "{ viewer { homes { consumption(resolution: DAILY, last: 6) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }

The funny thing is when I try to pass it in a persistant notification like in the other thread you linked:

service: notify.persistent_notification
data: 
  title: Hei
  message: >
    { "query": "{ viewer { homes { consumption(resolution: DAILY, last: {{ now().day-1 }}) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }

It fails as well with the following error:


2021-12-07 22:22:32 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Invalid data for call_service at pos 1: template value should be a string for dictionary value @ data['message']
2021-12-07 22:22:32 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [547954692592] Error handling message: template value should be a string for dictionary value @ data['message']. Got None

I’m starting to belive that the rest sensor isn’t the way to go for POST commands with dynamic variables. As long as there isn’t a way to debug the output of the POST, it’s really hard to figure what the payload string compiles to.

I found some threads earlier discussing templates in the payload string, guess I’ll read them again to see if I missed something.

@tom_l , thanks for the effort trying to help me with this. I could always try to solve it with node-red or appdeamon, its just nice keeping all the config for a set of functions at the same place.

petro · December 8, 2021, 12:54pm

What payload does the device require without using templates? Just post it. I’ll fix it for your resource_template.

petro · December 8, 2021, 1:01pm

Well I’m impatient, so here’s what your resource_template would look like:

Use multliline template to get rid of quoting confusion.

      resource_template: >
         {% raw %}https://api.tibber.com/v1-beta/gql?data={"query":"{viewer{homes{consumption(resolution: DAILY, last: {% endraw %}{{ now().day-1) }}{% raw %}){nodes{from to cost unitPrice unitPriceVAT consumption consumptionUnit}}}}}"}{% endraw %}

Basically, the problem is that the template editor see’s { and immediately wants to use it for templates. When it can’t it screws up. So you need to just remove it looking at that information by using {% raw %} {% endraw %}. You can put anything between {% raw %} and {% endraw %} and templates will completely ignore the contents.

Mario2 · December 9, 2021, 7:58am

Thanks a lot @petro ! Really appreciate it, you guys and this forum are awesome!

The {% raw %} notation is new to me, guess I didn’t read the Jinja docs good enough.

After a couple of hours of forum scanning, I came across several similar topic. They mostly concluded with giving up and using other solutions, so I did to. I have tested with a command line sensor using curl, and it seems to work. But as long as I use other rest sensors, the best solution is to get all of the sensors on the same platform and format. Also the curl sensor doesn’t pars the reply just as nice as the rest sensor.

You revived my faith in the rest sensor, I’m gonna continue testing this afternoon

Mario2 · December 10, 2021, 8:58am

Hi,

So now I have tested some more, but without result.

After performing some test i postman and some debugging in chrome from Tibbers own API explorer where you can do query’s directly.

So, when i enter the the URL in post man as you suggested:

https://api.tibber.com/v1-beta/gql?data={ "query"= "{ viewer { homes { consumption(resolution: DAILY, last: 1) { nodes { from to cost unitPrice unitPriceVAT consumption consumptionUnit }}}}}" }

I get the reply:

POST body missing. Did you forget use body-parser middleware?

When adding the JSON to the request body, and removed all parematers

{
  "query":"{viewer{homes{consumption(resolution: DAILY, last: 1){nodes{from to cost unitPrice unitPriceVAT consumption consumptionUnit}}}}}"
}

The URL in postman is set to:

https://api.tibber.com/v1-beta/gql

and a valid relpy was received:

{
    "data": {
        "viewer": {
            "homes": [
                {
                    "consumption": {
                        "nodes": [
                            {
                                "from": "2021-12-09T00:00:00.000+01:00",
                                "to": "2021-12-10T00:00:00.000+01:00",
                                "cost": 38.134279075,
                                "unitPrice": 0.533361,
                                "unitPriceVAT": 0.106672,
                                "consumption": 71.498,
                                "consumptionUnit": "kWh"
                            }
                        ]
                    }
                }
            ]
        }
    }
}

I’m on thin ice when it comes to REST API’s and HTML commands, but after some reading, my impression is that in a POST only parameters will be a part of the URL. But the GraphQl query must be sent in the request body. (How are parameters sent in an HTTP POST request? - Stack Overflow)

So if this was a GET command I can use a template in resource_template to pass parameters, but for a POST with a request body, payload must be used.

Or am I wrong?

jeffromgbg · October 19, 2022, 7:55am

Hi, this configuration worked for me:

sensor:
  - platform: rest
    name: tibber_current_price
    resource: https://api.tibber.com/v1-beta/gql
    method: POST
    payload: '{ "query": "{viewer {homes {currentSubscription {priceInfo {current {total energy tax startsAt }}}}}}" }'
    value_template: "{{ value_json.data.viewer.homes[0].currentSubscription.priceInfo.current.total }}"
    scan_interval: 3600
    unit_of_measurement: SEK/kWh
    headers:
      Authorization: !secret tibber_sensor_secret_token
      Content-Type: "application/json"