Restfull sensor: when is it updated, how to extract values?

gonzotek · December 8, 2022, 3:19pm

Sorry for the delay replying, busy week. It should be working. I gave an example of the service call, but not the entire automation. If you want to test to verify how update_entity service and the scan_interval setting both work, the World Time API would be an easy to use api that should clearly demonstrate what’s going on under the hood.

For example, I placed this in my configuration.yaml

  - resource: "http://worldtimeapi.org/api/timezone/America/New_York"
    scan_interval: 31536000
    sensor:
      - name: "WT API Current Time"
        value_template: >-
          {{ value_json["datetime"].split("T")[1].split(".")[0] }}

Which will give me a nicely formatted hh:mm:ss string in the sensor state.

Once I reloaded the rest entities it pulled down the time, but then it didn’t change again (caveats: it would have changed after 365 days or if I reloaded rest or restarted Home Assistant). But if I use the services tab of the developer tools, I can call update_entity on it and get an updated time:

service: homeassistant.update_entity
target:
  entity_id: sensor.wt_api_current_time

If I wanted to use this in a complete automation, it might look like this:

- id: fc70353e-11ec-4e6a-8248-5510654eb6ca
  alias: Mise à jour jours fériés
  trigger:
    - platform: time
      at: input_datetime.jour_ferie_update
  action:
    service: homeassistant.update_entity
    target:
      entity_id: sensor.wt_api_current_time

Also keep in mind, that since the data in your case doesn’t change often, the sensor state obviously wouldn’t update until the json had changed (ie there will be no logged change noted unless the data has actually changed).

kolia · December 10, 2022, 2:33pm

Thank you so much for this continuing discussion. The example you posted makes it perfectly clear . It seems to me it is more secure to take control on when the API is called (and since 2022.12 we’ve got calendar that makes things even more straightforward) than relying on scan_interval just because I want an update at least the 27th of the last month of the year.

The final code:

- resource_template: "https://calendrier.api.gouv.fr/jours-feries/metropole/{{ now().year + 1 if (now().month >= 12 and now().day >= 27) else now().year }}.json"
  scan_interval: 31536000
  sensor:
    - name: 1er janvier
      unique_id: jf_1er_janvier
      value_template: >
        {{ value_json.items() | selectattr('1', 'eq', '1er janvier') | map(attribute='0') | first | default }}
    - name: 1er janvier suivant
      unique_id: jf_1er_janvier_suivant
      value_template: >
        {{ (now().year + 1) | string ~ "-01-01" }}
    - name: Lundi de Pâques
      unique_id: jf_1erjanvier_lundi_de_paques
      value_template: >
        {{ value_json.items() | selectattr('1', 'eq', 'Lundi de Pâques') | map(attribute='0') | first | default }}
    - name: 1er mai
      unique_id: jf_1er_mai
      value_template: >
        {{ value_json.items() | selectattr('1', 'eq', '1er mai') | map(attribute='0') | first | default }}
    - name: 8 mai
      unique_id: jf_8_mai
      value_template: >
        {{ value_json.items() | selectattr('1', 'eq', '8 mai') | map(attribute='0') | first | default }}
    - name: Ascension
      unique_id: jf_ascension
      value_template: >
        {{ value_json.items() | selectattr('1', 'eq', 'Ascension') | map(attribute='0') | first | default }}
    - name: Lundi de Pentecôte
      unique_id: jf_lundi_de_pentecote
      value_template: >
        {{ value_json.items() | selectattr('1', 'eq', 'Lundi de Pentecôte') | map(attribute='0') | first | default }}
    - name: 14 juillet
      unique_id: jf_14_juillet
      value_template: >
        {{ value_json.items() | selectattr('1', 'eq', '14 juillet') | map(attribute='0') | first | default }}
    - name: Assomption
      unique_id: jf_assomption
      value_template: >
        {{ value_json.items() | selectattr('1', 'eq', 'Assomption') | map(attribute='0') | first | default }}
    - name: Toussaint
      unique_id: jf_toussaint
      value_template: >
        {{ value_json.items() | selectattr('1', 'eq', 'Toussaint') | map(attribute='0') | first | default }}
    - name: 11 novembre
      unique_id: jf_11_novembre
      value_template: >
        {{ value_json.items() | selectattr('1', 'eq', '11 novembre') | map(attribute='0') | first | default }}
    - name: Jour de Noël
      unique_id: jf_jour_de_noel
      value_template: >
        {{ value_json.items() | selectattr('1', 'eq', 'Jour de Noël') | map(attribute='0') | first | default }}

and automation with a yearly task event in local calendar, the 27th of december:

- id: 4db3d6ea-dddc-4a9b-b509-ca66429c5db9
  alias: Mise à jour annuelle des jours fériés
  description: ''
  trigger:
  - platform: calendar
    event: start
    offset: 0:0:10
    entity_id: calendar.taches_recurrentes
  condition:
  - condition: template
    value_template: '{{ "days off update" in trigger.calendar_event.summary }}'
  action:
    - service: homeassistant.update_entity
      target:
        entity_id: 
          - sensor.1er_janvier
          - sensor.lundi_de_paques
          - sensor.1er_mai
          - sensor.8_mai
          - sensor.ascension
          - sensor.lundi_de_pentecote
          - sensor.14_juillet
          - sensor.assomption
          - sensor.toussaint
          - sensor.11_novembre
          - sensor.jour_de_noel    
    - service: telegram_bot.send_message
      data:
        message: Les jours fériés de la nouvelle année ont été mis à jour
        parse_mode: html
  mode: queued
  max: 10

VasMan · April 17, 2024, 9:54am

I am trying to extract a value for one of my API rest calls. I read this thread (and others) very carefully and tried to replicate the same but I dont understand what I am doing wrong and the sensor is not even created after HA restart. My API resource/URL is fixed and does not need any variables (i.e. resource_template). The API response is as follows:

[{"id":"0005B9433CAF","deviceType":"0005B9 - LTE_Enterprise_C-RANSC_Cntrl","deviceStatus":"Out Of Service","deviceStatusReason":"Device is not reachable"}]

Or as seen by HA service as follows:

content:
  - id: 0005B9433CAF
    deviceType: 0005B9 - LTE_Enterprise_C-RANSC_Cntrl
    deviceStatus: Out Of Service
    deviceStatusReason: Device is not reachable
status: 200

I want to present to one of my Dashboards the state of the Cell which is the value of deviceStatus. Based on examples I have seen here I have added the following in my configuration.yaml but I dont see any sensor being created. Any guidance on what I am doing wrong? Only 2 differences I notice between my API response and the one posted on the top is that a) my API response is [{ }] vs { } and b) there is no space after : in my API response, but I dont understand if these are important and have to be taken care somehow or not.

Many Thanks!

rest:
  - resource: "http://192.168.2.20:8230/api/devices/device/deviceStatus?deviceIds=0005B9433CAF&deviceType=0005B9 - LTE_Enterprise_C-RANSC_Cntrl"
    scan_interval: 30
    sensor:
      - name: BC1 4G Cell Status
        unique_id: bc1_4g_cell_status
        value_template: >
          {{ value_json.items() | selectattr('0', 'eq', 'deviceStatus') | map(attribute='1') | first | default }}

Troon · April 17, 2024, 10:05am

Always use the template editor to try out your templates:

So yes, the [{ }] is important, it’s a single item list which doesn’t have an items() method. You’re overthinking it though:

value_template: "{{ value_json[0]['deviceStatus'] }}"

VasMan · April 17, 2024, 11:03am

Thanks so much for the tip on the template validation and the explanation. Quite new to HA, coming from a telecoms background rather than IT/Development so learning slowly slowly.

I am sorry for raising another stupid question, but in my sensor where the json response comes when the API call gets trggered every 30seconds (i.e. is not set fixed like during the template testing), how do I define the json_value coming from the API response. The below does not work as I was expecting, so the [0] has to be replaced with something else, right?

rest:
  - resource: "http://192.168.2.20:8230/api/devices/device/deviceStatus?deviceIds=0005B9433CAF&deviceType=0005B9 - LTE_Enterprise_C-RANSC_Cntrl"
    scan_interval: 30
    sensor:
      - name: BC1 4G Cell Status
        unique_id: bc1_4g_cell_status
        value_template: "{{ value_json[0]['deviceStatus'] }}"

Troon · April 17, 2024, 11:33am

The [0] refers to the first (and only) item in the list that you started your first post with, and the whole template pulls the value associated with the deviceStatus key from that item.

What responses are you getting from the API, and what are you wanting to extract from them?

value_json refers to the API response as a whole (assuming it can be parsed as JSON), then you use more operations to narrow down what you want to extract.

VasMan · April 17, 2024, 12:49pm

So if I understand well, it looks like the format of the sensor configuration and more specifically the value_tamplate that you suggested are correct. So the value_json[0] should work to parse the response of the API call and give me the deviceStatus. So now I need to figure out why a sensor is not been created.

In summary what I was hopping to achieve was to create a sensor by including the confg I added in my last post into my configuration.yaml. End target is to show in one of my HA dashboards the deviceStatus (which can have 2 states based on the response from API call, these In Service and Out Of Serivice) every 30seconds, nothing more.

Now need to figure out why the sensor is not being created, maybe the response cant be pasrsed as JSON as you mentioned and this is why the sensor is not being created (?), but how can I know this?

Troon · April 17, 2024, 12:52pm

You mean you don’t even see the sensor in your system? Did you restart HA after adding that section to your config? Whenever you add a new top-level integration (like rest: here), you need to do a full restart.

If it’s a problem with the template (the bit in {{ }}), it’d show up but with a state of unknown or unavailable.

That makes me think you’re still not fully understanding what’s going on. There are two types of data structure in use here, lists and dictionaries (dict for short). Read more here.

value_json is the response as a whole, which from your first post is this, spaced out a bit:

[
  {
    "id":"0005B9433CAF",
    "deviceType":"0005B9 - LTE_Enterprise_C-RANSC_Cntrl",
    "deviceStatus":"Out Of Service",
    "deviceStatusReason":"Device is not reachable"
  }
]

This is a list, which is notated as ['first item', 'second item', ...]. This particular list only has one item, which is a dict:

{
  "id":"0005B9433CAF",
  "deviceType":"0005B9 - LTE_Enterprise_C-RANSC_Cntrl",
  "deviceStatus":"Out Of Service",
  "deviceStatusReason":"Device is not reachable"
}

A dict is a collection of key / value pairs. One of these, in this example, is the key deviceStatus, with the value Out Of Service.

So:

value_json is the entire response: the list containing the dict
value_json[0] is the first item of that list, which is the dict
value_json[0]['deviceStatus'] is the value of the deviceStatus key in that dict, which is the string Out Of Service.

kolia · April 17, 2024, 1:24pm

json path is a good way to understand how to navigate inside a json structure

VasMan · April 17, 2024, 2:50pm

Yes I have fully restarted HA multiple times with no luck, no sensor with that name created under is available when I search in the section under Settings/Devices & Services/Entities. Other sensors I have added manually in configuration.yaml appear fine.

Troon · April 17, 2024, 4:27pm

Are you sure you don’t have a rest: heading elsewhere in the file?

Any errors in the logs?

You’re not running some bonkers-old version of HA are you?

VasMan · April 17, 2024, 6:47pm

Yes, no other rest: heading in my configuration.yaml, when you add a 2nd one it complaints anyway as duplicate. Running latest HA as per below.

Not sure if there is any conflict with rest_command but I do run the following on my configuarion.yaml.

# Loads default set of integrations. Do not remove.
default_config:

# Load frontend themes from the themes folder
frontend:
  themes: !include_dir_merge_named themes

automation: !include automations.yaml
script: !include scripts.yaml
scene: !include scenes.yaml
rest_command: !include rest_command.yaml
cover: !include cover.yaml

I removed (commented out) the rest_command: !include rest_command.yaml and fully restarted HA but made no difference.

VasMan · April 17, 2024, 8:21pm

@ Troon thank you so much for ALL your help today and allowing me to add what I was looking for. The problem with the configuration.yaml was due to a recent update of HA, there was a warning saying I need overwrite my config file. I thought config file was automatically saved every time you make a change, that was my experience so far, but I guess a recent update of HA might have triggered some new format and hence the warning to overwrite existing file. After that and a new HA restart I can now see my sensor correctly