Add option to force query REST entities one at a time

Feature Requests
1

I know the RESTful integration can query a single API endpoint and get multiple data, from which multiple sensors can be created. I’m using that already.

However, I still have a need to declare multiple entities, to the same API/host for different data, on a meterbus (MBus). Currently, there is no way to prevent the scan_interval from colliding, querying the different entities at the same time, which causes all but one of the entities to return no data. MBus can only be queried serially, not parallel.

So, if one has multiple MBus meters on the same MBus master, one would need to be able to specify that all entites pointing to that same MBus master, must be queried sequentially, even if the scan_interval times for different entites on that MBus master elapse at the same time.

Could this be added, for instance by adding something like a “unique_query_id” property for the entity? Then you could have the code account for that by not querying entities with the same “unique_query_id” at the same time, but staggering them sequentially, if and when the scan_interval elapses simultaneously for those entities.

2

I think you can already do this.

Set your scan interval to 0 to disable automatic updates, and then at times of your choosing you can call homeassistant.update_entity on one of the entities per rest integration, and it will do updates only when you want them to, so you can schedule your queries serially.

3

Yes I have read that somewhere, but that kind of feels like an unneccessarily obtuse workaround for a problem that, in my opinion, should be just fixed in the integration itself.

The purpose of the RESTful integration should be to easily create a range of sensors, controlled by some common sense.

Defeating the functionality of a core feature, scanning sensors at regular intervals, just to need some other workaround to get the desired data, does not seem like a logical means to an end.

What if you have many, many devices, each with many sensors. Do you need to create a separate automation for each and every entity and it’s sensors?

I’m sorry, but that feels silly, when, again in my opinion, there’s a simple and robust fix to be made that would benefit everyone at once. Just don’t query some entities at the same time, via some control method…

4

You’re new here? Lol

There has been frequent debate over this update issue. Refer to the ping changes and the speedtest changes, there are possibly others too.

5

Sorry but for now and the foreseeable future this is the accepted method.

6

Well, if that’s the case, I’ll look into doing it that way for the foreseeable future :wink:

The RESTful integration cannot be managed via the UI (yet?), so the instructions are not entirely complete for this specific case. Perhaps you could add a paragraph to explain how to disable polling for integrations that can only be configured via configuration.yaml?

7

image
image918×441 30.9 KB

Worse case, if the integration doesn’t accept 0, set an extremely large number like 3600000

8

I’ve been trying to use an automation to issue “update_entity” service_calls for my 2 separate MBus API calls, without collisions in the time of execution.

But, no matter what I try, I cannot seem to prevent simultaneous calls from occurring regularly, causing some entities to report ‘unknown’ in stead of the expected energy or power values.

I’ve set the scan_interval to 0 in the RESTful configuration, and defined 2 separate resources to the same API endpoint, but each with a different MBus address.

First, I tried this: in the “action” section of a single automation, I started a sequence with update_entity of entities defined in the first resource, then a delay, and then update_entity of entities defined in the second resource. According to several examples I found, this should call the execution in order, separated by the specified delay.

This did not prevent simultaneous calls from occurring sometimes.

So, I tried something else: now I start the action sequence by setting a variable to a random number (1 or 2), and depending on the outcome, call an update_entity for just the entities defined in one of the 2 API endpoints. This mechanism by itself seems to work, but still, this does not prevent simultaneous calls to the 2 different endpoints from occurring regularly.

It seems like some execution is delayed or somehow triggered outside of the running automation, and collides with a next execution, even though I have set the automation mode to “single”.

So, I would ask:

What would be a guaranteed way to prevent simultaneous calls to be made?

Below is my current automation config:

alias: Update MBus Entities
description: ""
trigger:
  - platform: time_pattern
    seconds: /7
condition: []
action:
  - variables:
      num: "{{ range(1,3)|random }}"
  - choose:
      - conditions:
          - condition: template
            value_template: "{{ num == 1 }}"
        sequence:
          - service: homeassistant.update_entity
            metadata: {}
            data: {}
            target:
              entity_id:
                - sensor.solar_system_active_power
                - sensor.solar_system_current
                - sensor.solar_system_frequency
                - sensor.solar_system_power_factor
                - sensor.solar_system_reactive_power
                - sensor.solar_system_voltage
            enabled: true
      - conditions:
          - condition: template
            value_template: "{{ num == 2 }}"
        sequence:
          - service: homeassistant.update_entity
            metadata: {}
            data: {}
            target:
              entity_id:
                - sensor.solar_system_energy_production_total
                - sensor.solar_system_energy_consumption_total
            enabled: true
mode: single
9

Ok, after some more time spent, I’m beginning to suspect that HomeAssistant itself is just not executing the actual commands immediately when the ‘update_entity’ service is called!

Everything I have debugged, seems to indicate that my automation(s) I’ve tried and tested, do exactly work as I intend, i.e. delaying the sequential ‘update_entity’ calls in the first attempt, and randomly selecting 1 of the 2 actions in the second attempt.

Inspecting the debug-log for homeassistant.core, it also shows that the ‘update_entity’ calls are fired (or at least logged) exactly at the expected times, every /7 seconds in my latest automation configuration.

However! … and here comes my shocked facial expression … the actual “action” to update the entities, in my case the RESTful URL call to my mbus-httpd API container, as defined in the RESTful integration configuration file, is often not performed at that same time!

Sometimes there’s a significant delay after the ‘update_entity’ is called/logged, before the actual API call is made to the RESTful resource. I’ve seen delays ranging from 1 to many seconds, and if it happens to be around 7 seconds between 2 different ‘update_entity’ calls, that causes a contention on the MBus, and failed updates as a result.

So, I guess the core issue is HomeAssistant’s scheduling of the actual commands needed to effectuate the ‘update_entity’ calls? Who knew?

Well, does anyone have some suggestions on how to proceed? I’d rather not increase the time between updates, especially for the instant power values from the Energy meter.

10

That’s the rest coordinator. It coordinators the rest calls to ensure the sensors that all use the same endpoint get hit once. It can’t do that if you separate the endpoint into 2 separate configurations.

The way you’re using this, you should really switch to a rest command with a response variable.

11

Maybe I don’t fully understand yet, or perhaps I have my terminology confused, but there are really 2 different endpoints as I see it, and each set of entities that I call an update_entity for, are for a single one of those endpoints.

To put all my proverbial cards on the table, here’s (part of) my RESTful configuration for the sensors (non-essential info left out):

- scan_interval: 0
  resource: http://localhost:8085/mbus/get/ttyUSB0/9600/1
  sensor:
    - name: "Solar System Energy Production Total"
      unique_id: solar_system_energy_production_total
    - name: "Solar System Energy Consumption Total"
      unique_id: solar_system_energy_consumption_total
- scan_interval: 0
  resource: http://localhost:8085/mbus/get/ttyUSB0/9600/10010000FFFFFFFF
  sensor:
    - name: "Solar System Voltage"
      unique_id: solar_system_voltage
    - name: "Solar System Current"
      unique_id: solar_system_current
    - name: "Solar System Active Power"
      unique_id: solar_system_active_power
    - name: "Solar System Reactive Power"
      unique_id: solar_system_reactive_power
    - name: "Solar System Power Factor"
      unique_id: solar_system_power_factor
    - name: "Solar System Frequency"
      unique_id: solar_system_frequency

I would think that these 2 resources are 2 different endpoints, as they are different URL’s (different MBus addresses), is that a correct assessment?

Building on that assumption, my automation is listed in the previous post, but I’ll paste the relevant parts of the “action” sequences here again:

        sequence:
          - service: homeassistant.update_entity
            target:
              entity_id:
                - sensor.solar_system_active_power
                - sensor.solar_system_current
                - sensor.solar_system_frequency
                - sensor.solar_system_power_factor
                - sensor.solar_system_reactive_power
                - sensor.solar_system_voltage

and:

        sequence:
          - service: homeassistant.update_entity
            target:
              entity_id:
                - sensor.solar_system_energy_production_total
                - sensor.solar_system_energy_consumption_total

As is evident, all (6) entities in the first sequence belong to a single defined endpoint (http://localhost:8085/mbus/get/ttyUSB0/9600/10010000FFFFFFFF), and all (2) of the entities in the second sequence belong to the other defined endpoint (http://localhost:8085/mbus/get/ttyUSB0/9600/1).

Most of the time, the corresponding endpoint is queried (once) almost immediately after calling the update_entity. But not always…

Sometimes there’s a (too) long delay before the commands are executed, and both differing endpoints are queried at the same time. Or, at least too close in time to be processed both, since these API endpoints take a few hundreds of milliseconds to return their data, it’s a serial bus after all.

Since the MBus can not be queried more than once simultaneously, one of the API calls will not return data at all, causing the corresponding entities to change their state to ‘unknown’.

That’s what I try to avoid.

Before I go and try to tinker with rest_commands and response_variables, would you have any comments on the above configuration?

12

They aren’t different endpoints, they are both talking to the same serial usb device. If the device can’t handle multiple things talking to it, then it will cause issues.

The only way to ensure 1 thing is talking to it at a time is for you to fully control it. I.e. use rest command and call the commands when you need them.

13

Ok, that does clear things up! If the endpoint, as the REST controller sees it, is the USB device, and not the MBus address, I see where things might get hung up so to speak…

Then I guess the quest continues (unless there’s a way to get the REST controller to consider the full URL/URI for deduplication :wink: )

I’ve quickly looked into the rest_command docs, but it is not immediately apparent how I update an entity (sensor) with data returned via such a rest-command, but I will give it a go…

I did already find references to some extra python script, but where exactly do I place it? Do I create a path named “.homeassistant/python_scripts” under the /config directory?

14

You make a template sensor using any trigger you want. In the action section, call the rest command and put the result into sensors. If you need help, I can help with this.

template:
- trigger:
  - platform: time_pattern
    seconds: '/30'
  action:
  - service: rest_command.get_energy
    response_variable: energy_response
  - delay: "00:00:01"
  - service: rest_command.get_info
    response_variable: info_response
  sensor:
  - name: Solar System Energy Production Total
    unique_id: solar_system_energy_production_total
    state: "{{ <code to get out info out of energy_response> }}"
    
  .... more sensors go here ...
15

Thanks for this setup!

I’ve added some testing sensors to the templates section, and the config seems to parse without obvious errors. Now I need to figure out how to get the data from the API into the “state:” property…

The MBus API returns XML, and in the RESTful integration, I defined these sensor values like this:

 value_template: "{{ value_json.MBusData.DataRecord[0].Value | int * 0.01 }}"

The RESTful integration automatically parses XML to JSON, so that works like a charm out of the box.

But the “template” sensors do not accept the “value_template” key, it requires a “state” key (which I have defined).

I know I should reference the name of the “response_variable” I declared, but the exact syntax to get the “Value” part out of it seems to elude me. I’ve tried several permutations, but my python knowledge isn’t sufficient I’m afraid.

Would you have a hint on how I can get the desired “Value” out of the response_variable[‘content’] return data?

Addendum: I’ve setup an “email” notify service, and mailed the output to myself. This is the contents of the “response_variable” (abbreviated to 1 DataRecord):

{'content': '<?xml version="1.0" encoding="ISO-8859-1"?>\n<MBusData>\n\n <SlaveInformation>\n <Id>1001</Id>\n <Manufacturer>___</Manufacturer>\n <Version>1</Version>\n <ProductName></ProductName>\n <Medium>Electricity</Medium>\n <AccessNumber>85</AccessNumber>\n <Status>00</Status>\n <Signature>0000</Signature>\n </SlaveInformation>\n\n <DataRecord id="0">\n <Function>Instantaneous value</Function>\n <StorageNumber>0</StorageNumber>\n <Unit>Energy (10 Wh)</Unit>\n <Value>64993</Value>\n <Timestamp>2024-03-27T19:22:02Z</Timestamp>\n </DataRecord>\n\n </MBusData>\n', 'status': 200}

So it seems the XML is not parsed to JSON in this case, but just put into the response_variable’s [‘content’] element as plain as it gets from the output…whoopee

Is there some function to explicitly parse XML directly?

16

Ok, looking at the code of the ‘rest_command’ component, it looks like there’s just no XML parsing supported.

The ‘rest_command’ response seems to only expect JSON, but tries “text” if JSON fails. This seems to be indicated in this part of the __init.py__:

169                    try:
170                        if response.content_type == "application/json":
171                            _content = await response.json()
172                        else:
173                            _content = await response.text()

So I guess that’s where this experiment stops for now, at least until someone adds XML to JSON conversion to the ‘rest_command’ component, when the Content-Type of the response is “application/xml”, just like the regular RESTful component does.

Would I need to create a new feature request to address this?

17

just do a regex search for each field you want in states

e.g.
for Timestamp

states: >
  {% set find = 'Timestamp' %}
  {{ response_variable['content'] | regex_findall('<{0}>(.*)</{0}>'.format(find)) | first }}
18

Well, that looked like it might work, and with some more frustrating time spent (mainly just fidgeting with the correct way to indent the YAML, and general troubleshooting), I finally got something to work with. My oh my, what a trip…

At long last, I have the config setup in a way that seems to be working!

So, if others are reading this to find a solution, here are some relevant details (I have separate configuration files for each integration, included from the main configuration.yaml):

rest_command: !include rest_commands.yaml
template: !include templates.yaml

Setup the rest_commands to query the relevant endpoints in rest_commands.yaml:

mbus_solar_energy:
  url: http://localhost:8085/mbus/get/ttyUSB0/9600/1
  method: POST
mbus_solar_power:
  url: http://localhost:8085/mbus/get/ttyUSB0/9600/10010000FFFFFFFF
  method: POST

Define the template sensors in templates.yaml (I only listed 2 sensors, one of each endpoint, but there are more):

- trigger:
  - platform: time_pattern
    seconds: /10
  action:
  - service: rest_command.mbus_solar_energy
    response_variable: solar_energy_response
  - variables:
      solar_energy_values: "{{ solar_energy_response['content']|regex_findall('<{0}>(.*)</{0}>'.format('Value')) }}"
  - delay: "00:00:01"
  - service: rest_command.mbus_solar_power
    response_variable: solar_power_response
  - variables:
      solar_power_values: "{{ solar_power_response['content']|regex_findall('<{0}>(.*)</{0}>'.format('Value')) }}"
  sensor:
  - name: "Solar System Energy Production Total"
    state: "{{ solar_energy_values[1] | int * 0.01 }}"
    device_class: "energy"
    state_class: "total_increasing"
    unit_of_measurement: "kWh"
    unique_id: solar_system_energy_production_total

  - name: "Solar System Voltage"
    state: "{{ solar_power_values[0] | int * 0.01 }}"
    state_class: "measurement"
    unit_of_measurement: "V"
    icon: "mdi:sine-wave"
    unique_id: solar_system_voltage

The response data is put into the relevant response_variables, and I choose to extract the “Values” into new variables in the action section, to reference their elements to set the states…

Many thanks to @petro for all the hints and guidance.