"zha-toolkit" - a big set of Zigbee commands on top of ZHA/zigpy

Hi @Hedda

zha-toolkit did not wait for such a feature for providing responses which can be fouind in the event data. There are some examples that demonstrate how to use that.

However, it would certainly beneficial to provide the data directly to the automation rather than having to retrieve it from the event (which is kind of complex to capture in an automation).
However, I did not find technical documentation on how to generate those responses in a component’s python by following the links in the release notes.

Home Assistant Developers Blog only mentions “service return values” which might be the same thing?

https://developers.home-assistant.io/blog/2023/06/14/service-calls/

https://github.com/home-assistant/architecture/discussions/777#discussioncomment-6127898

I did find this example on “Response Data” for services in the Home Assistant documentation:

https://www.home-assistant.io/docs/scripts/service-calls#use-templates-to-handle-response-data

and this about “service response data” in the developer’s documentation:

https://developers.home-assistant.io/docs/dev_101_services/#response-data

Release notes also mention that integration developers first need to enable it per service. So you can not get responses from just any service, instead, it has been enabled in the integration for each specific service that you want to be able to give a response.

https://developers.home-assistant.io/docs/dev_101_services/#supporting-response-data

As I understand, the release notes implies that other than for scripts it has so far only been enabled for calendar.list_events and conversation.process services in the Calander integration as an example so guess that integration developers would need to use that as a reference?

Perhaps @allenporter has more info or links on implementing responses for services in integrations?

You found everything… The last link is the developer docs for implementing this.

@Hedda Thanks for pulling this together.
@allenporter I agree that Integration Services | Home Assistant Developer Docs holds the relevant information.

The documentation suggests:

      hass.services.async_register(
          DOMAIN,
          SEARCH_ITEMS_SERVICE_NAME,
          search_items,
          schema=SEARCH_ITEMS_SCHEMA,
          SupportsResponse.ONLY,
      )

but then I get SyntaxError: positional argument follows keyword argument - I will find the correct way to do it (named argumetn or positional argument).

Submitted docs: fix Supporting Response Data example by pdecat · Pull Request #1853 · home-assistant/developers.home-assistant · GitHub to fix the documentation example.

zha-toolkit is now updated to provide response data and I updated one of the scripts to use it:

The response data is also provided by when calling the service interactively which is handier than having to check the even_data.

3 Likes

Hi

I’m trying to use Toolkit, which has been installed/activated correctly. I also activated the custom component under “log” in my configuration. I understand that the feedback from running the services will be listed in the Home Assistant core log file - correct? The problem is I see nothing in the log file.

Is there somebody who can point me in the direction of a solution?! :slight_smile:

image

One way to verify that zha-toolkit is correctly set up is to verify that they appear in “Developrer Tools > Services” (example in French):
image

Using that same location to run services, you can dynamically change the log level (no need to add them to you static configuration - the configuration is kept until reboot or another change):

service: logger.set_level
data:
  custom_components.zha_toolkit: debug
  zigpy.zcl: debug

It is true that the traces should occur in the log (/config/home-assistant.log).

With Home Assistant >=2023.7 and zha-toolkit>=v0.9.7, you can also see the response when using th e “Developer Tools>Services” without the need to activate logging or listen to events:

And when using a zha-toolkit service, you can get the result in a variable by adding “response_variable: …” to the service arguments.

Do you see the zha toolkit services? Do you have zha insdtalled as well (zha-toolkit ‘extends’ zha)? zha has services as well, so you can check that you have access to those:

image

@le_top have you looked at zha-toolkit using “Manufacturer OTA Firmware Update Support” (also referred to as “Generic OTA providers”) that was added with zigpy 0.56.3 release in HA 2023.8.1?

https://github.com/zigpy/zigpy/pull/1165

This allows for new OTA providers to be added to zigpy without extra code, which in turn also allows for users and manufacturers to distribute test feed URLs to customers without deploying them globally.

https://github.com/zigpy/zigpy/wiki/OTA-Information-for-Manufacturers

I believe that this could for example enable users to easier add to for example grab Zigbee OTA firmware files from Koenkk’s Zigbee OTA repo → https://github.com/Koenkk/zigbee-OTA/blob/ef0e28e5bdd0d5c6dca0bed4e5230b86bdea3e2c/index.json (as requested here → https://github.com/zigpy/zigpy/issues/535)

PS: puddly also added a new feature to the “zigpy-cli” tool that manufacturers can use to produce/generate a skeleton index from a given set of OTA files, and that way makes sure manufacturers OTA files are correctly-structured so that they will be compatible with this new JSON format for zigpy OTA providers out-of-the-box. Check out the README.md for the “zigpy-cli” (zigpy CLI tool) → https://github.com/zigpy/zigpy-cli/blob/dev/README.md#generate-ota-index-files

Thank you for your detailed response.

It looks like the Toolkit is installed correctly. I will give it a go again, using your inputs.

Thanks. - Thomas

zha_toolkit.scan_device is not reporting attributes for manufacturer cluster. In the scan file the cluster is there but for attribute I get only “attributes”: {},
Attributes are there as I can read there value in ZHA but none are listed in the scan. Is there something I need to do to improve scanning

IHMO the device is not reporting the attributes for the manufacturer clusters.
I have the result of a zha_toolkit.scan_device for Danfoss/eTRV0100 that also has the the details for the manufacturer attributes in the alongside the normal attributes.

Example:

            "0xfffd": {
              "attribute_id": "0xfffd",
              "attribute_name": "cluster_revision",
              "value_type": [
                "0x21",
                "uint16_t",
                "Analog"
              ],
              "access": "REPORT|READ",
              "access_acl": 5,
              "attribute_value": 1
            },
            "0x4000": {
              "attribute_id": "0x4000",
              "attribute_name": "sw_error_code",
              "value_type": [
                "0x19",
                "bitmap16",
                "Discrete"
              ],
              "access": "REPORT|WRITE|READ",
              "access_acl": 7,
              "manf_id": 4678,
              "attribute_value": 0
            },

I know that others don’t report them and I also know that at least one major vendor of zigbee components does not natively propose features in at least one of its zstack frameworks allowing to easily add manufacturer attributes so that they are properly reported: it’s up to the developer to re-implement all that… So given the amount of effort that represents, manufacturer attributes may not reported.

Ok so the only way to find out the attributes of the manufacturer cluster is to sniff the zigbee data ?

There is another method, slower requiring less effort and maybe faster if you consider the time you have to spend to decipher the communications:
To the clusters that you know exist and those that you think that exist read all attributes with the manufacturer id set. Readable attributes should reveal their type, write-only attributes may reveal they exist (not sure what error is reported in that case).

You can start from the script that reads several attributes in the basic cluster available as an example: https://github.com/mdeweerd/zha-toolkit/blob/c10f0995eb763accbedd9416af34fdf519f37e05/examples/script_read_basic_cluster.yaml .

You can adapt it to read another cluster (cluster<>0), more attributes (0 to 0xFFFF for the ‘full’ range - not sure that the last values are valid attirbutes), and add the manufacturer_id as a parameter to the reads.
You can stop the script when you think you found them all - bit some manufacturers will start their attributes with a high id such as 0xFC00 .
If you can read an attribute every second, then you can read all of them in about 18 hours.
Some devices do not respond when the attribute does not exist, so then it takes longer, and with sleepy devices it takes even longer.

However, patience will take you there and you can also accellerate if you can make smart guesses for the ranges.

The results are written to a CSV file - so you can discover what could be read and what not.

I’ve test ed as below but attribute 3 or any other I’ve tested are not there
warnings:

  • “Result: {3: 0} - Attribute 3 not in read ({}, {3: 0})”
    success: false
service: zha_toolkit.attr_read
data:
  ieee: 50:0b:91:40:00:02:26:6d
  manf: 17672
  endpoint: 1
  cluster: 65281
  attribute: 3
  event_success: my_read_success_trigger_event
  event_fail: my_read_fail_trigger_event
  event_done: my_read_done_trigger_event
  csvout: ../www/test.csv

do I miss something

When the read returns something, you would see this in the response:

result_read:
  - "3": 1
  - {}

while you have:

result_read:
  - {}
  - "3": 0

Which means that there was no value for the attribute, however there also was no error code (0 means everything is ok). “No value” might mean “empty value”, but I’ld prefer to check that (see below). As you can sniff as far as I understand it, this is an interesting case.

When I read an attribute that does not exist, I get this:

result_read:
  - {}
  - "30": 134
warnings:
  - >-
    Result: {30: <Status.UNSUPPORTED_ATTRIBUTE: 134>} - Attribute 30 not in read
    ({}, {30: <Status.UNSUPPORTED_ATTRIBUTE: 134>})

Which shows that 134 is reported as an unsupported attribute.

Do you have “attr_type” in your response? If you do, share the type. Maybe it’s a field that can be empty (zero length) and then I need to improve handling of this kind of fields.
Otherwise I would conclude that the device is not responding correctly and that the field is a write only field.

On my end, I can not find a device with an attribute that is WRITE only, so I can’t compare.

I have used Home Assistant for some time, mainly for controlling some lights. Now I intend to also control a few electric heaters. So I bought an Namrod Zigbee plug with thermostat, but this far I am only able to use it as an on/off plug. In the description that was included with the plug it is referred to a proprietary attribute that can be sett to activate it as a thermostat device , so I searched around and discovered zha-toolkit. I have installed it using hacs. However when I start to try to use it I get the same error, no mater what I do. scan_device, attr_read or attr_write all give med the same error: “‘ZHAData’ object is not subscriptable”

Probably it is just me doing something wrong,but I can’t figure out what.

A longer error message from the log when I was testing Scan Device is shown below:
(I have also tried to scan other devices, but get the same error.)

----
TypeError: 'ZHAData' object is not subscriptable
2023-10-05 22:04:33.232 INFO (MainThread) [custom_components.zha_toolkit] Running ZHA Toolkit service: <ServiceCall zha_toolkit.scan_device (c:01HC0Q28ADBSC4JHGZGRG5TA37): ieee=sensor.namron_as_4512749_n_hvac_action>
2023-10-05 22:04:33.238 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Unexpected error for call_service at pos 1: 'ZHAData' object is not subscriptable
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 468, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 704, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 666, in _async_run_long_action
    return long_task.result()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2012, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2049, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/zha_toolkit/__init__.py", line 672, in toolkit_service
    zha_gw = hass_ref.data["zha"]["zha_gateway"]

----

The ZHA version in 2023.10.0 introduced an incompatibility. Hopefully this is fixed now in v1.1.2 (beta). I have not updated yet, and I had to modify a user’s suggestion to ensure backward compatibility.

I’ll wait for the update to be able to debug zha-toolkit.scan device which do not see manufacturer cluster attributes. Same for the zha_toolkit.attr_read.

The update was released - a user confirmed it worked for him.
It does not change the behavior of the services, but adjusts to a breaking change in ZHA/2023.10.