"zha-toolkit" - a big set of Zigbee commands on top of ZHA/zigpy

zha_toolkit is a custom component providing useful features for ZHA that helps
fix, configure or analyze your zigbee devices.
zha_toolkit also helps with automations (e.g., store a zigbee attribute to a state).

About 2% of the ZHA installs use zha_toolkit.

In order, my top favorite features are:

  • Write values that were retrieved from a Zigbee read directly to a state;
  • Read Zigbee values from ZHA’s internal cache;
  • Go beyond the bindings offered by HA’s UI;
  • Finetune and check reporting configurations;
  • Perform a scan of the device to get a more detailed picture of what it offers (when the device allows it);
  • Write any value to any HA state (not zha-related, but a service offered by the toolkit).

A presumed popular feature is that it provides “online” backups of the Coordinator hardware (which has been validated for the ZNP radios (TI based key) but awaiting confirmation for bellows). (Now ZHA handles backups by itself).

:warning: HA2023.7 with zigpy 0.56.0 has breaking changes, you need to update to zha-toolkit v0.9.4

zha-toolkit is more technical than most integrations.

For example, you can check the bindings a device has and remove all of these binding (when the binding table is full for instance), or change reporting configurations - even for sleepy devices and different from the default ones (for example, a thermometer).

zha_toolkit is a fork of zha_custom and has facilitated getting data back from the events, easier access to the commands, documentation and new commands, read/write/read functionnality, backup, getting the device list, scanning the capabilities of a device, update/create Home Assistant states, writing data to CSV, read attributes from cache, … .

You can add this using HACS - it’s in the default repository list.

After adding the componant using HACS, you still need to update your configuration.yaml so that it is loaded after restart:

zha_toolkit:

This is an example of the options available for writing an attribute (there is zha_toolkit.attr_write as well for this):

service: zha_toolkit.attr_write
data:
  ieee: 5c:02:72:ff:fe:92:c2:5d
  # The endpoint is optional - when missing tries to find endpoint matching the cluster
  endpoint: 11
  cluster: 0x1706
  attribute: 0x0000
  attr_type: 0x41
  # Example of octet strings (the length is added because of attr_type)
  attr_val:  [41,33,8,45,52,46,50,191,55,57,136,60,100,102,63]
  # Optional manufacturer Id
  manf: 0x1021
  # Optional, state to write the read value to
  state_id: sensor.test
  # Optional, state attribute to write the value to, when missing: writes state itself
  state_attr: option
  # Optional, when true, allows creating the state (if not the state must exist)
  allow_create: True
  # The manufacturer should be set only for manufacturer attributes
  manf: 0x1202
  # You can set the next events to use as a trigger.
  # The event data has the result of the command (currently attr_read, attr_write)
  event_success: my_write_success_trigger_event
  event_fail: my_write_fail_trigger_event
  event_done: my_write_done_trigger_event
  # Settings for attr_write
  # Read attribute before writing it (defaults to True)
  read_before_write: True
  # Read attribute after writing it (defaults to True)
  read_after_write: True
  # Write attribute when the read value matches (defaults to False)
  write_if_equal: False

After updating zha-toolkit, most of the time you do NOT need to restart HA.

If you want to “hack” a few functions, just change the code on your platform and call the service - your code will be used - or you’ll see the error which you can fix and retry the service again.

As Developer Tools>Services is searchable, just typing part of the command will quickly lead to the right one and the leading icon (emojii) helps identify a zha-toolkit service.

Note: this head post has been heavily edited since its initial version.

8 Likes

I also added report configuration (not that I do this on my ‘dev’ branch). Works for a temperature sensor:

Configure Temperature reporting on a SonOff SNZB-02 (eWeLink/TH01).
Note that you (may) need to press the button on the thermometer just after
requesting the command.
I got timeouts for these commands, but at least one of them passed and
the temperature reporting was in practice every 20s (probably the minimum
of the device itself) for changes of minimum 0.10 degC.

service: zha_toolkit.execute
data:
  ieee: 00:12:4b:00:23:b3:da:a5
  command: conf_report
  command_data: 1,0x0402,0x0000,5,300,10

This can’t currently be added using HACS, but you can add it to your custom_components repository. Feel free to fork and propose the configuration for that.

2 Likes

I just added the possibility to backup the ZNP network data to a json file from within HA. Often used to migrate to a new network coordinator key.

Backup ZNP network data

Use to transfer to another ZNP key later, backup or simply get network key and other info.

The output is written to the customisation directory as ‘nwk_backup.json’

service: zha_toolkit.execute
data:
  command: znp_backup
2 Likes

This is awesome! Having a backup service in HA itself is a huge bonus. I tested it out and it generated the backup file without any issues (that I know of)

@Hedda putting this on your radar since I know you had a thread open about something like this. With this service, it opens up the possibility of doing nightly backups and such

1 Like

That is great! Though really wish that automatic nightly backups was included by default in ZHA UI, see:

https://community.home-assistant.io/t/zha-integration-to-do-nightly-backup-of-both-zigbee-coordinator-adapter-dongle-stick-and-zigbee-database/357558/

btw, know that puddly who develops zigpy-znp is planing to implement unified backup into zigpy-cli, see:

https://github.com/zigpy/zigpy-cli/pull/2

That will mean that once implemented you can use same backup command to backup all radio brands.

1 Like

FYI, mentioned this tip to zigpy/ZHA developers here → https://github.com/zigpy/zigpy/discussions/875

2 Likes

Happy that this component got noticed :wink: .

Ideally HA proposes at least the services to do this kind of stuff, but automation is typically left over to blueprint proposals.
I agree that some thinks would better be options in the integrations (e.g., an option in the ZHA integration to backup).
With regards to backups, I think the most important thing is that they are “externalized” and I have not looked into that yet - some functions require a bit too much time to set up.

But that is kind of a “side-track” to this “zha_toolkit” component. It’s mainly a way to expose ZHA functionnality as services. It was also (originally) implemented so that HA restarts are not required when changing the code: the imports are reloaded from the component on each call. So that made it easier to tune the code.

When the new backup function is available as an API it should be easy to add to zha_toolkit.
I’ve implemented backup for the ZNP as that is what I have, but it’s possible to add backup services for the other HW types and possibly even unify that in a single service (by detecting the type actually in use).

So I should make it an “official” HACS compatible component ?!

1 Like

@hedda Just to make it clear, this allows you to setup an automation to do the nightly backups. Just call the service with ‘znp_backup’ command to do the backup fo the ZNP.

2 Likes

Yes I understand, however, I would love for such a Zigbee network backup feature to be implemented into ZHA integration component by default and preferably for the feature to also be exposed in ZHA’s UI.

Having features included by default as standard + exposed in ZHA UI makes it much more user-friendly.

Know it is an ongoing goal for Home Assistant founders to make more essential features easier to use.

FYI, built-in automatic nightly backups have been standard in Zigbee2MQTT for a long time so think many users assume ZHA has it too, especially migrating from any other implementations that have it.

PS: Z2M always automatically stores Zigbee network backup to “data/coordinator_backup.json”.

https://www.zigbee2mqtt.io/guide/faq/#how-do-i-move-my-zigbee2mqtt-instance-to-a-different-environment

https://www.zigbee2mqtt.io/guide/faq/#how-do-i-migrate-from-one-adapter-to-another

1 Like

I agree that user friendlyness is tha way to go.

I discovered the original (“unfinished”/“unmaintained”) component and decided to dig it up to implement the custom attribute writes, “custom” cluster bindings not proposed in the UI and custom report commands after some lengthly discussions that resulted in a status quo of how ZHA currently is.
Some of this stuff may take a while until they are in ZHA, so it’s really handy to be able to add these things with developer skills and call them from the UI and integrations in a production setup.

In short: it’s hard to change HA, its easier to add/update a custom component which has less requirements.

I could also add the restore feature as a service, but I will not be testing it. It might make the migration easier for some users though as it probably could all be done from within the UI by calling the services.

2 Likes

Yes please consider doing so as will probably attract additional people who figure out ideas for many new functions, and if proven popular/useful to more might later be incorporated into HA core by default.

I’m willing to test out the restore feature with a small sunset of devices. I wanted to migrate over to z2m anyway so I don’t mind if I have to repair a few devices.

If both features are available I’m sure more people will be willing to use this and it should definitely be a havs official add on. This is all great work and may expedite those feature getting into zha.

As a side note, it might be a good idea to have a disclaimer saying that coordinator backups created with zigpy-znp 0.6.0+ are not compatible with older versions (not sure if older versions can be restored with versions 0.6 and higher)

1 Like

@abhi08638 I added the restore functionnality. I did some basic debugging to ensure the major steps, but I did not actually perform the restore or shutdown.

I did not have to restart HA after updating the component on my system. The component reloads its module and init.py files when called.

For safety, I backup the key that is restored to before the actual restore.

There is no specific error checking, I rely on exceptions being thrown, so the ‘Call Service’ button is an indicator: when it turns Red, something went wrong, when Green it’s likely ok.
You should enable debug, and check the home_assistant.log .

2 Likes

zha_toolkit can now send cluster specific commands.

1 Like

@naijaping, the procedure is explained in the readme, which you probably discovered, but it still has to be tested in practice.
The backup works, the restore itself needs to be tested (I validated the code, except for the restore itself).

@le_top , I have followed the read me but I don’t know why the local folder in my custom_components/zha_custom is empty after sending the call service.

Maybe it’s a matter of access rights. Can you check those?
Did you enable debug messages for zha_custom (you can do that by calling a service, see the README)? Anything useful in the log ?

Given your observation, I removed the directory from git versioning (it’s created by the script) and changed the hacs configuration so that it’s persistent.

FYI, in my log I get:

2022-01-06 13:23:52 INFO (MainThread) [custom_components.zha_toolit] Running custom service: <ServiceCall zha_toolkit.execute (c:d1c5de23a359c0b22309e8d5032cad8f): command=znp_backup>
2022-01-06 13:23:52 DEBUG (MainThread) [custom_components.zha_toolkit] module is <module 'custom_components.zha_custom' from '/config/custom_components/zha_toolkit/__init__.py'>
2022-01-06 13:23:52 WARNING (MainThread) [zigpy_znp.znp.security] Skipping hashed link key a1:0e:ba:de:c6:9b:35:12:db:f5:21:84:ec:c0:68:49 (tx: 1122, rx: 0) for unknown device 60:a4:23:ff:fe:e6:70:74

@le_top , thanks for your quick response , I will enable debug and post the rest here.

Many thanks

@le_top ,

here is my log:

Logger: zigpy_znp.api
Source: /usr/local/lib/python3.9/site-packages/zigpy_znp/api.py:700
First occurred: 20:17:28 (3 occurrences)
Last logged: 20:24:30

Received an unhandled command: ZDO.SrcRtgInd.Callback(DstAddr=0x3696, Relays=[])
Received an unhandled command: AF.IncomingMsg.Callback(GroupId=0x0000, ClusterId=0, SrcAddr=0x3696, SrcEndpoint=1, DstEndpoint=1, WasBroadcast=<Bool.false: 0>, LQI=63, SecurityUse=<Bool.false: 0>, TimeStamp=12719110, TSN=0, Data=b'\x18\x6E\x0A\x01\x00\x20\x56\xE2\xFF\x20\x13', MacSrcAddr=0x3696, MsgResultRadius=29)
Received an unhandled command: AF.DataConfirm.Callback(Status=<Status.MAC_TRANSACTION_EXPIRED: 240>, Endpoint=1, TSN=29)

After reboot I also get the following in the log:

Logger: homeassistant.components.zha.core.channels.base
Source: components/zha/core/channels/base.py:428
Integration: Zigbee Home Automation (documentation, issues)
First occurred: 21:01:25 (3 occurrences)
Last logged: 21:02:19

[0x8361:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>')]
[0x8361:1:0x0008]: async_initialize: all attempts have failed: [DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>')]
[0x8361:1:0x0300]: async_initialize: all attempts have failed: [DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>')]

Ok, these are related to the core/zha but not with the backup.

When you have logging (debug) enabled for zha_toolkit, there should be messages indicating that you called the service.
Example:

2022-01-06 13:23:52 INFO (MainThread) [custom_components.zha_toolkit] Running custom service: <ServiceCall zha_toolkit.execute (c:d1c5de23a359c0b22309e8d5032cad8f): command=znp_backup>

That indicates that znp_backup was called.

You can enable logging by using this yaml from the Developer > Service interface:`

service: logger.set_level
data: 
    custom_components.zha_toolkit: debug

That enables debug messages for the zha_toolkit component.
Once enabled, you can call znp_backup and something should appear in the log.