"zha-toolkit" - a big set of Zigbee commands on top of ZHA/zigpy

Hedda · January 5, 2022, 10:29am

Yes please consider doing so as will probably attract additional people who figure out ideas for many new functions, and if proven popular/useful to more might later be incorporated into HA core by default.

abhi08638 · January 5, 2022, 12:33pm

I’m willing to test out the restore feature with a small sunset of devices. I wanted to migrate over to z2m anyway so I don’t mind if I have to repair a few devices.

If both features are available I’m sure more people will be willing to use this and it should definitely be a havs official add on. This is all great work and may expedite those feature getting into zha.

As a side note, it might be a good idea to have a disclaimer saying that coordinator backups created with zigpy-znp 0.6.0+ are not compatible with older versions (not sure if older versions can be restored with versions 0.6 and higher)

le_top · January 5, 2022, 1:32pm

@abhi08638 I added the restore functionnality. I did some basic debugging to ensure the major steps, but I did not actually perform the restore or shutdown.

I did not have to restart HA after updating the component on my system. The component reloads its module and init.py files when called.

For safety, I backup the key that is restored to before the actual restore.

There is no specific error checking, I rely on exceptions being thrown, so the ‘Call Service’ button is an indicator: when it turns Red, something went wrong, when Green it’s likely ok.
You should enable debug, and check the home_assistant.log .

le_top · January 5, 2022, 9:26pm

zha_toolkit can now send cluster specific commands.

le_top · January 6, 2022, 8:59am

@naijaping, the procedure is explained in the readme, which you probably discovered, but it still has to be tested in practice.
The backup works, the restore itself needs to be tested (I validated the code, except for the restore itself).

naijaping · January 6, 2022, 11:52am

@le_top , I have followed the read me but I don’t know why the local folder in my custom_components/zha_custom is empty after sending the call service.

le_top · January 6, 2022, 12:25pm

Maybe it’s a matter of access rights. Can you check those?
Did you enable debug messages for zha_custom (you can do that by calling a service, see the README)? Anything useful in the log ?

Given your observation, I removed the directory from git versioning (it’s created by the script) and changed the hacs configuration so that it’s persistent.

FYI, in my log I get:

2022-01-06 13:23:52 INFO (MainThread) [custom_components.zha_toolit] Running custom service: <ServiceCall zha_toolkit.execute (c:d1c5de23a359c0b22309e8d5032cad8f): command=znp_backup>
2022-01-06 13:23:52 DEBUG (MainThread) [custom_components.zha_toolkit] module is <module 'custom_components.zha_custom' from '/config/custom_components/zha_toolkit/__init__.py'>
2022-01-06 13:23:52 WARNING (MainThread) [zigpy_znp.znp.security] Skipping hashed link key a1:0e:ba:de:c6:9b:35:12:db:f5:21:84:ec:c0:68:49 (tx: 1122, rx: 0) for unknown device 60:a4:23:ff:fe:e6:70:74

naijaping · January 6, 2022, 3:57pm

@le_top , thanks for your quick response , I will enable debug and post the rest here.

Many thanks

naijaping · January 6, 2022, 8:33pm

@le_top ,

here is my log:

Logger: zigpy_znp.api
Source: /usr/local/lib/python3.9/site-packages/zigpy_znp/api.py:700
First occurred: 20:17:28 (3 occurrences)
Last logged: 20:24:30

Received an unhandled command: ZDO.SrcRtgInd.Callback(DstAddr=0x3696, Relays=[])
Received an unhandled command: AF.IncomingMsg.Callback(GroupId=0x0000, ClusterId=0, SrcAddr=0x3696, SrcEndpoint=1, DstEndpoint=1, WasBroadcast=<Bool.false: 0>, LQI=63, SecurityUse=<Bool.false: 0>, TimeStamp=12719110, TSN=0, Data=b'\x18\x6E\x0A\x01\x00\x20\x56\xE2\xFF\x20\x13', MacSrcAddr=0x3696, MsgResultRadius=29)
Received an unhandled command: AF.DataConfirm.Callback(Status=<Status.MAC_TRANSACTION_EXPIRED: 240>, Endpoint=1, TSN=29)

After reboot I also get the following in the log:

Logger: homeassistant.components.zha.core.channels.base
Source: components/zha/core/channels/base.py:428
Integration: Zigbee Home Automation (documentation, issues)
First occurred: 21:01:25 (3 occurrences)
Last logged: 21:02:19

[0x8361:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>')]
[0x8361:1:0x0008]: async_initialize: all attempts have failed: [DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>')]
[0x8361:1:0x0300]: async_initialize: all attempts have failed: [DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>')]

le_top · January 6, 2022, 10:24pm

Ok, these are related to the core/zha but not with the backup.

When you have logging (debug) enabled for zha_toolkit, there should be messages indicating that you called the service.
Example:

2022-01-06 13:23:52 INFO (MainThread) [custom_components.zha_toolkit] Running custom service: <ServiceCall zha_toolkit.execute (c:d1c5de23a359c0b22309e8d5032cad8f): command=znp_backup>

That indicates that znp_backup was called.

You can enable logging by using this yaml from the Developer > Service interface:`

service: logger.set_level
data: 
    custom_components.zha_toolkit: debug

That enables debug messages for the zha_toolkit component.
Once enabled, you can call znp_backup and something should appear in the log.

naijaping · January 6, 2022, 11:48pm

@le_top , Thanks for the tips, unfortunately I cant see any ServiceCall zha_custom.execute in the log may be I am doing the debug bit wrong, The good news is that after changing the developer tools to yaml mode, i can now see the back up at custom_components/zha_custom/local/nwk_backup.json.

Something that is not clear now is that how do I restore this backup to a new coordinator, do I just pull out my CC2531 and insert my Sonoff CC2652P and call Restore? all I can see in the read me is to remove the key and insert the new key which is not clear.

Many thanks for your help.

le_top · January 7, 2022, 8:46am

Well, that’s good progress.

That you do not see anything in the logs is likely related to the log configuration in HA itself.

By key, I mean “USB Key” as the TI-ZNP hardware is usually a USB device.

So basically, once you have the backup file, the restore will use the file from the same location.
So by physically replacing the TI-ZNP key with the new destination key, you can restore to the destination key.

For safety, I recommend to restart HA after putting in the new key. And restart it again after the restore.

When you restore, you’ll see that another backup is made (of the key that is being restored to).

Note that you’re the first doing the actual restore this way. If the restore “fails” you should be able to use the previous key.

It’s also best that your backup is not too old - the restore method allows the definition of an offset for the TX counter which is probably used to avoid replays - end devices may not accept packets with a counter that is too small. This offset is 2500 by default, but if you have many devices that amount of packets can easily be reached in a day (it’s “only” 100 frames per hour, or 1.5 each minute).

The method indicated above to set the log level is probably lost after restart, so you need to redo it after restart.

naijaping · January 7, 2022, 10:19am

@le_top ,Thanks a lot for your help, I will test it out over the weekend, I can see “frame_counter”: 4008842 in my backup may be I will go with that value.

Best regards .

fantangelo · January 7, 2022, 3:22pm

Hi,
Thanks for your work on this. I have installed this and enabled the logger debug. When I try to call

service: zha_custom.execute
data:
  command: znp_backup

I get the following error:

Logger: homeassistant.helpers.script.websocket_api_script
Source: custom_components/zha_custom/znp.py:38
Integration: ZHA Custom Service ([documentation](https://github.com/mdeweerd/zha_custom), [issues](https://github.com/mdeweerd/zha_custom/issues))
First occurred: 10:14:26 AM (3 occurrences)
Last logged: 10:20:12 AM

websocket_api script: Error executing script. Unexpected error for call_service at pos 1: 'ControllerApplication' object has no attribute '_znp'

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 381, in _async_step await getattr(self, handler)() File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 584, in _async_call_service_step await service_task File "/usr/src/homeassistant/homeassistant/core.py", line 1495, in async_call task.result() File "/usr/src/homeassistant/homeassistant/core.py", line 1530, in _execute_service await handler.job.target(service_call) File "/config/custom_components/zha_custom/__init__.py", line 61, in custom_service await handler( File "/config/custom_components/zha_custom/znp.py", line 38, in znp_backup backup_obj = await backup_network(app._znp) AttributeError: 'ControllerApplication' object has no attribute '_znp'

I am running HASS OS in a Proxmox VM and have the latest versions of Core, Supervisor and Host.
thanks

le_top · January 7, 2022, 4:07pm

The ‘_znp’ field does not exist, suggesting that your zigbee coordinator is not using zigpy-znp .

fantangelo · January 7, 2022, 4:29pm

Yup, just checked and my HUSBZB-1 is not zigpy-znp. thanks

le_top · January 7, 2022, 4:51pm

There is a backup script for this, it should be doable to integrate this, but maybe somebody with a device could do that: bellows/bellows/cli/backup.py at acf47f6939c28b20b22d5715646e6f1a0bd70064 · zigpy/bellows · GitHub .

abhi08638 · January 7, 2022, 8:44pm

The backup calls works fine for me, I just tried the restore command after upgrading the firmware on my coordinator (it wiped the data). I think the restore call only looks for “nwk_backup.json”, so if you use a custom file name like “nwk_backup_20220107.json”, it throws an error. I renamed the file and did another restore and I got the following in my logs:

2022-01-07 15:33:31 INFO (MainThread) [custom_components.zha_custom] Running custom service: <ServiceCall zha_custom.execute (c:9a7d3a5f91bc24e161808b9d82257dea): command=znp_restore>
2022-01-07 15:33:31 DEBUG (MainThread) [custom_components.zha_custom] module is <module ‘custom_components.zha_custom’ from ‘/config/custom_components/zha_custom/init.py’>
2022-01-07 15:33:42 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall zha_custom.execute (c:9a7d3a5f91bc24e161808b9d82257dea): command=znp_restore>
Traceback (most recent call last):
File “/usr/src/homeassistant/homeassistant/core.py”, line 1511, in catch_exceptions
await coro_or_task
File “/usr/src/homeassistant/homeassistant/core.py”, line 1530, in _execute_service
await handler.job.target(service_call)
File “/config/custom_components/zha_custom/init.py”, line 61, in custom_service
await handler(
File “/config/custom_components/zha_custom/znp.py”, line 109, in znp_restore
await app._znp.pre_shutdown()
AttributeError: ‘ZNP’ object has no attribute ‘pre_shutdown’

DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’)]
2022-01-07 15:41:15 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x2DC7:1:0x0300]: async_initialize: all attempts have failed: [TimeoutError(), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’)]
2022-01-07 15:41:15 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x2DC7:1:0x0006]: async_initialize: all attempts have failed: [TimeoutError(), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’)]
2022-01-07 15:41:22 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x1B42:1:0x0300]: async_initialize: all attempts have failed: [DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’)]
2022-01-07 15:41:22 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x1B42:1:0x0008]: async_initialize: all attempts have failed: [DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’)]
2022-01-07 15:41:22 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x1B42:1:0x0006]: async_initialize: all attempts have failed: [DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’)]
2022-01-07 15:41:56 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x2DC7:1:0x0702]: async_initialize: all attempts have failed: [DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’)]
2022-01-07 15:42:02 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x1B42:1:0x0702]: async_initialize: all attempts have failed: [DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’)]

le_top · January 7, 2022, 9:19pm

Yes, the file has to be called exactly ‘nwk_backup.json’ for the restore. Maybe I’ll add the possibility to set the exact file name later (I now know how to add more parameters than just “command_data”).
The restore has apparently been executed (but I forgot to ‘await’ the asynchronous command).
The last step ‘pre_shutdown’ was not found. That is strange because its in the original cli code and it exists in the common Controller implementation .

I added ‘await’ now, maybe that helps.

The errors may berelated to the restore and a need to restart.

For all commands: I modified the code so that the ‘ieee’ parameters can also receive the short network address or the entity name.

abhi08638 · January 7, 2022, 9:42pm

Once I get some time I’ll re-pair some devices, create a new backup, reset the coordinator and then try a restore and see if it is successful

Is the nvram_reset command support in your build?