"zha-toolkit" - a big set of Zigbee commands on top of ZHA/zigpy

abhi08638 · January 7, 2022, 11:32pm

We’re in business! I didnt get to try out the nvram reset command just because the restore worked on the first try
This was my environemnt:
Backup → lower firmware version (backup contained 4 sengled bulbs, 1 Ikea signal repeater, and 1 Smart outlet)
ZHA → Deleted both the integration and zigbee.db, rebooted and installed a fresh instance of ZHA with no devices
Restore → Issued restore command from my backup and restarted (the below is roughly what I saw in the log)

“Write done, call pre_shutdown(). Restart the device/HA after this.”
“Write done, call pre_shutdown(). Restart the device/HA after this.”
2022-01-07 15:33:31 DEBUG (MainThread) [custom_components.zha_custom] module is <module ‘custom_components.zha_custom’ from ‘/config/custom_components/zha_custom/init .py’>
2022-01-07 15:33:42 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall zha_custom.execute (c:9a7d3a5f91bc24e161808b9d82257dea): command=znp_restore>
Traceback (most recent call last):
File “/usr/src/homeassistant/homeassistant/core.py”, line 1511, in catch_exceptions
await coro_or_task
File “/usr/src/homeassistant/homeassistant/core.py”, line 1530, in _execute_service
await handler.job.target(service_call)
File “/config/custom_components/zha_custom/init .py”, line 61, in custom_service
await handler(
File “/config/custom_components/zha_custom/znp.py”, line 109, in znp_restore
await app._znp.pre_shutdown()
AttributeError: ‘ZNP’ object has no attribute ‘pre_shutdown’

On Restart → I saw one bulb join and then after a few minutes a total of 3 bulbs and 1 signal repeater joined. 1 bulb and 1 smart outlet did not join automatically but I just power cycled those devices for a few seconds and they were added back.

So far it seems to be working from what I have tested!

le_top · January 8, 2022, 10:15am

Great news.

Regarding the procedure (to know if/how the README.md should be updated):

is it essential to add and remove the integration and the database?

[Edit: I amended them any way and added a link to your post]

Regarding the delay of “rejoining” the network: the zigbee specifications requires devices to “back-off” when they have difficulties to communicate in order not to flood the network. I believe the request is to wait 15 minutes to try again, but not all devices do this the same way.

The ‘pre_shutdown()’ call is apparently a precaution to shutdown the database correctly in this case. I added an extra debug message to show all methods in the ZNP instance and in my case it lists ‘pre_shutdown’ . (I am not trying to call that method as I don’t want to mess too much with my home network).

Anyway, your test confirms that the user can avoid setting up a command line interface (cli) on another computer to restore his ZNP network which will make the operation easier to perform for most users.

Thanks for testing!

le_top · January 8, 2022, 11:30am

To make a daily backup for ZNP devices, I added the following automation to my setup.
It will backup to ‘local/nwk_backup_DAY.json’ and ‘local/nvram_backup_DAY.json’ .

alias: Daily ZNP Backup - Monthly rotation
description: Backup ZNP Zigbee configuration, monthly rotation
trigger:
  - platform: time
    at: '04:00'
condition: []
action:
  - service: zha_custom.execute
    data:
      command: znp_backup
      command_data: '{{ now().strftime("_%d") }}'
  - service: zha_custom.execute
    data:
      command: znp_nvram_backup
      command_data: '{{ now().strftime("_%d") }}'
mode: restart

abhi08638 · January 8, 2022, 1:09pm

I don’t believe it’s necessary to deleted the db and integration as that will lead to having to rename all the devices. I can test again by just doing a backup, nvram reset and restore and seeing if that works.

le_top · January 8, 2022, 1:15pm

If you can test it, I am sure there will be some grateful people.

I saw a video where using the CLI interface, the data was migrated from one coordinator to another and there was no talk of removing the database.
The integration was disabled though and the port and zigbee device type was changed. But in this case I don’t think that disabling the integration will make it work.

I suppose that this could even be more automated in the case of migrating from/to devices with different “ports”. But first, the other backup and restore methods need to be added.

abhi08638 · January 8, 2022, 2:14pm

Yeah prior to this, if you wanted to make a backup on HA, the ZHA integration needed to be disabled and you needed to ssl into HA to access the coordinator (he other option is using another environment altogether). For ZHA_CUSTOM, ZHA needs to be enabled otherwise the commands do not work.

I just did a nvram reset and here are the logs (everything looks okay, nvram backup was created successfully as well)

2022-01-08 09:09:45 INFO (MainThread) [custom_components.zha_custom] Running custom service: <ServiceCall zha_custom.execute (c:d0c61e312e531a68ba6ddcde5e90dec6): command=znp_nvram_reset>
2022-01-08 09:09:45 DEBUG (MainThread) [custom_components.zha_custom] module is <module ‘custom_components.zha_custom’ from ‘/config/custom_components/zha_custom/init.py’>
2022-01-08 09:09:45 INFO (MainThread) [custom_components.zha_custom.znp] Reading NVRAM from device
2022-01-08 09:09:59 INFO (MainThread) [custom_components.zha_custom.znp] Saving NVRAM to ‘/config/custom_components/zha_custom/local/nvram_backup_20220108_090945.json’
2022-01-08 09:09:59 INFO (MainThread) [custom_components.zha_custom.znp] NVRAM backup saved to ‘/config/custom_components/zha_custom/local/nvram_backup_20220108_090945.json’
2022-01-08 09:09:59 INFO (MainThread) [custom_components.zha_custom.znp] Reset NVRAM

Rebooted and did an NVRAM Restore (devices did not come back online after a restart)

2022-01-08 09:19:12 INFO (MainThread) [custom_components.zha_custom] Running custom service: <ServiceCall zha_custom.execute (c:9aba205011eaf778e1f71555824cc4d0): command=znp_nvram_restore>
2022-01-08 09:19:12 DEBUG (MainThread) [custom_components.zha_custom] module is <module ‘custom_components.zha_custom’ from ‘/config/custom_components/zha_custom/init.py’>
2022-01-08 09:19:12 INFO (MainThread) [custom_components.zha_custom.znp] Reading NVRAM from device
2022-01-08 09:19:27 INFO (MainThread) [custom_components.zha_custom.znp] Saving NVRAM to ‘/config/custom_components/zha_custom/local/nvram_backup_20220108_091912.json’
2022-01-08 09:19:27 INFO (MainThread) [custom_components.zha_custom.znp] NVRAM backup saved to ‘/config/custom_components/zha_custom/local/nvram_backup_20220108_091912.json’
2022-01-08 09:19:27 INFO (MainThread) [custom_components.zha_custom.znp] Restoring NVRAM from ‘/config/custom_components/zha_custom/local/nvram_backup.json’
2022-01-08 09:19:27 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall zha_custom.execute (c:9aba205011eaf778e1f71555824cc4d0): command=znp_nvram_restore>
Traceback (most recent call last):
File “/usr/src/homeassistant/homeassistant/core.py”, line 1511, in catch_exceptions
await coro_or_task
File “/usr/src/homeassistant/homeassistant/core.py”, line 1530, in _execute_service
await handler.job.target(service_call)
File “/config/custom_components/zha_custom/init.py”, line 73, in custom_service
await handler(
File “/config/custom_components/zha_custom/znp.py”, line 180, in znp_nvram_restore
nvram_obj = json.load(f)
File “/usr/local/lib/python3.9/json/init.py”, line 293, in load
return loads(fp.read(),
io.UnsupportedOperation: not readable

Reboot and did a network restore

2022-01-08 09:26:45 INFO (MainThread) [custom_components.zha_custom] Running custom service: <ServiceCall zha_custom.execute (c:6d70b0fa91d29d99c0be404ce7905b91): command=znp_restore>
2022-01-08 09:26:45 DEBUG (MainThread) [custom_components.zha_custom] module is <module ‘custom_components.zha_custom’ from ‘/config/custom_components/zha_custom/init.py’>
2022-01-08 09:26:55 INFO (MainThread) [custom_components.zha_custom.znp] Restore from ‘/config/custom_components/zha_custom/local/nwk_backup.json’
2022-01-08 09:26:55 INFO (MainThread) [custom_components.zha_custom.znp] Validating backup contents
2022-01-08 09:26:55 INFO (MainThread) [custom_components.zha_custom.znp] Backup contents validated
2022-01-08 09:26:55 INFO (MainThread) [custom_components.zha_custom.znp] Writing to device
2022-01-08 09:27:34 DEBUG (MainThread) [custom_components.zha_custom.znp] List of attributes/methods in znp [‘class’, ‘delattr’, ‘dict’, ‘dir’, ‘doc’, ‘eq’, ‘format’, ‘ge’, ‘getattribute’, ‘gt’, ‘hash’, ‘init’, ‘init_subclass’, ‘le’, ‘lt’, ‘module’, ‘ne’, ‘new’, ‘reduce’, ‘reduce_ex’, ‘repr’, ‘setattr’, ‘sizeof’, ‘str’, ‘subclasshook’, ‘weakref’, ‘_app’, ‘_config’, ‘_listeners’, ‘_port_path’, ‘_skip_bootloader’, ‘_sync_request_lock’, ‘_uart’, ‘_unhandled_command’, ‘_znp_config’, ‘callback_for_response’, ‘callback_for_responses’, ‘capabilities’, ‘capture_responses’, ‘close’, ‘connect’, ‘connection_lost’, ‘connection_made’, ‘detect_zstack_version’, ‘frame_received’, ‘load_network_info’, ‘network_info’, ‘node_info’, ‘nvram’, ‘remove_listener’, ‘request’, ‘request_callback_rsp’, ‘reset’, ‘set_application’, ‘version’, ‘wait_for_response’, ‘wait_for_responses’, ‘write_network_info’]
2022-01-08 09:27:34 INFO (MainThread) [custom_components.zha_custom.znp] Write done, call pre_shutdown(). Restart the device/HA after this.
2022-01-08 09:27:34 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall zha_custom.execute (c:6d70b0fa91d29d99c0be404ce7905b91): command=znp_restore>
Traceback (most recent call last):
File “/usr/src/homeassistant/homeassistant/core.py”, line 1511, in catch_exceptions
await coro_or_task
File “/usr/src/homeassistant/homeassistant/core.py”, line 1530, in _execute_service
await handler.job.target(service_call)
File “/config/custom_components/zha_custom/init.py”, line 73, in custom_service
await handler(
File “/config/custom_components/zha_custom/znp.py”, line 109, in znp_restore
await app._znp.pre_shutdown()
AttributeError: ‘ZNP’ object has no attribute ‘pre_shutdown’

After the network restore, the following devices came back online immediately
1/4 sengled A19 RGB bulbs
1/1 Ikea Signal Repeater
1/1 Smart Outlet

After waiting about 5 minutes, the rest of the sengled bulbs came back online!

Conclusion: Deletion of the ZHA integration and Zigbee.db is not needed AND a power cycle of mains powered devices is not necessary (I dont have any battery powered devices to test with but I assume those would rejoin if you wake them manually). See procedure below:

The procedure should be the following (follow “a” steps if migrating to a new coordinator):

Backup using the znp_backup command in the zha_custom service. Verify that the nwk_backup.json file is generated in the local directory
Remove the original coordinator from your system. Insert the new Coordinator.
2a) Remove the ZHA Integration from Home Assistant (only needed for migrating to a new coordinator with a different serial path; the alternative is to modify HA’s config file directly to update the current integration’s serial path and baudrate)
2b) Rename/Move the zigbee.db file (should not be needed. If this is done then the restore will not remember entity names)
Restart Home Assistant
3a) Add the ZHA Integration to Home Assistant (needed if you removed it)
Restore using the znp_restore command (If you used a custom file name for the backup then make sure you name it back to nwk_backup.json)
Check the logs (currently the pre_shutdown call failed for the first successful test, but that is not critical)
Restart HA
Check that everything is ok

NOTES:

Devices may take a while to rejoin the network as the Zigbee specification requires them to “back-off” in case of communication problems
You may speed up the process by power cycling devices
Devices may not be instantly responsive due to the zigbee mesh needing to be recreated

abhi08638 · January 8, 2022, 3:12pm

You should definitely add this automation as an example on the git page so other users can use it as a template!

le_top · January 8, 2022, 4:56pm

I did something better. I created a blueprint to add the backup automation: .

I updated the procedure in the readme (thank you for your adjustments) and mentionned the blueprint as well.

le_top · January 8, 2022, 6:50pm

Note that the NVRAM restore did not succeed because the nvram_backup.json file did not exist (it’s created only with a nvram_backup service without custom_data parameter. You’ld need to copy the file to nvram_backup.json in your case.

abhi08638 · January 8, 2022, 8:15pm

Yeah just realized I forgot to rename the nvram backup file. Currently when executing znp_nvram_reset, it calls znp_nvram_backup and it generates the file as nvram_backup_date_time.json. Next I’ll try just a normal nvram backup/reset/restore to see if that works

2022-01-08 15:34:54 INFO (MainThread) [custom_components.zha_custom] Running custom service: <ServiceCall zha_custom.execute (c:e4cba7611269e3c79cd451767b80a50e): command=znp_nvram_restore>
2022-01-08 15:34:54 DEBUG (MainThread) [custom_components.zha_custom] module is <module ‘custom_components.zha_custom’ from ‘/config/custom_components/zha_custom/init.py’>
2022-01-08 15:34:54 INFO (MainThread) [custom_components.zha_custom.znp] Reading NVRAM from device
2022-01-08 15:34:56 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x7EC9:1:0x0702]: async_initialize: all attempts have failed: [DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’)]
2022-01-08 15:35:08 INFO (MainThread) [custom_components.zha_custom.znp] Saving NVRAM to ‘/config/custom_components/zha_custom/local/nvram_backup_20220108_153454.json’
2022-01-08 15:35:09 INFO (MainThread) [custom_components.zha_custom.znp] NVRAM backup saved to ‘/config/custom_components/zha_custom/local/nvram_backup_20220108_153454.json’
2022-01-08 15:35:09 INFO (MainThread) [custom_components.zha_custom.znp] Restoring NVRAM from ‘/config/custom_components/zha_custom/local/nvram_backup.json’
2022-01-08 15:35:09 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall zha_custom.execute (c:e4cba7611269e3c79cd451767b80a50e): command=znp_nvram_restore>
Traceback (most recent call last):
File “/usr/src/homeassistant/homeassistant/core.py”, line 1511, in catch_exceptions
await coro_or_task
File “/usr/src/homeassistant/homeassistant/core.py”, line 1530, in _execute_service
await handler.job.target(service_call)
File “/config/custom_components/zha_custom/init.py”, line 73, in custom_service
await handler(
File “/config/custom_components/zha_custom/znp.py”, line 180, in znp_nvram_restore
nvram_obj = json.load(f)
File “/usr/local/lib/python3.9/json/init.py”, line 293, in load
return loads(fp.read(),
io.UnsupportedOperation: not readable

naijaping · January 8, 2022, 8:22pm

@le_top , latest update has messed up the files, can you please have a look. zha_custom folder now contains two folders another zha_custom folder and local folder.

le_top · January 8, 2022, 8:41pm

When performing a znp_nvram_reset, the backup is made for safety and a timestamp is added to avoid overwriting a backup that is actually restored from.
The znp_nvram_backup will backup to the nvram_backup.json file directly (if command_data is empty or not set).

naijaping · January 8, 2022, 8:45pm

@le_top , I have manually moved the files back and deleted the extra folders, and re started HA and everything is now back to normal

le_top · January 8, 2022, 8:46pm

I manage two setups, on one of them I installed zha_custom through HACS.

In case zha_custom is in the custom_components folder, then the zha_custom/zha_custom subfolder will not be of any effect and you can delete that - I do no know why that appeared. The ‘local’ folder is where the output files are written and it should be maintained across updates.

le_top · January 8, 2022, 8:48pm

Ok, that should be fine.
You can install using HACS - I’ve submitted a request to add zha_custom as a default repository, but there is some delay. In the mean time you can add this as a custom component by providing the github repository url directly.

naijaping · January 8, 2022, 8:51pm

@le_top ,Thats exactly what I did, I installed via HACS, I just dont know what went wrong during the update. even my backup was deleted automatically . but everything is fine now because I have made another backup now and it went smoothly .

Thanks for your help

abhi08638 · January 8, 2022, 9:20pm

Yeah I just don’t seem to have any luck with my devices rejoining after doing an NVRAM restore. The nvram backup seems to generates fine in both the backup and reset methods however. In any case, since the high level backup works I think that should be sufficient for most users.

le_top · January 8, 2022, 10:31pm

Maybe the NVRAM restore is not that efficient if the FrameCounter is not increased.
In the high level restore this increase is part of the parameters.

As far as I can see, the FrameCounter is located - LSB first - in:

        "LEGACY_NWK_SEC_MATERIAL_TABLE_START+0": "c46647005fbff3469813b72c",

c4664700 (lsb first) → 0x004766C4 (hex, MSB first) → 4679364 (base 10) = the FrameCounter in the nwk_backup.json .

So maybe increasing that number (first 4 bytes of the field above) helps. Adding 2500 is adding 0x0800 or 0x1000 to make it simple in hex and keeping close to that increment.
So “c4664700” would become “c46e4700” (+0x800) or “c4764700” (0x1000), the latter probably being the easiest in most cases. If that’s the trick, then this could be “automated” as well.
But as the more generic restore is functionnal, we can very well skip this.

abhi08638 · January 9, 2022, 5:23pm

Gotcha, unfortunately my HA instance crapped out on me so I ended up migrating everything to z2m. I don’t have another TI based coordinator so I can’t test znp commands anymore but I’m glad it’s in a state where the high level restore can be done without too many issues.

Eddie46848 · January 10, 2022, 6:54am

Hi
I have 30 devices in zha, but when I open the backup JSON file there is consistently 18 devices backed up.
9 of the devices missing are hive mains powered bulbs and a sonoff motion sensor.
It managed to back up the other sonoff motion sensors.

Any reason why there isn’t the 30 devices backed up?

Thanks