"zha-toolkit" - a big set of Zigbee commands on top of ZHA/zigpy

Yeah just realized I forgot to rename the nvram backup file. Currently when executing znp_nvram_reset, it calls znp_nvram_backup and it generates the file as nvram_backup_date_time.json. Next I’ll try just a normal nvram backup/reset/restore to see if that works

2022-01-08 15:34:54 INFO (MainThread) [custom_components.zha_custom] Running custom service: <ServiceCall zha_custom.execute (c:e4cba7611269e3c79cd451767b80a50e): command=znp_nvram_restore>
2022-01-08 15:34:54 DEBUG (MainThread) [custom_components.zha_custom] module is <module ‘custom_components.zha_custom’ from ‘/config/custom_components/zha_custom/init.py’>
2022-01-08 15:34:54 INFO (MainThread) [custom_components.zha_custom.znp] Reading NVRAM from device
2022-01-08 15:34:56 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x7EC9:1:0x0702]: async_initialize: all attempts have failed: [DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’), DeliveryError(‘Request failed after 5 attempts: <Status.NWK_INVALID_REQUEST: 194>’)]
2022-01-08 15:35:08 INFO (MainThread) [custom_components.zha_custom.znp] Saving NVRAM to ‘/config/custom_components/zha_custom/local/nvram_backup_20220108_153454.json’
2022-01-08 15:35:09 INFO (MainThread) [custom_components.zha_custom.znp] NVRAM backup saved to ‘/config/custom_components/zha_custom/local/nvram_backup_20220108_153454.json’
2022-01-08 15:35:09 INFO (MainThread) [custom_components.zha_custom.znp] Restoring NVRAM from ‘/config/custom_components/zha_custom/local/nvram_backup.json’
2022-01-08 15:35:09 ERROR (MainThread) [homeassistant.core] Error executing service: <ServiceCall zha_custom.execute (c:e4cba7611269e3c79cd451767b80a50e): command=znp_nvram_restore>
Traceback (most recent call last):
File “/usr/src/homeassistant/homeassistant/core.py”, line 1511, in catch_exceptions
await coro_or_task
File “/usr/src/homeassistant/homeassistant/core.py”, line 1530, in _execute_service
await handler.job.target(service_call)
File “/config/custom_components/zha_custom/init.py”, line 73, in custom_service
await handler(
File “/config/custom_components/zha_custom/znp.py”, line 180, in znp_nvram_restore
nvram_obj = json.load(f)
File “/usr/local/lib/python3.9/json/init.py”, line 293, in load
return loads(fp.read(),
io.UnsupportedOperation: not readable

@le_top , latest update has messed up the files, can you please have a look. zha_custom folder now contains two folders another zha_custom folder and local folder.

When performing a znp_nvram_reset, the backup is made for safety and a timestamp is added to avoid overwriting a backup that is actually restored from.
The znp_nvram_backup will backup to the nvram_backup.json file directly (if command_data is empty or not set).

@le_top , I have manually moved the files back and deleted the extra folders, and re started HA and everything is now back to normal

I manage two setups, on one of them I installed zha_custom through HACS.

In case zha_custom is in the custom_components folder, then the zha_custom/zha_custom subfolder will not be of any effect and you can delete that - I do no know why that appeared. The ‘local’ folder is where the output files are written and it should be maintained across updates.

Ok, that should be fine.
You can install using HACS - I’ve submitted a request to add zha_custom as a default repository, but there is some delay. In the mean time you can add this as a custom component by providing the github repository url directly.

@le_top ,Thats exactly what I did, I installed via HACS, I just dont know what went wrong during the update. even my backup was deleted automatically . but everything is fine now because I have made another backup now and it went smoothly .

Thanks for your help

Yeah I just don’t seem to have any luck with my devices rejoining after doing an NVRAM restore. The nvram backup seems to generates fine in both the backup and reset methods however. In any case, since the high level backup works I think that should be sufficient for most users.

Maybe the NVRAM restore is not that efficient if the FrameCounter is not increased.
In the high level restore this increase is part of the parameters.

As far as I can see, the FrameCounter is located - LSB first - in:

        "LEGACY_NWK_SEC_MATERIAL_TABLE_START+0": "c46647005fbff3469813b72c",

c4664700 (lsb first) → 0x004766C4 (hex, MSB first) → 4679364 (base 10) = the FrameCounter in the nwk_backup.json .

So maybe increasing that number (first 4 bytes of the field above) helps. Adding 2500 is adding 0x0800 or 0x1000 to make it simple in hex and keeping close to that increment.
So “c4664700” would become “c46e4700” (+0x800) or “c4764700” (0x1000), the latter probably being the easiest in most cases. If that’s the trick, then this could be “automated” as well.
But as the more generic restore is functionnal, we can very well skip this.

Gotcha, unfortunately my HA instance crapped out on me so I ended up migrating everything to z2m. I don’t have another TI based coordinator so I can’t test znp commands anymore but I’m glad it’s in a state where the high level restore can be done without too many issues.

Hi
I have 30 devices in zha, but when I open the backup JSON file there is consistently 18 devices backed up.
9 of the devices missing are hive mains powered bulbs and a sonoff motion sensor.
It managed to back up the other sonoff motion sensors.

Any reason why there isn’t the 30 devices backed up?

Thanks

At least we have this in common - I only have 8 devices in my backup and there are 18 in my network and I have only 2 listed as children.

According to the documentation of the “open coordinator format” this list concerns the children and devices with APS link keys shared with the coordinator. I suppose that the other devices are linked to their routers.

The backup procedure and format is implemented in zigpy-znp - zha_custom is merely reuses/adapts the cli interface code to provide it as a service.

@le_top Would you consider extending “zha custom” with Zigbee dongle firmware flashing operations?

zigpy-znp have tools for flashing the older CC2530 and CC2531 based dongles but not any newer:

https://github.com/zigpy/zigpy-znp/blob/d37ac29ea7f164e43445c0a87b5b85dd0067de0d/TOOLS.md#flash-operations

JelmerT’s cc2538 cc2538-bsl can however very easily flash CC2652/CC1352/CC2538 via Auto-BSL:

https://github.com/JelmerT/cc2538-bsl (requires ITead_Sonoff_Zigbee-delay patch for Sonoff)

Alternatively the fork llama-bsl for additional board auto-detection features (but no Sonoff patch yet):

https://github.com/electrolama/llama-bsl

Most newer Silicon Labs dongles also support Auto-BSL but need Elelabs Firmware Update Utility:

https://github.com/Elelabs/elelabs-zigbee-ezsp-utility

PS: Not sure if worth flashing CC2530/CC2531 since those older chips are no longer recommended.

FYI, zigpy-cli devs plan on soon migrating to a new unified backup commands for all radio libraries, see:

https://github.com/zigpy/zigpy-cli/pull/2

Initial implementation of cross-adapter network backup/restore

change introduces a network backup/restore utility for any adapter implementing the new radio API

I am willing to add new items to zha_custom if I do not need to spend too much time porting them. Time is my main limitation.

For instance, adding the flashing procedures already provided in Zigpy-* is fairly straightforward.

Adding JemerT’s project is too much work:

  • it’s not provided as a library (no pip install AFAICS);
  • If there is a unintegrated patch, it’s likely not part of the pip install (but I think one can pip install from a repository if it supports pip install);
  • Testing. Firmware upgrade is a critical procedure and takes time to test.

Stopping (disabling) and restaring ZHA could probably be added to the service during the upgrade procedure. Data backup (for safety) and restore (if needed) could also be added to the procedure.

I do not nkow to what extend the Python Scripts Integration gives the scripts access to the serial ports/sockets.
That integration could be a first step to porting to zha_custom by testing the firmware upgrades scripts as they are in the HA environment.

I suppose that zha_custom could probably call another python script natively.

The external scripts can possibly also be fetched using the exposed zip deliveries in the repositories.

Ultimately - if the respective licences allow it - the firmware binary could also be preloaded or fetched from an external source (listed in some file in zha_custom so that an upgrade to “latest” is possible).

But it’s mainly my personal “human” bandwidth problem. I can help out with blocking issues and integration, but not do all the development.

1 Like

I’ll update zha_custom when that is available - I already noticed something was in the making.
Backing up was not my initial reason for extending the original zha_custom ! But I did add a daily coordinator backup to my system.

1 Like

I made it a little bit easier to setup a custom command from the UI.

In UI mode, all commands are available in the drop-down and you can select a ZHA Entity from the second drop down.
For quite a few commands that will do the job.
If you need more than just text for ‘command_data’, you can still select the command and the entity in UI mode and then switch to YAML mode to finish setting up the service call.

You can still enter the IEEE address in Yaml mode if you want to.

1 Like

Any tips on getting/retrieving a list of all active and/or inactive devices known to the Zigbee coordinator?

Nice to get lists with ID, IEEE, LQI, RSSI, Last Seen, power mode, etc. so could example sort by LQI.

Kind of as alternative to zha-device-exporter → https://github.com/dmulcahey/zha-device-exporter

Another use-case scenario could be to export data to Prometheus or other monitoring/alerting tools, e.g.

https://community.home-assistant.io/t/zha-prometheus-export-zigbee-device-informations-to-prometheus/360860

GitHub - dmulcahey/zha-network-card: Custom Lovelace card that displays ZHA network and device information does this for the UI.

What would the benefit have to be versus zha-device-exporter (I have the network card, I did not check the exporter) ?

Ok, this is something that will be useful to those that want to avoiding reading zigbee attributes from the database.

The zigbee attributes you read or write (and read back) can now be written directly to home assistant states or state attributes

For instance, I want to read apparent power from my electrical meter and it’s not available through the standard functionnallity, I can now call this service repeatedly:

service: zha_custom.execute
data:
  command: attr_read
  ieee: sensor.my_smartenergy_metering
  cluster: 0xb04
  attribute: 0x50f
  state_id: sensor.test
  state_attr: apparent_power
  allow_create: True

That reads the attribute and on success updates the attribute. Without “state_attr”, it updates the attribute itself (and you’ll get the updated time as well).