ZHA integration to do nightly backup of both Zigbee Coordinator adapter/dongle/stick and Zigbee network/database?

[Hedda]

@Waparo

I find that there is no practical way to maintain this system into the future with ZHA as the radio network is apparently stored on the zigbee radio stick.

With the ZHA component being a native integration it really makes ZHA a very good entry point for Home Assistant beginners, so IMHO each additional feature that makes it even simpler and easier to use is great news for new users. Super fun when HA beginners say “plug and play” and “it just works”.

However, I think the ability to easily make backup and perform recovery should be equally important.

@Waparo

It would be very helpful if we could see some HA road map to guide us.

The only thing we do know for sure is that the official “Amber” hardware appliance for Home Assistant will come with a Silicon Labs (EZSP) based Zigbee module and that those are only working well with ZHA today. So my guess, therefore, is that Nabu Casa / Home Assistant founders will prioritize ZHA.

https://www.home-assistant.io/blog/2021/09/13/home-assistant-amber/

“Home Assistant Amber includes the latest and greatest Zigbee radio chip from Silicon Labs to communicate with smart devices in your home. The chip is forward-compatible with the upcoming Matter smart home connectivity standard, in which Silicon Labs is heavily involved.”

https://www.crowdsupply.com/nabu-casa/home-assistant-amber

…and while Zigbee2MQTT did get initial support for Silicon Labs (EZSP) Zigbee adapters that code was all done by one developer and he is no longer maintaining it in Z2M so there is no future roadmap for Silicon Labs (EZSP) support in Zigbee2MQTT.

@Waparo

Should I switch to zigbee2mqtt? I am prepared to get new coordinators if necessary to continue with ZHA as I thought ZHA was the officially sanctioned future path for zigbee on HA.

ZHA is the only Zigbee implementation that is a natively integrated component of Home Assistant so I hope that the Home Assistant founders will primarily promote that over third-party integrations, but I have not read anywhere that ZHA is “the officially sanctioned future path for zigbee on HA” or that “HA devs have identified zigbee with ZHA as one of two protocols for communication between a HA system and zigbee compliant devices”.

@Waparo

The HA devs have identified zigbee with ZHA as one of two protocols for communication between a HA system and zigbee compliant devices.

I assume that the other of the two protocols mentioned is Z-Wave, and it is understandable that the Z-Wave userbase in Home Assistant will be much easier to grasp as there is really only Silicon Labs based chips for it so far and most users will probably the Z-Wave JS integration for that if they are not using commerical gateways/bridges/hubs like Samsung SmartThings Hub, Telldus Live, and Fibaro Hub.
[/wrap]

Too me it does not sound like the statistics from Home Assistant Analytics give much insight into Zigbee users because can not really tell how many users or who is using what type of devices in third-party integrations unless the proprietary gateway/bridge/hub in practice only support one single protocol (like IKEA Trådfri Gateway and Philips Hue Bridge), see discussion here:

https://community.home-assistant.io/t/home-assistant-founders-believe-there-is-currently-only-around-50-000-installations-of-zha-integration-what-do-you-zigbee-users-in-the-community-think-about-those-statistics/357305/

@Waparo

I cannot practically make a second Conbee 2 stick copy on a regular basis.

If you switch to either a newer “Texas Instruments” (ZNP) or “Silicon Labs” (EZSP) based Zigbee dongle then you can automate backups via Batch Scripts using “zigpy-znp” or “bellows” CLI tools respectively.

https://www.home-assistant.io/integrations/zha#compatible-hardware

…but for ConBee/RaspBee I believe that using the deCONZ application and its GUI is the only way to backup them(?) so in that case not easy to automate ConBee/RaspBee backups if using ZHA or Zigbee2MQTT(?).

[Waparo]

Hedda, thanks for your insights and very detailed understanding of the situation. I am comfortable getting a couple of new coordinators along the lines you have indicated and strongly prefer this to further entanglement with a propriatory deconz environment. However, I had hoped to avoid using batch scripts , etc. Doable but time consuming given my current lack of these specific skills. (I am comfortable using Python in Appdaemon.)
If the Nabu Casa / HA is likely to provide a UI for managing zigbee backup in the near future I would prefer to wait for that. Alternatively, does the Everything Smart Home video which you included above implement the batch script to which you refer. I could probably follow that path.

[Hedda]

@Waparo

If the Nabu Casa / HA is likely to provide a UI for managing zigbee backup in the near future I would prefer to wait for that

No guarantees of that, I’m not even sure if any Home Assistant founders are using ZHA themselves(?).

@Hedda

Some zigpy radio libraries (like zigpy-znp and bellows) which the ZHA integration depends already feature integrated CLI tools for manual backup. The video by Everything Smart Home linked above goes over the basic steps to do backups and restoring manually with needed CLI commands covered in his gist.

FYI, zigpy developer puddly has however started development on a unified command line interface for zigpy radios so zigpy could hopfully contain all needed backup commands for any radio in the future(?).

https://github.com/zigpy/zigpy/issues/557

https://github.com/zigpy/zigpy/pull/848

https://github.com/zigpy/zigpy/issues/842

Perhaps if and when all unified backup commands are in zigpy then the “zigpy-cli” will get those as well.

https://github.com/zigpy/zigpy-cli

@Waparo

does the Everything Smart Home video which you included above implement the batch script to which you refer.

No it only contains bellows and zigpy-znp commands he used to do manual backup and restore, see:

https://gist.github.com/EverythingSmartHome/990f2436608fd347cd20cf0009b5cef4

https://github.com/zigpy/zigpy/wiki/Coordinator-Backup-and-Migration

https://github.com/zigpy/zigpy-znp/blob/4ac608c82b6f48d304bab9a20bd648450bed54a7/TOOLS.md#tools

https://community.home-assistant.io/t/backup-your-zha-husbzb-1-stick-and-even-seamlessly-migrate-to-a-new-stick-without-re-pairing/229044

So today you would have to write your own batch script and automate scheduling when it will run.

[CKolumbus (Chris Drexler)]

@all Hi everyone: any idea how this can be done on an HA raspberry based appliance? bellows depends on some crypto packages that need compilation. So even with a virtualenv to isolate the python packages this does not work on the Home Assistant OS installation.

[silverstone777]

Did somebody find a way to backup the Conbee II ? Following the instructions on Phoscon website by backing the zll.db file I get nowhere, I cannot find the file. Furthermore I am using ZHA and trying to load DeconZ only 3 devices show up …

[Hedda]

@silverstone777

Did somebody find a way to backup the Conbee II ?

zigpy developers are apparently working on deconz backup and there is a draft pull request for it here:

https://github.com/zigpy/zigpy-cli/pull/2

At least that is how I interpret answer from puddly when posted this feature request for zigpy-deconz:

https://github.com/zigpy/zigpy-deconz/issues/177

Started separate forum discussion thread on zigpy-deconz / zigpy-cli backup of ConBee/RaspBee here:

https://community.home-assistant.io/t/zha-libraries-will-soon-support-conbee-raspbee-backup-and-restore-that-can-be-used-for-zigbee-network-migrations/374782

Note! Understand that zigpy and the zigpy-deconz radio libraries which ZHA depends on are all free and open-source library with only volunteering developers working on them without pay so remember and you and anyone else are also are more than welcome to code the feature for it (the zigpy developers have so far been more than happy to help and other volenteering developers who are willing to step in to assist with helping to fix bugs or add features).

@CKolumbus

any idea how this can be done on an HA raspberry based appliance? bellows depends on some crypto packages that need compilation. So even with a virtualenv to isolate the python packages this does not work on the Home Assistant OS installation.

Off-topic, please either start a new separate discussion thread or reply to that existing thread instead:

https://community.home-assistant.io/t/backup-your-zha-husbzb-1-stick-and-even-seamlessly-migrate-to-a-new-stick-without-re-pairing/229044/

[Hedda]

btw, know that puddly who develops zigpy-znp is planing to implement unified backup into zigpy-cli that will mean that once implemented you can use same backup command to backup all radio brands. See:

https://github.com/zigpy/zigpy-cli/pull/2

Implement new radio API

[Hedda]

Tip! @le_top added possibility to backup your Zigbee network to a JSON file within Home Assistant by using “zha_custom” custom component by mdeweerd (originally fork of zha_custom by Adminiuga):

https://community.home-assistant.io/t/zha-custom-service-to-send-custom-zha-commands-extra-functions/373346

@le_top

I just added the possibility to backup the ZNP network data to a json file from within HA. Often used to migrate to a new network coordinator key.

Backup ZNP network data

Use to transfer to another ZNP key later, backup or simply get network key and other info.

The output is written to the customisation directory as ‘nwk_backup.json’

service: zha_custom.execute
data:
  command: znp_backup

[Hedda]

@Hedda

Tip! @le_top added possibility to backup your Zigbee network to a JSON file within Home Assistant by using “zha_custom ” custom component by mdeweerd (originally fork of zha_custom by Adminiuga):

"zha-toolkit" - a big set of Zigbee commands on top of ZHA/zigpy

Mentioned “zha_custom” fork by @le_top has been renamed to “zha-toolkit” and moved to a new repository from mdeweerd on GitHub and a Blueprint make it a bit easier to achieve nightly backups:

https://community.home-assistant.io/t/daily-znp-zigbee-coordinator-backup-blueprint/377149

https://github.com/mdeweerd/zha-toolkit

https://github.com/mdeweerd/zha-toolkit/blob/dev/README.md#znp_backup-backup-znp-network-data

https://community.home-assistant.io/t/zha-toolkit-toolkit-service-providing-low-and-high-level-zigbee-commands-through-zha-zigpy/373346

[bartkummel]

Hi,

I’m planning to migrate to a new coordinator. (I’ve ordered one of those Sonoff Zigbee 3.0 USB Dongle Plus devices.) I was trying to make a backup of my current coordinator. I’ve installed zha_toolkit. However, when I execute the following command:

service: zha_toolkit.execute
data:
  command: znp_backup

I get this error:

'ControllerApplication' object has no attribute '_znp'

What’s going wrong here and how can I fix that?

[francisp (Francis)]

See this:

"zha-toolkit" - a big set of Zigbee commands on top of ZHA/zigpy

I tentatively added ezsp_backup - it’s untested. service: zha_toolkit.execute data: command: ezsp_backup # Optional command_data, string added to the basename. # With this example the backup is written to `nwk_backup_20220105.json` command_data: _20220105 The core of the backup “wrapper” in the toolkit is in ezsp.py/function ezsp_backup . It may need adjustments.

znp_backup is for CC2xxx (TI) adapters, ezsp_backup for Silicon Labs based adapters.

[bartkummel]

Ah, that wasn’t mentioned in the readme. Are the backups compatible? In other words, can I restore my EZSP backup to my new CC2xxx stick?

[bartkummel]

2022-01-19 17:17:43 DEBUG (MainThread) [custom_components.zha_toolkit.utils] Parameters '{'command': 'ezsp_backup'}'
2022-01-19 17:17:43 DEBUG (MainThread) [custom_components.zha_toolkit] module is <module 'custom_components.zha_toolkit' from '/config/custom_components/zha_toolkit/__init__.py'>
2022-01-19 17:17:43 DEBUG (MainThread) [custom_components.zha_toolkit] event_data {'ieee_org': None, 'ieee': 'None', 'command': 'ezsp_backup', 'start_time': '2022-01-19T16:17:43.731062+00:00', 'errors': ['AttributeError("\'ControllerApplication\' object has no attribute \'_znp\'")'], 'params': {'cmd_id': None, 'endpoint_id': None, 'cluster_id': None, 'attr_id': None, 'attr_type': None, 'attr_val': None, 'min_interval': None, 'max_interval': None, 'reportable_change': None, 'dir': None, 'manf': None, 'tries': 1, 'expect_reply': True, 'args': [], 'state_id': None, 'state_attr': None, 'allow_create': False, 'event_success': None, 'event_fail': None, 'event_done': None, 'read_before_write': True, 'read_after_write': True, 'write_if_equal': False}, 'success': False}
2022-01-19 17:17:43 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Unexpected error for call_service at pos 1: 'ControllerApplication' object has no attribute '_znp'

Well, ezsp_backup gives me exactly the same result…

[francisp (Francis)]

Maybe open an issue on his github ?

[bartkummel]

Done. It’s acutally the very first issue in the repo: Weird error · Issue #1 · mdeweerd/zha-toolkit · GitHub

[le_top]

It’s mentionned in the readme, but you can always suggest an improvement !

I prepared a more general backup command that will direct to the right backup procedure.

[Hedda]

@bartkummel

I’m planning to migrate to a new coordinator`.

@bartkummel

Are the backups compatible? In other words, can I restore my EZSP backup to my new CC2xxx stick?

First of all, please respect that this specific thread is a feature request discussion and meant for support so please start a separate thread if asking for support/help and not discussing the feature request.

Understand that for support will need Zigbee Coordinator chip and firmware have today to migrate from.

As franscisp mentioned above; “znp_backup” (backup via zigpy-znp) is only for Texas Instruments Z-Stack / ZNP adapters/firmware, and “ezsp_backup” (backup via bellows) is for only Silicon Labs EZSP (Silabs EmberZNet Serial Protocol) adapters/firmware. There is as of yet no backup support for other types of Zigbee Coordinator like “deconz” (ConBee/RaspBee), “zigate” (ZiGate), or “xbee” (Digi XBee).

Both “znp_backup” and “ezsp_backup” save/restore using “Open ZigBee Coordinator Backup Format” which indeed means that you can backup a Silabs EZSP adapter and restore to TI ZNP or vice versa in order to use it as a migration method.

https://github.com/zigpy/open-coordinator-backup

GitHub - zigpy/open-coordinator-backup: Open Zigbee coordinator backup format

The same Open ZigBee Coordinator Backup Format can obviously be used for migrating from older to newer adapter hardware/firmware with old respectively new chips from same chip manufacturer. Read:

https://github.com/zigpy/zigpy/wiki/Coordinator-Backup-and-Migration

Coordinator Backup and Migration

[Hedda]

@Hedda

Request that ZHA integration run backup every night for your Zigbee network…

I believe that Zigbee network + Zigbee Coordinator backups in the ZHA integration are also part of what Home Assistant founders indirectly thought about in new blog-post about “Streamlining Experiences”:

https://www.home-assistant.io/blog/2022/01/19/streamlining-experiences/

“With Home Assistant the last few years we’ve been focusing on making things easier, stable, and faster. More things can be managed via the UI, most YAML-based integrations can be reloaded without restarting and if something breaks, safe mode and built-in backups have your back.”

[Bliksembaard (Tijs Rijnders)]

I really hope so.
I’m using the ZHA + ConBee2 setup and the single point of failure this poses is gnawing at the back of my head for a while now.
It does make me happy to see so much dedicated people working an an Esperanto version of the Zigbee network. This gives me the reassurance a decent option will be available in the near future.
Needless to say I upvoted this feature request!

[basmeijer]

Indeed. I was in the assumption I could just briefly disable ZHA, switch to zigbee2mqtt to test something, disable and stop that, enable ZHA again but… wrong!!!
No more working devices, everything needs to be re-paired again (opening up all those wall switches and build in devices etc.). I mean, the devices are still there and recognised in the integration, even if you delete it and add it again, but they keep greyed out and basically the network is just dead.

Home Assistant Core definitely needs this backup function build in. Even with the nice Home Assistant Blue (OS) Odroid nothing can go wrong or you can run around with your screwdriver (again) to copy/paste the right entity ID’s and walk trough all automations etc.
Became a bit reserved now expanding the Zigbee network on a regular base and will choose a strategy to wait until I have done all tests and with new devices that we want and then put it all in at some point and do not touch it ever again after. And pray the coordinator stick never dies.
Little extra’s could also be done with Z-Wave, that is way more stable with adding/excluding (though crazy expensive).