ZHA integration UI options to apply OTA firmware update (OTAU) per Zigbee device?

Hedda · July 9, 2022, 8:04am

Request Home Assistant frontend devs add UI options to ZHA integration for OTA firmware update(s).

Please see related zigpy development discussion → https://github.com/zigpy/zigpy/discussions/951

Back-story; This feature request for ZHA’s UI to feature “Update Device Firmware” function for OTA firmware updates of Zigbee devices is a continuation of the ZHA OTA updates notification idea related to “zha-toolkit” discussion with @le_top here → [Feature] Notify OTA updates · Issue #24 · mdeweerd/zha-toolkit · GitHub in regards to ota_notify in feature zigpy GitHub - mdeweerd/zha-toolkit: 🧰 Zigbee Home Assistant Toolkit - service for "rare" Zigbee operations using ZHA on Home Assistant (also related to mdeweerd’s script zha-toolkit/fetchOTAfw.sh at 7ed030fa51ea18c79e2e592d2252695cc528d42f · mdeweerd/zha-toolkit · GitHub which download additional Zigbee OTA images from GitHub - Koenkk/zigbee-OTA: A collection of Zigbee OTA files).

While the ZHA integration already does support Zigbee OTA updates it currently simply applies all available Zigbee OTA updates as soon as it finds any OTAU files are available, without any UI options for individual Zigbee devices.

So firstly, would it be possible for ZHA users to only get notifications about Zigbee OTA updates and have UI options to update per device?

Secondly, would it be possible for ZHA users to get UI options to just select which specific Zigbee devices they would like to OTA update?

Could ZHA integration and its UI frontend utilize Home Assistant’s new update entity type to notify about ZHA OTA updates availability?

https://www.home-assistant.io/integrations/update

https://www.home-assistant.io/integrations/#updates

What does ZHA component + Home Assistant core and frontend UI developers think?

(I guess that if something like that was implemented then it could be reused for Z-Wave JS as well as possibly Matter/Thread in the future?).

Read news in Home Assistant 2022.4 release that introduced update as a brand new entity type which seams made for features like these?

https://www.home-assistant.io/blog/2022/04/06/release-20224/#introducing-update-entities

You can now upgrade your devices from inside of Home Assistant just as you would for Home Assistant Releases. You will also notice the number of updates pending in the sidebar by the indicator badge on your configuration cog icon. It’s now possible to read the release notes, skip or even install the upgrade to your device in one convenient place.

Introducing update entities

Say “hi!” to a brand new entity type: update.

Update entities can tell you if an update is available for your device and service and, in some cases, allow you to install the update straight from Home Assistant!

And the beautiful thing is: They show up in your configuration Dashboard, just like a Home Assistant, Home Assistant OS, or add-on update. As a matter of fact, those are now update entities too!

Now updates are entities, other integrations can provide them too! For example, a WLED firmware update for your LED strip will show up and installs with a single click.

Screen recording showing a WLED firmware update can be installed with a clickA WLED firmware update can now be installed just like any other update.

These update entities can be provided by integrations and provide more information about the update, like: Version information, links to release notes, and the possibility to skip that specific version offered.

The blue number indicator on the configuration cog icon in your sidebar tells you how many updates are pending.

The Home Assistant Supervisor (providing Core, OS, and add-on updates), WLED, Pi-hole, and Synology DSM integrations have implemented these brand new update entities in this release.

And because they are all entities, it means you can automate with them. For example, you could send a notification when a new add-on update is available for installation.

Hedda · July 9, 2022, 8:08am

FYI, Home Assistant and Z-Wave JS integration devs exposed OTA updates for Z-Wave devices in UI from Home Assistant 2022.7 release,

https://www.home-assistant.io/blog/2022/07/06/release-20227/#update-z-wave-devices-directly-from-home-assistant

and they also created a new Firmware Update Service (web service with an API) to power semi-automatic firmware updates in Z-Wave JS:

https://github.com/zwave-js/firmware-updates

https://www.home-assistant.io/blog/2022/07/06/release-20227/#update-z-wave-devices-directly-from-home-assistant

Z-Wave JS Firmware Update Service

This is the web service powering semi-automatic firmware updates in Z-Wave JS.

How it works

Firmware upgrades are defined in the files in the firmware directory. This information is parsed by the web service and used to respond to requests.

Z-Wave JS will query the web service on demand, providing the device identification of the device to be updated. If an update is available, the web service will respond with the necessary information to display to the user.

When desired, Z-Wave JS will download the update(s) from the provided URL, verify their integrity and install them

Update Z-Wave devices directly from Home Assistant

All software has bugs, including the software on your Z-Wave devices. But how to update those? Have no fear! As of today, we have a solution for this!

Thanks to the hard work and collaboration between @AlCalzone from Z-Wave JS and @raman325 from Home Assistant, you can now install updates on your Z-Wave devices directly from the Home Assistant interface!

Screenshot showing the new upload firmware item on the device page of a Z-Wave device page

On the device page of your Z-Wave device, there is now a menu item allowing you to install Z-Wave firmware updates onto your device manually.

Get a software update for your Z-Wave device from the manufacturer, and start an update in Home Assistant, which allows you to upload the update file you got from the manufacturer. The rest is pure magic!

Screenshot showing the a Z-Wave device that has firmware installation in progress

During update installation, you’ll be presented with the update’s progress.

It is that easy

Hedda · August 7, 2022, 9:12am

@puddly do you have any thoughts of the feature request feasibility of Zigbee OTA in ZHA’s UI?

JonatanRek · September 8, 2022, 7:05am

Are there any plans to integrate this feature to ZHA ? i see some ZHA work been made in 2022.9 but just regarding the backup

Hedda · September 8, 2022, 9:41am

Noticed too new Z-Wave JS firmware update service is now available in UI as of HA 2022.9 release:

https://www.home-assistant.io/blog/2022/09/07/release-20229/#z-wave-firmware-updates-now-live

Z-Wave Firmware updates now live

In 2022.7, we added support for updating Z-Wave device firmware, but the catch was that you needed the firmware files from the manufacturer in order to complete the update. @AlCalzone wasn’t satisfied with this approach and began working on a better solution. That better solution is here with the introduction of the Z-Wave JS Firmware Update Service! For supported manufacturers, Z-Wave JS can now automatically detect, download, and install a firmware update if it is available for your device. This will be done the same way you are currently updating Home Assistant. You will see a notification in your settings menu and you can update right from there. Thanks @raman325

For now, only Jasco products are supported. Nabu Casa has been talking to other manufacturers to get more devices supported. To ensure our users can take advantage of this feature, we added update entities for your devices that will check once a day for new firmware updates. Nifty!

Hedda · January 9, 2023, 1:24pm

Based on puddly’s proof-of-concept synchronous OTA manager posted in the comments in this other pull request for zigpy it sounds as if he maybe has plans for device-specific OTA updates in ZHA(?):

https://github.com/zigpy/zigpy/pull/1102

github.com/zigpy/zigpy

Application-level request/response matching

zigpy:dev ← puddly:puddly/request-matching

opened 11:18PM - 15 Nov 22 UTC

puddly

+520 -24

I'm hoping to move request/response matching into the main ControllerApplication…, allowing for more granular control over requests/responses within async code and for dynamically attaching objects that expect responses. In the future, this will allow for sending requests to NWK addresses directly and to wait for a response, without the need for intermediate `Device` objects. As a proof of concept, below is a synchronous OTA manager that can send a specific OTA image to a specific device, without modifying the global ZCL cluster definitions: ```Python from __future__ import annotations import typing import asyncio import logging import contextlib LOGGER = logging.getLogger(__name__) from zigpy.zcl import foundation from zigpy.zcl.clusters.general import Ota class OTAManager: def __init__( self, device: zigpy.device.Device, image: zigpy.ota.image.BaseOTAImage, progress_callback=None, ) -> None: self.device = device self.ota_cluster = None for ep in device.non_zdo_endpoints: try: self.ota_cluster = ep.out_clusters[0x0019] break except KeyError: pass else: raise ValueError("Device has no OTA cluster") self.image = image self._image_data = image.serialize() self.progress_callback = progress_callback self._upgrade_end_future = asyncio.get_running_loop().create_future() self.stack = contextlib.ExitStack() def __enter__(self) -> OTAManager: self.stack.enter_context( self.device._application.callback_for_response( src=self.device, filters=[ self.ota_cluster.commands_by_name["query_next_image"].schema(), ], callback=self._image_query_req, ) ) self.stack.enter_context( self.device._application.callback_for_response( src=self.device, filters=[ self.ota_cluster.commands_by_name["image_block"].schema(), ], callback=self._image_block_req, ) ) self.stack.enter_context( self.device._application.callback_for_response( src=self.device, filters=[ self.ota_cluster.commands_by_name["upgrade_end"].schema(), ], callback=self._upgrade_end, ) ) return self def __exit__(self, *exc_details) -> None: self.stack.close() async def _image_query_req(self, hdr, command): await self.ota_cluster.query_next_image_response( status=foundation.Status.SUCCESS, manufacturer_code=self.image.header.manufacturer_id, image_type=self.image.header.image_type, file_version=self.image.header.file_version, image_size=self.image.header.image_size, tsn=hdr.tsn, ) async def _image_block_req(self, hdr, command): block = self._image_data[ command.file_offset : command.file_offset + command.maximum_data_size ] if not block: await self.ota_cluster.image_block_response( status=foundation.Status.MALFORMED_COMMAND, tsn=hdr.tsn, ) return await self.ota_cluster.image_block_response( status=foundation.Status.SUCCESS, manufacturer_code=self.image.header.manufacturer_id, image_type=self.image.header.image_type, file_version=self.image.header.file_version, file_offset=command.file_offset, image_data=block, tsn=hdr.tsn, ) if self.progress_callback is not None: self.progress_callback( command.file_offset + len(block), len(self._image_data) ) async def _upgrade_end(self, hdr, command): await self.ota_cluster.upgrade_end_response( manufacturer_code=self.image.header.manufacturer_id, image_type=self.image.header.image_type, file_version=self.image.header.file_version, current_time=0x00000000, upgrade_time=0x00000000, tsn=hdr.tsn, ) self._upgrade_end_future.set_result(command.status) async def notify(self): await self.ota_cluster.image_notify( payload_type=(self.ota_cluster.ImageNotifyCommand.PayloadType.QueryJitter), query_jitter=100, ) async def wait(self): return await self._upgrade_end_future async def send_firmware( device: zigpy.device.Device, image: zigpy.ota.image.BaseOTAImage, ) -> typing.Any: # Force it to send image.header.file_version = 0xFFFFFFFF - 1 def progress(current, total): LOGGER.info( "OTA progress for %s: (%d / %d): %0.4f%%", device, current, total, (100 * current) / total, ) with OTAManager(device, image, progress_callback=progress) as ota: await ota.notify() await ota.wait() ```

Hedda · January 12, 2024, 3:24pm

FYI, looks like dmulcahey begun working on the update platform code for the ZHA integration, see:

https://github.com/home-assistant/core/pull/107612

and

https://github.com/zigpy/zigpy/pull/1321

As requested this should add initial support for Home Assistant’s universal update platform and expose Zigbee OTA updates for devices in the same unified way as other updates are in Home Assistant:

https://www.home-assistant.io/blog/2022/04/06/release-20224/#introducing-update-entities

Backstory if you missed it; Home Assistant core has that relatively new unified update platform and “update” entity type that can be initiated from the same place from UI/GUI by end users:

https://www.home-assistant.io/integrations/update

Again, several other integrations for Home Assistant, (such as the similar Z-Wave integration), are already utilizing thos new update platform to add update entities UI for device OTAU updates:

https://www.home-assistant.io/integrations/#updates

“Now updates are entities, other integrations can provide them too! For example, a WLED firmware update for your LED strip will show up and installs with a single click.”

“These update entities can be provided by integrations and provide more information about the update, like: Version information, links to release notes, and the possibility to skip that specific version offered.”

More references:

https://www.home-assistant.io/blog/2022/09/07/release-20229/#z-wave-firmware-updates-now-live

https://www.youtube.com/watch?v=wmYocYd0wts&t=390s&ab_channel=EverythingSmartHome

https://www.home-assistant.io/blog/2022/07/06/release-20227/#update-z-wave-devices-directly-from-home-assistant

Hedda · January 17, 2024, 12:55pm

Note! Manufacturers must add public URL in zigpy for OTA update images for automatic download, see:

OTA Information for Manufacturers · zigpy/zigpy Wiki · GitHub

Sadly not many manufacturers are publishing their OTA images publicly yet. If you are a user of their products then consider contacting manufacturers and encourage them to do so for all their products.

Otherwise the end user must manually provide the OTAU files from unofficial sources, if so, check out:

OTA Device Firmware Updates · zigpy/zigpy Wiki · GitHub

Screenshots from dmulcahey for the above-linked pull requests showing Zigbee OTA update in the UI:

Hedda · February 3, 2024, 4:15pm

Hedda:

FYI, looks like dmulcahey begun working on the update platform code for the ZHA integration, see:

https://github.com/home-assistant/core/pull/107612

and

https://github.com/zigpy/zigpy/pull/1321

As requested this should add initial support for Home Assistant’s universal update platform and expose Zigbee OTA updates for devices in the same unified way as other updates are in Home Assistant:

https://www.home-assistant.io/blog/2022/04/06/release-20224/#introducing-update-entities

Backstory if you missed it; Home Assistant core has that relatively new unified update platform and “update” entity type that can be initiated from the same place from UI/GUI by end users:

Update - Home Assistant

Again, several other integrations for Home Assistant, (such as the similar Z-Wave integration), are already utilizing thos new update platform to add update entities UI for device OTAU updates:

Integrations - Home Assistant

FYI, those pull requests by dmulcahey has now been merged so this should be in Home Assistant 2024.2

github.com/home-assistant/core

Add update platform to ZHA (bumps zigpy to 0.61.0)

home-assistant:dev ← dmulcahey:dm/zha-better-firmware-updates

opened 11:01PM - 08 Jan 24 UTC

dmulcahey

+1290 -18

## Proposed change  This PR adds update entities to ZHA. **REQUIRES:** https://github.com/zigpy/zigpy/pull/1321 Zigpy bumped to 0.61.0: https://github.com/zigpy/zigpy/releases/tag/0.61.0 ~There is still a lot to do here:~ - [x] ~error handling~ - [x] ~tests~ - [x] ~docs PR~ - [x] ~code cleanup~ ![device-page](https://github.com/home-assistant/core/assets/1335687/d2609657-0460-4c82-a504-a94e98b02f9f) ![dialog](https://github.com/home-assistant/core/assets/1335687/3667fb13-60f5-49e6-a840-e7c6c939e100) ![dialog-action](https://github.com/home-assistant/core/assets/1335687/9a7a5439-7113-4785-99c8-91bbc840fbb2) ![dialog-progress](https://github.com/home-assistant/core/assets/1335687/17420fd3-a054-4683-a950-a47b06b5faf6) ## Type of change  - [x] Dependency upgrade - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New integration (thank you!) - [X] New feature (which adds functionality to an existing integration) - [ ] Deprecation (breaking change to happen in the future) - [ ] Breaking change (fix/feature causing existing functionality to break) - [ ] Code quality improvements to existing code or addition of tests ## Additional information  - This PR fixes or closes issue: fixes # - This PR is related to issue: - Link to documentation pull request: https://github.com/home-assistant/home-assistant.io/pull/30824 ## Checklist  - [X] The code change is tested and works locally. - [X] Local tests pass. **Your PR cannot be merged unless tests pass** - [X] There is no commented out code in this PR. - [X] I have followed the [development checklist][dev-checklist] - [X] I have followed the [perfect PR recommendations][perfect-pr] - [X] The code has been formatted using Ruff (`ruff format homeassistant tests`) - [X] Tests have been added to verify that the new code works. If user exposed functionality or configuration variables are added/changed: - [X] Documentation added/updated for [www.home-assistant.io][docs-repository] If the code communicates with devices, web services, or third-party tools: - [ ] The [manifest file][manifest-docs] has all fields filled out correctly. Updated and included derived files by running: `python3 -m script.hassfest`. - [ ] New or updated dependencies have been added to `requirements_all.txt`. Updated by running `python3 -m script.gen_requirements_all`. - [ ] For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description. - [ ] Untested files have been added to `.coveragerc`.  To help with the load of incoming pull requests: - [ ] I have reviewed two other [open pull requests][prs] in this repository. [prs]: https://github.com/home-assistant/core/pulls?q=is%3Aopen+is%3Apr+-author%3A%40me+-draft%3Atrue+-label%3Awaiting-for-upstream+sort%3Acreated-desc+review%3Anone+-status%3Afailure  [dev-checklist]: https://developers.home-assistant.io/docs/development_checklist/ [manifest-docs]: https://developers.home-assistant.io/docs/creating_integration_manifest/ [quality-scale]: https://developers.home-assistant.io/docs/integration_quality_scale_index/ [docs-repository]: https://github.com/home-assistant/home-assistant.io [perfect-pr]: https://developers.home-assistant.io/docs/review-process/#creating-the-perfect-pr

github.com/zigpy/zigpy

Update OTA handling

zigpy:dev ← dmulcahey:dm/ota-updates

opened 04:42PM - 08 Jan 24 UTC

dmulcahey

+666 -269

Original Ota manager implementation from the description here: https://github.co…m/zigpy/zigpy/pull/1102. (Thanks @puddly) All Ota handling is removed from the cluster definition in Zigpy except for image querying. Thinking the usage will be something like this: HA update entity executes some flavor of this code at some pre defined interval to determine if there is an update available: ```python await self._ota_cluster_handler.image_notify( payload_type=( self._ota_cluster_handler.cluster.ImageNotifyCommand.PayloadType.QueryJitter ), query_jitter=100, ) ``` This will cause Zigpy to check for a new FW and if one is found an event with the firmware will be pushed to HA. when the user clicks the update button in HA `async def update_firmware` is executed passing in the progress callback from HA. Note: I tried to keep as much of the logging similar to what was being sent before.

https://rc.home-assistant.io/blog/2024/01/31/release-20242/#update-entities-for-zha-devices

Update entities for ZHA devices

ZHA (Zigbee Home Automation, our built-in Zigbee integration) now provides update entities to Home Assistant!

This means you can get notified when a device has a firmware update available, and you can trigger the update from Home Assistant!

Noticed the weird version numbers? Unfortunately, that is how version numbers in the Zigbee standard work and look, we currently have no way to translate them to something more human-readable format.

Currently, it supports updating Inovelli, OSRAM/Ledvance, Sonoff, and ThirdReality devices, and we are looking at adding support for more brands in the future.

Hedda · February 15, 2024, 2:08pm

FYI; puddly is now working on refactoring/rewriting the zigpy code for Zigbee OTA providers and OTA image metadata logic for better OTAU image matching:

github.com/zigpy/zigpy

OTA v2: provider and metadata rewrite

zigpy:dev ← puddly:puddly/ota-provider-rewrite

opened 09:05PM - 12 Feb 24 UTC

puddly

+5545 -3543

Corresponding HA Core branch: https://github.com/puddly/core/tree/puddly/zigpy-n…ew-ota --- This is a rewrite of all the current current OTA providers and OTA image metadata logic. Now that ZHA handles initiating updates, no provider will ever be contacted by default on startup, even if you enable them: the index will be downloaded only if a device compatible with the provider requests an image. The main reason for this rewrite is image matching and metadata. OTA images alone do not contain enough data to exclude bad devices (i.e. Tuya). Secondary metadata needs to be bundled with the OTA image. For this reason, all classes dealing with OTA will now work with `OtaImageWithMetadata` objects instead of raw OTA images. New OTA metadata: - `min_current_file_version` and `max_current_file_version`: allows for chained updates (i.e. A -> B -> C) - `manufacturer_names`: whitelist only specific manufacturers - `model_names`: whitelist only specific models - `specificity`: boost the numerical specificity of a specific OTA image. All of this metadata will be used to exclude bad devices (i.e. Tuya). From an end-user perspective, `otau_dir` is gone and has been replaced with `z2m_local_index`/`z2m_remote_index`. If you want to risk updating with OTA files without metadata, `advanced_ota_dir` exists. Example: ```yaml zigpy: ota: # It's enabled by default enabled: true # You probably do not need this, it's for development advanced_ota_dir: /path/to/raw/ota/files allow_advanced_ota_dir: ... # Points to Z2M OTA `index.json` if stored locally z2m_local_index: /path/to/z2m-ota/index.json # Points to Z2M OTA `index.json` URL z2m_remote_index: "https://raw.githubusercontent.com/Koenkk/zigbee-OTA/master/index.json" ```

FYI, that work is also indirectly related to these two zigpy wiki pages about Zigbee OTA inside zigpy/ZHA: