Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

ecchodun · July 15, 2023, 7:36pm

Zigbee running off HUSBZB-1 for a few months now problem free. Today I’ve encountered “Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>” for several of my in-wall Enbrighten Zigbee switches. Nothing changed on the configuration side. ZHA Zigbee. Running on a VM Windows 11. Reset server, reset devices. Any ideas?

Error log reads as follows:

Logger: homeassistant.components.websocket_api.http.connection
Source: components/zha/light.py:360
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 2:11:35 PM (11 occurrences)
Last logged: 2:31:16 PM

[140321278311312] Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>
Traceback (most recent call last):
File “/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py”, line 205, in handle_call_service
await hass.services.async_call(
File “/usr/src/homeassistant/homeassistant/core.py”, line 1965, in async_call
response_data = await coro
^^^^^^^^^^
File “/usr/src/homeassistant/homeassistant/core.py”, line 2005, in _execute_service
return await cast(
^^^^^^^^^^^
File “/usr/src/homeassistant/homeassistant/helpers/entity_component.py”, line 235, in handle_service
return await service.entity_service_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/src/homeassistant/homeassistant/helpers/service.py”, line 848, in entity_service_call
response_data = task.result() # pop exception if have
^^^^^^^^^^^^^
File “/usr/src/homeassistant/homeassistant/helpers/entity.py”, line 1192, in async_request_call
return await coro
^^^^^^^^^^
File “/usr/src/homeassistant/homeassistant/helpers/service.py”, line 892, in _handle_entity_call
result = await task
^^^^^^^^^^
File “/usr/src/homeassistant/homeassistant/components/light/init.py”, line 582, in async_handle_light_on_service
await light.async_turn_on(**filter_turn_on_params(light, params))
File “/usr/src/homeassistant/homeassistant/components/zha/light.py”, line 360, in async_turn_on
result = await self._on_off_cluster_handler.on()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/zigpy/util.py”, line 132, in retry
return await func()
^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/zigpy/zcl/init.py”, line 375, in request
return await self._endpoint.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py”, line 253, in request
return await self.device.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/site-packages/zigpy/device.py”, line 293, in request
await self._application.request(
File “/usr/local/lib/python3.11/site-packages/zigpy/application.py”, line 824, in request
await self.send_packet(
File “/usr/local/lib/python3.11/site-packages/bellows/zigbee/application.py”, line 831, in send_packet
raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

cikaybe · July 21, 2023, 8:33am

I’m having the same problem. Any ideias?

ecchodun · July 21, 2023, 2:45pm

Bought a Sonoff dongle instead and switched to channel 22 and all is seemingly working.

egelinotte · August 11, 2023, 7:57am

Same issue for me. Many automations frequently failed for the same reason.

Zigbee network issue ? Coordinator issue ?

Anthony_Vd_Koppel · August 28, 2023, 4:47pm

I also have the same issue. Updated recently from 2023.6.x to 2023.8.x.

2023.6.x no issues

Now with 2023.8.x having issues with a script which fails with exactly this error.

Setup:
Proxmox
Supervised HA OS
Skyconnect USB
ZHA

Anthony_Vd_Koppel · August 28, 2023, 5:15pm

Ok, my Zigbee network is inresponsive more often, that’s why I use scripts to correct light changes / states (I have over 100 Zigbee devices running)…

Now as earlier mentioned, my script stops running when a Zigbee network error is presented (which was not the case on 2023.6.x).

I managed to “fix” this by ignoring the Zigbee error: with the continue_on_error parameter:

github.com/home-assistant/core

Add ability to continue scripts/automations on error

home-assistant:dev ← home-assistant:frenck-2022-0870

opened 07:58PM - 13 Apr 22 UTC

frenck

+225 -8

## Proposed change  Ideally, services throw errors when something goes wrong. Like, anything at all. We do a review on this nowadays to make everything more consistent. However, many services don't do that (e.g., when a device is offline, or on API errors), because it stops scripts and automation sequences. (And users sometimes raise issues for those occurrences, while actually correct). So that means we have a gap now. A user may want to ignore an error... and might not care if that single step failed or not. This PR introduces: `continue_on_error`. It can be set on each step in a script sequence and continues the sequence if the error is reasonable to handle. For example, broken templates and not handled errors by integration libraries (for example), are still raised, even when this option is used. ```yaml action: - alias: "If this one fails..." continue_on_error: true service: notify.super_unreliable_service_provider data: message: "I'm going to error out..." - alias: "I'm still OK! :)" service: persistent_notification.create data: title: "Hi there!" message: "I'm fine..." ``` The name `continue_on_error` has been chosen to be consistent with the existing `continue_on_timeout` we use in other places. ## Type of change  - [ ] Dependency upgrade - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New integration (thank you!) - [x] New feature (which adds functionality to an existing integration) - [ ] Breaking change (fix/feature causing existing functionality to break) - [ ] Code quality improvements to existing code or addition of tests ## Additional information  - This PR fixes or closes issue: fixes # - This PR is related to issue: - Link to documentation pull request: https://github.com/home-assistant/home-assistant.io/pull/22390 ## Checklist  - [x] The code change is tested and works locally. - [x] Local tests pass. **Your PR cannot be merged unless tests pass** - [x] There is no commented out code in this PR. - [x] I have followed the [development checklist][dev-checklist] - [x] The code has been formatted using Black (`black --fast homeassistant tests`) - [x] Tests have been added to verify that the new code works. If user exposed functionality or configuration variables are added/changed: - [ ] Documentation added/updated for [www.home-assistant.io][docs-repository] If the code communicates with devices, web services, or third-party tools: - [ ] The [manifest file][manifest-docs] has all fields filled out correctly. Updated and included derived files by running: `python3 -m script.hassfest`. - [ ] New or updated dependencies have been added to `requirements_all.txt`. Updated by running `python3 -m script.gen_requirements_all`. - [ ] For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description. - [ ] Untested files have been added to `.coveragerc`. The integration reached or maintains the following [Integration Quality Scale][quality-scale]:  - [x] No score or internal - [ ] 🥈 Silver - [ ] 🥇 Gold - [ ] 🏆 Platinum  To help with the load of incoming pull requests: - [x] I have reviewed two other [open pull requests][prs] in this repository. [prs]: https://github.com/home-assistant/core/pulls?q=is%3Aopen+is%3Apr+-author%3A%40me+-draft%3Atrue+-label%3Awaiting-for-upstream+sort%3Acreated-desc+review%3Anone+-status%3Afailure  [dev-checklist]: https://developers.home-assistant.io/docs/en/development_checklist.html [manifest-docs]: https://developers.home-assistant.io/docs/en/creating_integration_manifest.html [quality-scale]: https://developers.home-assistant.io/docs/en/next/integration_quality_scale_index.html [docs-repository]: https://github.com/home-assistant/home-assistant.io

ecchodun · August 28, 2023, 5:36pm

The problem with continue_on_error is that it “skips” the initial action even though the initial action is either the only action desired or one that would run eventually if it continually tried. To this end, I implemented a check-first function (which admittedly is tedious) for those particular routines I “need” to run. It looks like the following:

This part continues the script until the device is on for at least one second.

        sequence:
          - repeat:
              until:
                - condition: device
                  type: is_on
                  device_id: 7c05fbc28fdf57db072d77b053221111
                  entity_id: switch.fence_lights
                  domain: switch
                  for:
                    hours: 0
                    minutes: 0
                    seconds: 1

And then this part turns it on which would result in an error if failed to deliver thus, because of the prior language, it keeps trying:

           sequence:
                - type: turn_on
                  device_id: 7c05fbc28fdf57db072d77b08869b0bd
                  entity_id: switch.fence_lights
                  domain: switch

Anthony_Vd_Koppel · August 28, 2023, 5:46pm

I use a repeat too in my script, but I also use a timeout between each repeat so my Zigbee network has the chance to recover again from all the Zigbee traffic going around (that is what causes this error, I believe

Markus99 · September 14, 2023, 12:46pm

This is ridiculous to have to implement these repeats and checks and such. Seems like DEFAULT HA behavior should be to retry if the error occurs and THEN error out after 3-5 repeats of the same error - and notify.

It’s even worse that it’s KILLING scripts / automations after the error. Like I want to put ‘continue_on_error’ into 100’s of scripts / automations…

Killed my kids wake-up routine this morning b/c the stupid front hall lamp failed to turn on…

I’m the same here. Been running ZWave / Zigbee networks on my HA/NUC combo for years and never had the # of issues I’ve had in the past week or so. What gives HA?

Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

OvalZyre · September 15, 2023, 12:34pm

Brand new to HA but just got this message for an automation to turn my lights off. Is there a fix other than setting up the repeat as mentioned above?

osmaa · September 15, 2023, 4:50pm

I replaced my aging IKEA Trådfri hub with a SONOFF Zigbee 3.0 Dongle-E (V2) and re-added all my (mostly IKEA) lights and other devices (41) altogether. This same error is now a constant occurence. For example, trying to adjust 4 blinds, or turn a group of light bulbs on or off, it’s likely that one of them will not respond, and the UI will pop up the “failed to deliver message” error. This doesn’t appear entirely like a sporadic single-message delivery problem - retrying the same action, it’s likely that the same device will fail again, until some later time when the mesh may have reconfigured.

So not only would implementing a retry logic in every automation be a crazy effort as @Markus99 wrote above, it probably wouldn’t even work. There’s something wrong at a deeper layer of mesh network maintenance.

The HA+Trådfri combo had its issues, like the tradfri integration losing connection and requiring a reload very frequently, but this behavior of losing zigbee connectivity to single devices is new to me.

francisp · September 16, 2023, 2:30pm

Do you use an USB extension cable on your dongle ? If not, you really should.

ecchodun · September 16, 2023, 4:25pm

Yes. I do have one. The errors are sporadic and not consistent.

Error while executing automation automation.new_automation_lights_new: Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

MikeDeltaHH · September 18, 2023, 8:37am

I have HA installed on an Intel NUC, use ZHA via SkyConnect and since a while exactly the same problem - the SkyConnect is connected to a USB 2.0 port with a 1m extension cable, any USB 3.0 interference sources are not in range. Every few days the same error pattern - some of my Zigbee sockets are suddenly no longer accessible or automations fail with the above message - if I restart HA then everything works again for a few days…

Gig13 · September 20, 2023, 5:04am

Seeing this same issue with my HA build - worked fine until getting into 8.x - now ZigBee is incredibly unreliable… This error pops up randomly and half my network is just dead. Restarting, or reseating the SkyConnect dongle does nothing - just gotta wait for HA to figure itself out…

bluetrevian · September 20, 2023, 9:41pm

Observing this same error–

Failed to call service light/turn_on. Failed to send request: Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

HA: Home Assistant Yellow 2023.9.2 Supervisor 2023.09.2 Operating System 10.5
Destination: VZM31-SN by Inovelli - Firmware: 0x01020200

Will attempt to change Zigbee/Thread channels to 25 in an attempt to avoid interference from other devices.

802-15-4_802-11_channels

ecchodun · September 20, 2023, 11:50pm

I tried channel 25 myself and it got worse.

thepaperpilot · September 24, 2023, 2:00pm

I’m having this issue as well - every couple of days I’ll wake up and find out lights etc. weren’t toggled on/off automatically like they’re supposed to - and typically resolve it by restarting HA and re-running the automations. I wish there was at least an easier way to detect these errors - there’s no indication they failed outside of the logs in the docker container. I’d love to get a notification in the notification drawer, and perhaps even an event so I can try telling it to restart HA (although I’m not sure how I’d make it restart and then try the automation again in a few minutes).

Zsoltt · October 3, 2023, 7:01pm

Hi all!

I’m coming from other solutions like Domoticz, Homey where I did not see similar errors…
But started to experience this <EmberStatus.DELIVERY_FAILED: 102> error…
I’m currently on ZHA with Sonoff Zigbee adapter running HA on Proxmox, on an Intel NUC. A few minutes ago I added an USB extension cable also, so could be that it will help also.

Is there anyone who migrated to Zigbee2mqtt and it solved the problem? (I have 27 devices and some of them are hard to reach for a reset…)

Thanks!

Jozo81 · October 9, 2023, 4:45pm

Got same problems too in September with Skyconnect. Before that everything worked perfectly for few months when I moved from old legacy Sonoff dongle and z2mqtt.

After ZigBee network became too unreliable to use I tested Sonoff zdongle-E and problems continued like with Skyconnect. After that I went back to my old Sonoff dongle (Texas_Instruments_TI_CC2531) and system is now functioning error free for few days. Must be some problem with Silicon labs chip/ZHA -connection.

Hopefully this issue is fixed on coming updates and I can move back to Skyconnect.