Issue with all_lights after 2023.7

I have a goodnight routine that I use to do many things, one of which is turn off a light group I created called light.all_lights. Since upgrading to 2023.7.x I have gotten an error when getting to that step of my automation. The automation stops and I get the following errors in my logs.

Logger: homeassistant.components.automation.good_night
Source: components/zha/light.py:471
Integration: Automation (documentation, issues)
First occurred: July 13, 2023 at 9:45:03 PM (22 occurrences)
Last logged: July 17, 2023 at 9:40:59 PM

🏠 House Mode: Night: Parallel action at step 1: parallel 2: Error executing script. Unexpected error for call_service at pos 1:
🏠 House Mode: Night: Error executing script. Unexpected error for parallel at pos 1:
While executing automation automation.good_night
🏠 House Mode: Night: Error executing script. Unexpected error for call_service at pos 4:
🏠 House Mode: Night: Error executing script. Unexpected error for call_service at pos 13:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 310, in request
    return await req.result
           ^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 468, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 703, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 665, in _async_run_long_action
    return long_task.result()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 1965, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2005, in _execute_service
    return await cast(
           ^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 848, in entity_service_call
    response_data = task.result()  # pop exception if have
                    ^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1192, in async_request_call
    return await coro
           ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 892, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 593, in async_handle_light_off_service
    await light.async_turn_off(**filter_turn_off_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/group/light.py", line 194, in async_turn_off
    await self.hass.services.async_call(
  File "/usr/src/homeassistant/homeassistant/core.py", line 1965, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2005, in _execute_service
    return await cast(
           ^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 848, in entity_service_call
    response_data = task.result()  # pop exception if have
                    ^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1192, in async_request_call
    return await coro
           ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 892, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 593, in async_handle_light_off_service
    await light.async_turn_off(**filter_turn_off_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/group/light.py", line 194, in async_turn_off
    await self.hass.services.async_call(
  File "/usr/src/homeassistant/homeassistant/core.py", line 1965, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2005, in _execute_service
    return await cast(
           ^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 848, in entity_service_call
    response_data = task.result()  # pop exception if have
                    ^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1192, in async_request_call
    return await coro
           ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 892, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 593, in async_handle_light_off_service
    await light.async_turn_off(**filter_turn_off_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 471, in async_turn_off
    result = await self._on_off_cluster_handler.off()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 309, in request
    async with asyncio_timeout(timeout):
  File "/usr/local/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

and

Logger: homeassistant
Source: components/zha/light.py:471
First occurred: July 13, 2023 at 10:07:09 PM (11 occurrences)
Last logged: July 17, 2023 at 9:47:04 PM

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 310, in request
    return await req.result
           ^^^^^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1192, in async_request_call
    return await coro
           ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 892, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 593, in async_handle_light_off_service
    await light.async_turn_off(**filter_turn_off_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/group/light.py", line 194, in async_turn_off
    await self.hass.services.async_call(
  File "/usr/src/homeassistant/homeassistant/core.py", line 1965, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2005, in _execute_service
    return await cast(
           ^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 848, in entity_service_call
    response_data = task.result()  # pop exception if have
                    ^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1192, in async_request_call
    return await coro
           ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 892, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 593, in async_handle_light_off_service
    await light.async_turn_off(**filter_turn_off_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/light.py", line 471, in async_turn_off
    result = await self._on_off_cluster_handler.off()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/util.py", line 132, in retry
    return await func()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 375, in request
    return await self._endpoint.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/endpoint.py", line 253, in request
    return await self.device.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/zigpy/device.py", line 309, in request
    async with asyncio_timeout(timeout):
  File "/usr/local/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

automation trace and config below

alias: "🏠 House Mode: Night"
trigger:
  - entity_id:
      - input_select.house_mode
    platform: state
    to: Night
condition:
  - condition: template
    value_template: "{{ trigger.from_state.state != 'unavailable' }}"
  - condition: time
    after: "20:00:00"
    before: "02:00:00"
  - condition: or
    conditions:
      - condition: state
        entity_id: input_select.house_mode
        state: Evening
      - condition: state
        entity_id: input_select.house_mode
        state: Night
action:
  - data: {}
    service: switch.turn_on
    target:
      entity_id:
        - switch.lockup_home
        - group.alexa_do_not_disturb_switch
  - service: script.music_sound_machine
    data:
      volume: 0.3
      device: media_player.master_bedroom_echo_show
    enabled: true
  - service: fan.set_percentage
    data:
      percentage: 50
    target:
      entity_id: fan.master_bedroom_fan
    enabled: true
  - service: fan.turn_off
    data: {}
    target:
      entity_id: fan.downstairs_fans
  - service: script.living_room_fan_turn_off
    data: {}
  - data: {}
    entity_id: group.tvs
    service: homeassistant.turn_off
  - service: number.set_value
    target:
      entity_id:
        - number.milford_base_station_volume
        - number.entryway_keypad_volume
        - number.basement_keypad_volume
    data:
      value: "10"
  - service: switch.turn_off
    target:
      entity_id: switch.master_bedroom_laptop_charger
    data: {}
  - data: {}
    entity_id: group.adaptive_lights_sleep_mode
    service: switch.turn_on
  - delay:
      hours: 0
      minutes: 0
      seconds: 30
      milliseconds: 0
  - data: {}
    service: script.house_secure_notification
  - service: light.turn_off
    data: {}
    target:
      entity_id: light.outside_lights
  - service: light.turn_off
    data: {}
    enabled: true
    target:
      entity_id: light.all_lights
mode: single

You aren’t alone. I’m seeing this too. I can’t reproduce it regularly but my night routine has messed up the last two nights. I know my zigbee network is at pretty high utilization but it seems like now it fails a lot less gracefully.

I’m wondering if this has something to do with the service calls now responding with data

Same here, found a lot of my automations have become flaky since last update, not completely failing, just not a sloid as they used to be, however, I have rebuilt a few of them, not from scratch, just changed something ,then back again to force a “Save” and that seems to have stabilised them a little.

That didn’t seem to work for me. What I ended up doing as a workaround was move the light turn off to the last action in the automation. It still fails, but on the last step so all the actions complete.

Which Zigbee radio are you using?

I’ve also been having problems with Zigbee light groups since the latest update. Same thing, any automation or script that turns the group on or off, stops at that line. Not 100% of the time, but most of the time.

Call a service 'Light: Turn off' on All Lights except Ceiling
Stopped because an error was encountered at July 19, 2023 at 2:21:27 PM (runtime: 23.35 seconds)
Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>

This is with a Sonoff ZigBee Dongle Plus V2 (Dongle E)

I’m running a zzh stick

Did anyone raise an issue in github, especially since it seems to happen with the last release? I can’t see any open issue that looks like what you’re describing.

Please help the community by raising an issue and providing logs and insights.

Thanks!

Yes it looks to be related to this