2023.8: Update breaks automations

[erik3 (Erik)]

@Edwin_D

Are you sure that you’re just waiting for mqtt send and not waiting for the transition to complete too? Have you looked at the timings in the trace?

Yes, the transition is made internally in the lamps. It is possible to set a transition period of say 60 seconds, and the automation (in queued mode) will complete in less than a second anyway.

@EndUser

Is there any negative to use queued ?

Not generally, but it depends on your use case. In my case, it is going to be a problem (given that I have 2s debounce). Lets say you quickly press buttons 5 times. Then “something” will be going on in the background for the coming 10 seconds at least. If you press different buttons, then it is going to be “replayed” the way you clicked them. This might be what your want, but also it might not.

[123 (Taras)]

@EndUser

any negative to use queued ?

It represents a fundamental change in how Home Assistant handles script execution.

Imagine I gave you a jigsaw puzzle and asked you to complete it. While you are busy assembling the puzzle, I ask you to assemble the same puzzle again.

• In single mode you would ignore my second request and finish building the puzzle.

• In restart mode you would stop building the puzzle and start over again.

• In queued mode you would finish building the puzzle before starting over again.

So depending on what your script is designed to do, changing its mode may significantly impact its behavior (when there’s another request to execute the in-progress script).

[Lux4rd0]

@Edwin_D

      - type: custom:auto-entities
        card:
          show_name: true
          show_icon: false
          show_state: false
          type: glance
          title: Lopende automatiseringen
          columns: 1
        filter:
          include:
            - domain: automation
          exclude:
            - attributes:
                current: 0
        sort:
          method: last_triggered
        show_empty: true

Ohhhhh - I love this! like LOVE LOVE this!! I needed to reformat a little:

type: custom:auto-entities
card:
  show_name: true
  show_icon: false
  show_state: false
  type: glance
  title: Running Automations
  columns: 1
filter:
  include:
    - domain: automation
  exclude:
    - attributes:
        current: 0
sort:
  method: last_triggered
show_empty: true

But super useful. I can see the triggers running and falling off. Since I’m now on 2023.06 everything is back to normal. I’ll install 2023.08 and see what results I get - now that I know I can get back to stable with forcing a core version change.

[EndUser]

@123

In queued mode you would finish building the puzzle before starting over again.

I have an automation in one room that sometimes does not run properly and queue might help. I believe it may be because the “controller” device is a ZWave 500 series, which I may need to replace.

[thorshammer]

@123

Or they fail regardless if the entities are based on Zwave or Zigbee?

Z-Wave may be another red herring. I myself do not use Z-Wave at all. Thus, I doubt it’s related to this issue.

[Lux4rd0]

I upgraded just now to test:

Home Assistant 2023.8.1
Supervisor 2023.07.1
Operating System 10.4
Frontend 20230607.0 - latest

Right away, I have three stuck automations using Zigbee to Z-Wave:

They’re all quite similar to this:

alias: Stairs Day On
description: ""
trigger:
  - platform: state
    to: "on"
    entity_id: binary_sensor.stairs_motion_motion
condition:
  - condition: state
    state: Home
    entity_id: input_select.mode
action:
  - type: turn_on
    device_id: 6f5858284a352c2eee0b93ccd9dac991
    entity_id: light.loft_overhead
    domain: light
    brightness_pct: 100
  - type: turn_on
    device_id: 63f8e9c813765c081ea8a3c6e70ea612
    entity_id: light.loft_lamp_18
    domain: light
    brightness_pct: 100
  - type: turn_on
    device_id: a7d3b9e7ed72b7d33b77190d8638097b
    entity_id: light.stairs_overhead
    domain: light
    brightness_pct: 100
mode: single

Zigbee motion sensor to Z-Wave wall switches.

[parautenbach (Pieter Rautenbach)]

@erik3

If i change to mode: queued the problem goes away instantly, and the automation never locks. If I change back; then I need to restart home assistant after the first press of that button.

Thanks for making the effort to confirm this. This is an important piece of information.

[Lux4rd0]

Still piling up… one after another… Not all at once.

Just before the first stuck automation, I see these in the logs:

2023-08-04 15:44:54.269 WARNING (MainThread) [homeassistant.components.zha.core.cluster_handlers] [0xD1C2:1:0x0001]: async_initialize: all attempts have failed: [DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>'), DeliveryError('Failed to deliver message: <EmberStatus.DELIVERY_FAILED: 102>')]

Followed by a different integration:

2023-08-04 15:45:59.719 WARNING (MainThread) [homeassistant.components.automation.update_tesla_location_as_mqtt_location_updates] Update Tesla location as MQTT location updates: Already running

These are all right after I did the upgrade to 2023.8.1

None of these failures happened before 2023.07

[petro]

Please, can you guys add this information to the issue. It’s been asked multiple times now in the thread and the GitHub issue is bare. Nothing will get fixed if the information stays here.

[Lux4rd0]

Thanks for the nudge @petro - I didn’t see the github request. I added my details.

[123 (Taras)]

@Lux4rd0

I have three stuck automations using Zigbee to Z-Wave

Plus the example you posted has mode: single as opposed to what many have been reporting, namely mode: restart.

single is the default mode for automations and scripts which brings up the question: So why aren’t all automations for all users experiencing this problem?

[Tillsy]

Because as @petro seems to have correctly identified, the bug seems to occur where a device doesn’t respond in an expected timeframe. I have 15 identical copies of same automation but simply with different devices for the different rooms - only one is exhibiting this 2023.7 bug, and it happens to also be the only one that has a device that sometimes doesn’t respond as quickly as the rest of my devices. All of my automations are mode single.

[123 (Taras)]

@Tillsy

only one is exhibiting this 2023.7 bug, and it happens to also be the only one that has a device that sometimes doesn’t respond as quickly as the rest of my devices

If all it takes to cause the problem is a single slow-responding device, based on any integration, then I’m surprised it doesn’t affect far more users.

BTW, EndUser claims to have a solid Zwave network, consisting of dozens of devices, yet provided an example of a very simple automation (turns on 2 switches) that has the problem. Not sure how that particular example is explained by the ‘slow reply’ theory. (Unless the Zwave network isn’t as ‘solid’ as was assumed.)

[EndUser]

I updated core to 2023.8.1 and a few hours later same issue again, with in fact nothing working anymore. I posted my logs on github.
I power cycled the Yellow Box and now it seems to be working again.

[Tillsy]

You’ve just described every squeak, fault, recall, or bug - they’re not planned or intentional, they’re a malfunction, so not always predictable.

In this case not yet fully understood so hard to make judgement but it seems a few people have knowledge of a specific change and circumstance that this seems to align with.

My Z-Wave network is beyond solid - it’s phenomenal. Whole of downstairs, whole of upstairs, even my letterbox responds at start of property, and even my chicken coop responds at very end of property. Why just one sensor is a bit quirky I’ve no idea - it works fine 90% of the time, 10% it has a bit of a delay. I figure it’s because of three walls the signal is passing through, dunno haven’t felt the need to halt my life and investigate as it works well enough. Until this bug seemed to get fussy over it. They’ll work it out

[123 (Taras)]

@Tillsy

My Z-Wave network is beyond solid - it’s phenomenal.
…
Why just one sensor is a bit quirky I’ve no idea - it works fine 90% of the time, 10% it has a bit of a delay.

Seems like a contradiction to refer to a network as ‘solid’ and ‘phenominal’ when it has one or more devices known to be less than 100% reliable. Anyway, based on other reports, it doesn’t seem to matter how ‘solid’ the Zwave (or Zigbee ) network is claimed to be, they still experience the problem.

@Tillsy

In this case not yet fully understood so hard to make judgement

If you review the three open Issues reporting this problem, there’s still only a limited amount of information that has been presented. It’s up to users experiencing the problem to flesh out the details to help the developers solve it.

[EndUser]

@123

If you review the three open Issues reporting this problem, there’s still only a limited amount of information that has been presented. It’s up to users experiencing the problem to flesh out the details to help the developers solve it.

I posted the full log on Github and I hope that more experienced HA users can find the cause.

[123 (Taras)]

I looked at your first log file.

Are you aware that your system is plagued with numerous errors that are unrelated to the reported problem? Not merely warning messages but outright errors due to problems with integrations, templates, etc.

The error relevant to the reported problem is this one:

zwave_js_server.exceptions.FailedZWaveCommand: Z-Wave error 200: Timeout while waiting for a callback from the controller (ZW0200)

[EndUser]

Thanks !!!
I was able to understand some of these errors and either corrected or deleted the automations. Others are more complicated and I would love some assistance.

With regards to Z-Wave error 200, is that specific to my setup or is it a bug ?

[123 (Taras)]

@EndUser

I am not sufficiently proficient to understand these errors and would love some guidance !!!

Create a separate topic requesting guidance.

@EndUser

Z-Wave error 200, is that specific to my setup or is it a bug ?

It’s your system complaining about a problem it encountered while communicating with the Zwave integration. I haven’t seen other users logs so I can’t confirm if others have it but I suspect you’re not alone.

The error message supports the theory that an automation ‘hangs’ when it communicates with an integration that doesn’t reply promptly. There may actually be more to it than that but so far that’s the leading clue.