ZHA: retry aborted firmware updates, one device at a time

Open your Home Assistant instance and show the blueprint import dialog with a specific blueprint pre-filled.

Zigbee firmware updates in ZHA often abort on an unstable link. This blueprint runs the waiting updates one device at a time and starts the same update again after an abort. It does not make the radio more reliable, and it does not fix an image the device rejects (for example INVALID_IMAGE after a completed transfer).

Create three helpers first:

  • an input_boolean, the kill switch
  • a timer, the watchdog
  • a second timer, the cooldown after a ZHA reload

Turn the kill switch on to start. Turn it off to stop further attempts. A transfer that is already running is left alone.

After an abort it waits 30 seconds and tries the same device again. After a success, or after you turn the kill switch on, it waits 15 seconds and starts the next device. If a start never reaches “installing”, the watchdog retries every 150 seconds. The pending list rotates on that same period, so one device that never starts does not block the others.

“Reload ZHA when an update hangs” is off by default. When you enable it, an update that stays in progress for 6 minutes without an attribute change reloads ZHA, waits 90 seconds, and tries again. That disconnects the whole Zigbee network for a short time, and it will not happen again for 15 minutes.

Gist: ZHA OTA serial retry · GitHub

blueprint:
  name: ZHA OTA serial retry
  author: KienzlerA
  domain: automation
  description: >
    Runs waiting Zigbee firmware updates one device at a time and starts the
    same update again after an abort. Create three helpers before you import
    an automation from this blueprint. An input_boolean is the kill switch.
    One timer is the watchdog. A second timer is the cooldown after a ZHA
    reload.


    Turn the kill switch on to start. Turn it off to stop further attempts.
    A transfer that is already running is left alone.


    Timing, as used for a Sonoff S60 campaign: 30 seconds after an abort,
    15 seconds after success or after the kill switch turns on, then the
    next device. If a start never reaches "installing", the watchdog retries
    every 150 seconds. The pending list rotates on that same 150-second
    period, so one device that never starts does not block the others.


    "Reload ZHA when an update hangs" is off by default. When enabled, an
    update that stays in progress for 6 minutes without an attribute change
    reloads ZHA, waits 90 seconds, and tries again. Reloading ZHA disconnects
    the whole Zigbee network for a short time, and it will not happen again
    for 15 minutes. Leave this off unless updates sit in progress forever.


    Retrying does not fix an image the device rejects (for example
    INVALID_IMAGE after a completed transfer).

  input:
    firmware_updates:
      name: Firmware updates
      description: >
        Update entities to process one after another. State "on" means an
        update is still available.
      selector:
        entity:
          multiple: true
          filter:
            - domain: update
              device_class: firmware
    kill_switch:
      name: Kill switch
      description: >
        Turn on to start the campaign. Turn off to stop further retries.
      selector:
        entity:
          filter:
            - domain: input_boolean
    watchdog_timer:
      name: Watchdog timer
      description: >
        Timer helper. The blueprint restarts it for 2 minutes 30 seconds
        after each attempt.
      selector:
        entity:
          filter:
            - domain: timer
    reload_cooldown_timer:
      name: Reload cooldown timer
      description: >
        Second timer helper. Used only when hang reload is enabled. The
        blueprint then starts it for 15 minutes.
      selector:
        entity:
          filter:
            - domain: timer
    reload_zha_on_hang:
      name: Reload ZHA when an update hangs
      description: >
        Reload the ZHA integration when an update stays in progress for 6
        minutes without an attribute change. This briefly disconnects every
        Zigbee device.
      default: false
      selector:
        boolean:
    zha_config_entry:
      name: ZHA configuration entry
      description: >
        Used only when hang reload is enabled. Pick the Zigbee Home
        Automation entry.
      selector:
        config_entry:
          integration: zha

mode: queued
max: 10

variables:
  ota_entities_input: !input firmware_updates
  ota_entities: >-
    {% if ota_entities_input is string %}
      {{ [ota_entities_input] }}
    {% else %}
      {{ ota_entities_input | list }}
    {% endif %}
  reload_on_hang: !input reload_zha_on_hang

triggers:
  - trigger: state
    entity_id: !input firmware_updates
    from: installing
    to: "on"
    id: abort
  - trigger: state
    entity_id: !input firmware_updates
    from: installing
    to: "off"
    id: success
  - trigger: state
    entity_id: !input kill_switch
    to: "on"
    id: kick
  - trigger: state
    entity_id: !input kill_switch
    to: "off"
    id: stop
  - trigger: event
    event_type: timer.finished
    event_data:
      entity_id: !input watchdog_timer
    id: watchdog

actions:
  - choose:
      - conditions:
          - condition: trigger
            id: stop
        sequence:
          - action: timer.cancel
            target:
              entity_id: !input watchdog_timer
      - conditions:
          - condition: trigger
            id: abort
          - condition: state
            entity_id: !input kill_switch
            state: "on"
        sequence:
          - delay:
              seconds: 30
          - condition: state
            entity_id: !input kill_switch
            state: "on"
          - condition: template
            value_template: "{{ is_state(trigger.entity_id, 'on') }}"
          - condition: template
            value_template: >-
              {{ (ota_entities | select('is_state', 'installing') | list | count == 0)
                 and (ota_entities | map('state_attr', 'in_progress') | select('true') | list | count == 0) }}
          - action: update.install
            continue_on_error: true
            target:
              entity_id: "{{ trigger.entity_id }}"
          - action: timer.start
            target:
              entity_id: !input watchdog_timer
            data:
              duration: "00:02:30"
    default:
      - condition: or
        conditions:
          - condition: trigger
            id: success
          - condition: trigger
            id: kick
          - condition: trigger
            id: watchdog
      - condition: state
        entity_id: !input kill_switch
        state: "on"
      - choose:
          - conditions:
              - condition: or
                conditions:
                  - condition: trigger
                    id: success
                  - condition: trigger
                    id: kick
            sequence:
              - delay:
                  seconds: 15
      - condition: state
        entity_id: !input kill_switch
        state: "on"
      - variables:
          pending_rot: >-
            {% set p = ota_entities | select('is_state', 'on') | list %}
            {% set n = p | count %}
            {% if n == 0 %}
              []
            {% else %}
              {% set i = (now().timestamp() | int // 150) % n %}
              {{ (p[i:] + p[:i]) | list }}
            {% endif %}
          hung_otas: >-
            {% set ns = namespace(items=[]) %}
            {% for e in ota_entities %}
              {% if state_attr(e, 'in_progress') %}
                {% set age = (as_timestamp(now()) - as_timestamp(states[e].last_updated)) | int %}
                {% if age >= 360 %}
                  {% set ns.items = ns.items + [e] %}
                {% endif %}
              {% endif %}
            {% endfor %}
            {{ ns.items }}
      - if:
          - condition: template
            value_template: "{{ reload_on_hang | bool and (hung_otas | count > 0) }}"
        then:
          - if:
              - condition: not
                conditions:
                  - condition: state
                    entity_id: !input reload_cooldown_timer
                    state: active
            then:
              - action: persistent_notification.create
                data:
                  notification_id: zha_ota_hang_kill
                  title: ZHA OTA hang
                  message: >-
                    Firmware update stalled (no progress for 6 minutes):
                    {{ hung_otas | join(', ') }}. Reloading ZHA, then retrying.
              - action: homeassistant.reload_config_entry
                data:
                  entry_id: !input zha_config_entry
              - action: timer.start
                target:
                  entity_id: !input reload_cooldown_timer
                data:
                  duration: "00:15:00"
              - delay:
                  seconds: 90
              - action: timer.start
                target:
                  entity_id: !input watchdog_timer
                data:
                  duration: "00:02:30"
            else:
              - action: timer.start
                target:
                  entity_id: !input watchdog_timer
                data:
                  duration: "00:02:30"
        else:
          - if:
              - condition: template
                value_template: "{{ pending_rot | count == 0 }}"
            then:
              - if:
                  - condition: template
                    value_template: >-
                      {{ (ota_entities | select('is_state', 'installing') | list | count == 0)
                         and (ota_entities | map('state_attr', 'in_progress') | select('true') | list | count == 0) }}
                then:
                  - action: timer.cancel
                    target:
                      entity_id: !input watchdog_timer
                  - action: input_boolean.turn_off
                    target:
                      entity_id: !input kill_switch
                  - action: persistent_notification.create
                    data:
                      notification_id: zha_ota_done
                      title: ZHA OTA finished
                      message: >-
                        Every firmware update in the list has finished.
                        The kill switch was turned off.
                else:
                  - action: timer.start
                    target:
                      entity_id: !input watchdog_timer
                    data:
                      duration: "00:02:30"
            else:
              - repeat:
                  for_each: "{{ pending_rot }}"
                  sequence:
                    - condition: template
                      value_template: >-
                        {{ (ota_entities | select('is_state', 'installing') | list | count == 0)
                           and (ota_entities | map('state_attr', 'in_progress') | select('true') | list | count == 0) }}
                    - action: update.install
                      continue_on_error: true
                      target:
                        entity_id: "{{ repeat.item }}"
                    - delay:
                        seconds: 5
              - action: timer.start
                target:
                  entity_id: !input watchdog_timer
                data:
                  duration: "00:02:30"
1 Like

Giving this a try. I have one temp sensor that has an update that always fails when I try it. Maybe this persistence will get it done.

Thanks for trying it. One sensor that always fails is a useful test.

If it is battery powered, press its button once when an attempt starts so it stays awake long enough to request the image. The blueprint only starts the update again. It cannot keep a sleepy device awake.

Please let me know whether it finished, or where it stopped: it never left “on”, it aborted part way, or it sat in progress. That shows whether a retry is enough for this kind of device.