Zigbee devices that don't like scenes

Interloping · August 7, 2022, 6:24am

I’ve been having some weird problems with a few zigbee devices that just won’t play nice with Home Assistant scenes. One example is this one here: Nue / 3A WL-SD001-9W control via MQTT | Zigbee2MQTT

I have some of these that I originally had paired to my Hue hub. When paired to the Hue hub these lights don’t (seem to) respond when I call the scene.turn_on service. (Maybe they do respond - they may be turning on to a minimum brightness, it’s hard to tell.) If I then call the service immediately a second time, they take the state the scene defines for them.

So … until today I had scripts calling the service twice if these lights were part of the scene. But I’m in the process of trying to move from Hue to zigbee2mqtt with a Conbee II stick. I’ve got a bunch of actual Philips Hue bulbs I’ve transferred and these are behaving as expected when it comes to scenes. However, these Nue fixtures just won’t cooperate at all. I can turn them on and off individually in the UI and set all the colours, brightness or colour temperature just as expected. However, if I apply a scene nothing at all happens, although I do, some time later, get an error like this in the zigbee2mqtt logs:

Zigbee2MQTT:error 2022-08-07 15:25:55: Publish 'set' 'state' to 'Driveway spot 3' failed: 'Error: Command 0xc12f0211e00bfdb0/1 genOnOff.off({}, {"sendWhen":"immediate","timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (no response received)'
Zigbee2MQTT:info  2022-08-07 15:25:55: MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"message":"Publish 'set' 'state' to 'Driveway spot 3' failed: 'Error: Command 0xc12f0211e00bfdb0/1 genOnOff.off({}, {\"sendWhen\":\"immediate\",\"timeout\":10000,\"disableResponse\":false,\"disableRecovery\":false,\"disableDefaultResponse\":false,\"direction\":0,\"srcEndpoint\":null,\"reservedBits\":0,\"manufacturerCode\":null,\"transactionSequenceNumber\":null,\"writeUndiv\":false}) failed (no response received)'","meta":{"friendly_name":"Driveway spot 3"},"type":"zigbee_publish_error"}'

Does anyone else who uses zigbee2mqtt or Nue devices have any ideas? At this stage I’ve had to move all those devices back to my Hue bridge and keep using my hacky scripts.

Interloping · August 13, 2022, 6:14am

This is really bugging me. I can’t fathom that I’m the only person who’s encountered this.

I set debugging in the zigbee2mqtt logging to see what is happening when I turn a light on and off via the UI (which works) compared to via a scene (works to turn lights on, but not off, and not to turn on and set a colour state).

A normal turn on via the UI produces this in the debug log:

Zigbee2MQTT:debug 2022-08-13 15:48:19: Received MQTT message on 'zigbee2mqtt/Driveway spot 2/set' with data '{"state":"ON"}'
Zigbee2MQTT:debug 2022-08-13 15:48:19: Publishing 'set' 'state' to 'Driveway spot 2'
Zigbee2MQTT:info  2022-08-13 15:48:19: MQTT publish: topic 'zigbee2mqtt/Driveway spot 2', payload '{"brightness":254,"color_mode":"color_temp","color_temp":305,"linkquality":255,"state":"ON"}'

In comparison, turning on by applying a scene:

Zigbee2MQTT:debug 2022-08-13 15:49:03: Received MQTT message on 'zigbee2mqtt/Driveway spot 2/set' with data '{"state":"ON","brightness":254,"color_temp":305}'
Zigbee2MQTT:debug 2022-08-13 15:49:03: Publishing 'set' 'brightness' to 'Driveway spot 2'
Zigbee2MQTT:debug 2022-08-13 15:49:03: Publishing 'set' 'color_temp' to 'Driveway spot 2'
Zigbee2MQTT:info  2022-08-13 15:49:03: MQTT publish: topic 'zigbee2mqtt/Driveway spot 2', payload '{"brightness":254,"color_mode":"color_temp","color_temp":305,"linkquality":255,"state":"ON"}'

Okay, there’s some differences there. More detail is being sent by the scene being set. The light turns on as expected but fails to apply the color_temp or brightness.

Now turning off via the UI:

Zigbee2MQTT:debug 2022-08-13 15:48:41: Received MQTT message on 'zigbee2mqtt/Driveway spot 2/set' with data '{"state":"OFF"}'
Zigbee2MQTT:debug 2022-08-13 15:48:41: Publishing 'set' 'state' to 'Driveway spot 2'
Zigbee2MQTT:info  2022-08-13 15:48:41: MQTT publish: topic 'zigbee2mqtt/Driveway spot 2', payload '{"brightness":254,"color_mode":"color_temp","color_temp":305,"linkquality":255,"state":"OFF"}'

Compared to turning off by applying a scene that sets that light as off:

Zigbee2MQTT:debug 2022-08-13 15:49:21: Received MQTT message on 'zigbee2mqtt/Driveway spot 2/set' with data '{"state":"OFF"}'
Zigbee2MQTT:debug 2022-08-13 15:49:21: Publishing 'set' 'state' to 'Driveway spot 2'
Zigbee2MQTT:info  2022-08-13 15:49:21: MQTT publish: topic 'zigbee2mqtt/Driveway spot 2', payload '{"brightness":254,"color_mode":"color_temp","color_temp":305,"linkquality":255,"state":"OFF"}'

If I’m not blind that’s identical. BUT when the turn off is issued via the UI it turns off, whereas if it’s issued by the scene being set, it stubbornly stays on (but HA sees it as now off). Applying the scene again makes no difference.

One caveat: If I turn on the light by applying a scene, the next attempt to turn it off via the UI will fail. The UI will report that it went off, but the light stays on. Cycling it on and off again via the UI is necessary to turn it off.

What is going on here!?

Interloping · August 19, 2022, 9:40pm

With further testing it appears this is a zigbee2mqtt issue not a Home Assistant issue. I’ve tested using the zigbee2mqtt built-in scenes functionality and am getting the same problems as when trying to use scenes via home assistant. I am left curious, though, about what is different in how home assistant is issuing commands between a straightforward on/off via the UI and when it is done through a scene to get a different outcome.

Mortalitas · January 4, 2023, 11:51am

Sorry about the full on information dump.
TLDR: The lights firmware is faulty, I’m slowed down by testing and need help, the manufacturer’s level of support is bad.

I tested 4 Nue / 3A WL-SD001-9W and then topped up to 28 lights for my house only to realise that my testing was nowhere near good enough and that these lights have way too many problems.
Are you sure that your issues are related to Z2M or scenes?

You are absolutely not the only person. The amount of money and time I’ve wasted on these lights is really upsetting to me. I’ve been emailing the manufacturer since early October, we’re at 29 emails back and forth, and he keeps passing blame to Home Assistant and whatever else I’ve tested with instead of acknowledging the problems with his lights. In his defence, I didn’t have the network traffic until now and I’m still trying to write that up in a clear way because I think I only have 1 shot at sending it.

Z2M debug logs can only tell you so much. I got the network traffic using Wireshark and an nRF52840 flashed as a sniffer (I think I followed this). This way nobody can blame HA, Z2M, Hue or anything else because I can literally see what is on the network, and in most cases, I can see the light’s response message where it acknowledges the command it received even though it’s state didn’t change.

This is related to attaching attributes to the turn on command. e.g. using Move to Level with OnOff (0x04) instead of On (0x01).
The light receives the command, acknowledges it with a success (0x00) status and then stays on no matter how many times you try to send the Off (0x00) or Fade to Off (0x40/0x00/0x00) commands. The workaround is to send the on command again, and then off. Or toggle the physical switch. But that’s a horrible and stupid workaround that makes any form of automation impossible. Especially since the lights report themselves as off and still acknowledge the off commands with a success status.

I’ve also been able to break the lights in the same way using colour instead of brightness, but I’ve not captured the data for it yet.

This is related to how the Hue Bridge turns the lights off. It uses a fading effect over 0.8s instead of an instantaneous off command. These lights advertise that they work with Philips Hue, but I would argue that they absolutely do not because the turn off command Hue uses breaks the lights.
I’ve made the lights come back on with 1% brightness when it should have remained at 100% since brightness wasn’t changed, and also 100% when it should have remained at 1%. I’m still testing to separate the two outcomes.
I’ve also made the lights turn on with 1% brightness, just by telling them to turn on with some brightness level or colour other than what it was previously set to. But that’s a different issue.

Instantaneous Off

Command: Off (0x00)

Fade to Off

Command: Off with effect (0x40)
    Payload
        Effect Identifier: Delayed All Off (0x00)
        Effect Variant: Fade to off in 0.8 seconds (0x00)

Also, transition effects don’t work but a lot of these problems seem to be related which might mean there are less things to fix. I’m still testing, and given how much this has already cost me, I’m determined to keep pushing until I get a positive result. My biggest concern is that even with the network traffic, the manufacturer may still pass the blame or tell me to send the commands twice. But I’m really hoping that doesn’t keep happening.

The lights also don’t retain their power on behaviour setting when configured from Z2M, but they do when using the Hue Bridge and app. I think this might be a Z2M issue though.

The main thing slowing down my testing is that it takes so long to mess around with the Z2M dev console. I need a way to attach zigbee cluster commands to a button on a dashboard card, or node red, or something. I’ve asked for help here:

HA Forum: Automated way to send Zigbee cluster command (Zigbee2MQTT)
node-red-contrib-zigbee2mqtt: [Feature Request] Send manually constructed commands using Out node (or new node) #105
zigbee2mqtt: How do you send the “Off with effect (0x40)” command and payload to a Light? #15765
Z2M Discord channel

My backup plan was to replace the processor PCB inside these lights (I’m an electronics engineer) and program it myself, but I really hope to avoid that given how expensive and unreliable that could be.
I’ve already tried finding documentation for the existing processor to reprogram it, but that’s hit a dead end.

This is what’s inside the lights (digital & RF section)
This is my attempt at finding documentation (and I also emailed the IC manufacturer with no luck).

Interloping · January 4, 2023, 8:20pm

Hi, that’s an incredible amount of work you have done! I actually had a bit of a breakthrough of sorts where the performance of my lights improved enough to be vaguely usable with some hacks. I discovered that if I set a transition value of 0.1 in the settings for each of these lights they stopped getting stuck in previous states. Different or no transition values both seemed to cause much more performance issues. Applying scenes now works, though I still need to apply them twice to be sure all 4 lights take on the intended state. I’d be curious if you have played with this transition setting (in “settings (specific)” or by putting a transition value in the config file) and whether you find a 0.1 value impacts those behaviours you’ve observed.

Mortalitas · January 6, 2023, 10:53am

Thanks Interloping,

I still consider that fix to be a problem because this isn’t how zigbee lights should behave. But thank you, I think this may help narrow one or two of the issues down a bit and it’s definitely worth including with my data when I finally send it to them.

I’ve redone the test where we can get the light stuck in the on state, but with 0.1s set as the transition time. But I haven’t had a chance to rerun other tests with this change yet.

Test Steps:

Start light at 100% brightness.
Turn light off.
Turn light on with a 20% brightness.

Test Variables

Transition time
Where transition time is set.

No Transition Time (default)

Result:
The light turned on with 20% brightness, but got stuck and can’t turn off.
Transition time too fast to check.

Coordinator Message - Turn On

ZigBee Cluster Library Frame
    Frame Control Field: Cluster-specific (0x01)
        .... ..01 = Frame Type: Cluster-specific (0x1)
        .... .0.. = Manufacturer Specific: False
        .... 0... = Direction: Client to Server
        ...0 .... = Disable Default Response: False
    Sequence Number: 58
    Command: Move to Level with OnOff (0x04)
    Payload
        Level: 51
        Transition Time: 0.0 seconds

Light Response

ZigBee Cluster Library Frame, Command: Default Response, Seq: 58
    Frame Control Field: Profile-wide (0x18)
        .... ..00 = Frame Type: Profile-wide (0x0)
        .... .0.. = Manufacturer Specific: False
        .... 1... = Direction: Server to Client
        ...1 .... = Disable Default Response: True
    Sequence Number: 58
    Command: Default Response (0x0b)
    Response to Command: 0x04
    Status: Success (0x00)

Coordinator Message - Turn Off

ZigBee Cluster Library Frame
    Frame Control Field: Cluster-specific (0x01)
        .... ..01 = Frame Type: Cluster-specific (0x1)
        .... .0.. = Manufacturer Specific: False
        .... 0... = Direction: Client to Server
        ...0 .... = Disable Default Response: False
    Sequence Number: 59
    Command: Off (0x00)

Light Response

ZigBee Cluster Library Frame, Command: Default Response, Seq: 59
    Frame Control Field: Profile-wide (0x18)
        .... ..00 = Frame Type: Profile-wide (0x0)
        .... .0.. = Manufacturer Specific: False
        .... 1... = Direction: Server to Client
        ...1 .... = Disable Default Response: True
    Sequence Number: 59
    Command: Default Response (0x0b)
    Response to Command: 0x00
    Status: Success (0x00)

Transition = 0.1 attached to the message alongside brightness

Result:
The light turned on with 20% brightness, but got stuck and can’t turn off.
Transition time too fast to check.

Coordinator Message - Turn On

ZigBee Cluster Library Frame
    Frame Control Field: Cluster-specific (0x01)
        .... ..01 = Frame Type: Cluster-specific (0x1)
        .... .0.. = Manufacturer Specific: False
        .... 0... = Direction: Client to Server
        ...0 .... = Disable Default Response: False
    Sequence Number: 66
    Command: Move to Level with OnOff (0x04)
    Payload
        Level: 51
        Transition Time: 0.1 seconds

Light Response

ZigBee Cluster Library Frame, Command: Default Response, Seq: 66
    Frame Control Field: Profile-wide (0x18)
        .... ..00 = Frame Type: Profile-wide (0x0)
        .... .0.. = Manufacturer Specific: False
        .... 1... = Direction: Server to Client
        ...1 .... = Disable Default Response: True
    Sequence Number: 66
    Command: Default Response (0x0b)
    Response to Command: 0x04
    Status: Success (0x00)

Coordinator Message - Turn Off

ZigBee Cluster Library Frame
    Frame Control Field: Cluster-specific (0x01)
        .... ..01 = Frame Type: Cluster-specific (0x1)
        .... .0.. = Manufacturer Specific: False
        .... 0... = Direction: Client to Server
        ...0 .... = Disable Default Response: False
    Sequence Number: 67
    Command: Off (0x00)

Light Response

ZigBee Cluster Library Frame, Command: Default Response, Seq: 67
    Frame Control Field: Profile-wide (0x18)
        .... ..00 = Frame Type: Profile-wide (0x0)
        .... .0.. = Manufacturer Specific: False
        .... 1... = Direction: Server to Client
        ...1 .... = Disable Default Response: True
    Sequence Number: 67
    Command: Default Response (0x0b)
    Response to Command: 0x00
    Status: Success (0x00)

Transition = 0.1 from Z2M Settings (Specific) Menu

The light turned on with 20% brightness, and then turned off.
Transition time too fast to check.

Coordinator Message - Turn On

ZigBee Cluster Library Frame
    Frame Control Field: Cluster-specific (0x01)
        .... ..01 = Frame Type: Cluster-specific (0x1)
        .... .0.. = Manufacturer Specific: False
        .... 0... = Direction: Client to Server
        ...0 .... = Disable Default Response: False
    Sequence Number: 75
    Command: Move to Level with OnOff (0x04)
    Payload
        Level: 51
        Transition Time: 0.1 seconds

Light Response

ZigBee Cluster Library Frame, Command: Default Response, Seq: 75
    Frame Control Field: Profile-wide (0x18)
        .... ..00 = Frame Type: Profile-wide (0x0)
        .... .0.. = Manufacturer Specific: False
        .... 1... = Direction: Server to Client
        ...1 .... = Disable Default Response: True
    Sequence Number: 75
    Command: Default Response (0x0b)
    Response to Command: 0x04
    Status: Success (0x00)

Coordinator Message - Turn Off

ZigBee Cluster Library Frame
    Frame Control Field: Cluster-specific (0x01)
        .... ..01 = Frame Type: Cluster-specific (0x1)
        .... .0.. = Manufacturer Specific: False
        .... 0... = Direction: Client to Server
        ...0 .... = Disable Default Response: False
    Sequence Number: 76
    Command: Move to Level with OnOff (0x04)
    Payload
        Level: 0
        Transition Time: 0.1 seconds

Light Response

ZigBee Cluster Library Frame, Command: Default Response, Seq: 76
    Frame Control Field: Profile-wide (0x18)
        .... ..00 = Frame Type: Profile-wide (0x0)
        .... .0.. = Manufacturer Specific: False
        .... 1... = Direction: Server to Client
        ...1 .... = Disable Default Response: True
    Sequence Number: 76
    Command: Default Response (0x0b)
    Response to Command: 0x04
    Status: Success (0x00)

Transition = 0.1 from Z2M Settings (Specific) Menu but 1.0s attached to the message alongside brightness

Result:
The light turned on with 20% brightness, and then turned off.
The light transitioned instantly. As expected, the transition time that was manually added to the message overwrote the settings menu, but the transition time was ignored.

Coordinator Message - Turn On

ZigBee Cluster Library Frame
    Frame Control Field: Cluster-specific (0x01)
        .... ..01 = Frame Type: Cluster-specific (0x1)
        .... .0.. = Manufacturer Specific: False
        .... 0... = Direction: Client to Server
        ...0 .... = Disable Default Response: False
    Sequence Number: 90
    Command: Move to Level with OnOff (0x04)
    Payload
        Level: 51
        Transition Time: 1.0 seconds

Light Response

ZigBee Cluster Library Frame, Command: Default Response, Seq: 90
    Frame Control Field: Profile-wide (0x18)
        .... ..00 = Frame Type: Profile-wide (0x0)
        .... .0.. = Manufacturer Specific: False
        .... 1... = Direction: Server to Client
        ...1 .... = Disable Default Response: True
    Sequence Number: 90
    Command: Default Response (0x0b)
    Response to Command: 0x04
    Status: Success (0x00)

Coordinator Message - Turn Off

ZigBee Cluster Library Frame
    Frame Control Field: Cluster-specific (0x01)
        .... ..01 = Frame Type: Cluster-specific (0x1)
        .... .0.. = Manufacturer Specific: False
        .... 0... = Direction: Client to Server
        ...0 .... = Disable Default Response: False
    Sequence Number: 91
    Command: Move to Level with OnOff (0x04)
    Payload
        Level: 0
        Transition Time: 0.1 seconds

Light Response

ZigBee Cluster Library Frame, Command: Default Response, Seq: 91
    Frame Control Field: Profile-wide (0x18)
        .... ..00 = Frame Type: Profile-wide (0x0)
        .... .0.. = Manufacturer Specific: False
        .... 1... = Direction: Server to Client
        ...1 .... = Disable Default Response: True
    Sequence Number: 91
    Command: Default Response (0x0b)
    Response to Command: 0x04
    Status: Success (0x00)

Transition = 1.0s from Z2M Settings (Specific) Menu

Result:
The light turned on with 20% brightness, and then turned off.
The light transitioned instantly and ignored the 1.0s setting.

Coordinator Message - Turn On

ZigBee Cluster Library Frame
    Frame Control Field: Cluster-specific (0x01)
        .... ..01 = Frame Type: Cluster-specific (0x1)
        .... .0.. = Manufacturer Specific: False
        .... 0... = Direction: Client to Server
        ...0 .... = Disable Default Response: False
    Sequence Number: 109
    Command: Move to Level with OnOff (0x04)
    Payload
        Level: 51
        Transition Time: 1.0 seconds

Light Response

ZigBee Cluster Library Frame, Command: Default Response, Seq: 109
    Frame Control Field: Profile-wide (0x18)
        .... ..00 = Frame Type: Profile-wide (0x0)
        .... .0.. = Manufacturer Specific: False
        .... 1... = Direction: Server to Client
        ...1 .... = Disable Default Response: True
    Sequence Number: 109
    Command: Default Response (0x0b)
    Response to Command: 0x04
    Status: Success (0x00)

Coordinator Message - Turn Off

ZigBee Cluster Library Frame
    Frame Control Field: Cluster-specific (0x01)
        .... ..01 = Frame Type: Cluster-specific (0x1)
        .... .0.. = Manufacturer Specific: False
        .... 0... = Direction: Client to Server
        ...0 .... = Disable Default Response: False
    Sequence Number: 110
    Command: Move to Level with OnOff (0x04)
    Payload
        Level: 0
        Transition Time: 1.0 seconds

Light Response

ZigBee Cluster Library Frame, Command: Default Response, Seq: 110
    Frame Control Field: Profile-wide (0x18)
        .... ..00 = Frame Type: Profile-wide (0x0)
        .... .0.. = Manufacturer Specific: False
        .... 1... = Direction: Server to Client
        ...1 .... = Disable Default Response: True
    Sequence Number: 110
    Command: Default Response (0x0b)
    Response to Command: 0x04
    Status: Success (0x00)