Help understanding ZHA quirks

Configuration Zigbee
#1

I have quite a few danfoss ally thermostats which I am happy with. I know it exposes some attributes which are not visible in home assistant e.g. “load”
I can read this attrribute to an input number using zha toolkit

    service: zha_toolkit.execute
    data:
      command: attr_read
      ieee: climate.spise_thermostat
      cluster: 513
      attribute: 16458
      state_id: input_number.load_spise
      allow_create: false

However when I look in the “quirk” for danfoss ally (zha-device-handlers/thermostat.py at 5272b4c3e180f1c0c43c2a412c8428c12549e772 · zigpy/zha-device-handlers · GitHub) the following is defined

    attributes = Thermostat.attributes.copy()
    attributes.update(
        {
            0x4000: ("etrv_open_windows_detection", t.enum8, True),
            0x4003: ("external_open_windows_detected", t.Bool, True),
            0x4010: ("exercise_day_of_week", t.enum8, True),
            0x4011: ("exercise_trigger_time", t.uint16_t, True),
            0x4012: ("mounting_mode_active", t.Bool, True),
            0x4013: ("mounting_mode_control", t.Bool, True),
            0x4014: ("orientation", t.Bool, True),
            0x4015: ("external_measured_room_sensor", t.int16s, True),
            0x4016: ("radiator_covered", t.Bool, True),
            0x4020: ("control_algorithm_scale_factor", t.uint8_t, True),
            0x4030: ("heat_available", t.Bool, True),
            0x4031: ("heat_supply_request", t.Bool, True),
            0x4032: ("load_balancing_enable", t.Bool, True),
            0x4040: ("load_radiator_room_mean", t.uint16_t, True),
            0x404A: ("load_estimate_radiator", t.uint16_t, True),
            0x404B: ("regulation_setPoint_offset", t.int8s, True),
            0x404C: ("adaptation_run_control", t.enum8, True),
            0x404D: ("adaptation_run_status", t.bitmap8, True),
            0x404E: ("adaptation_run_settings", t.bitmap8, True),
            0x404F: ("preheat_status", t.Bool, True),
            0x4050: ("preheat_time", t.uint32_t, True),
            0x4051: ("window_open_feature_on_off", t.Bool, True),
            0xFFFD: ("cluster_revision", t.uint16_t, True),
        }
    )

Why are these not available when opening the device? In other words what is needed in order for these to be shown in device?

1 Like
#2

In this case the quirk defines the attributes that are not standard, and in a particular case it will also send a command after the setpoint temperature is updated when writing that attribute.

So nothing really happens with the attribiutes.

You can look at this other quirk and see that some attribute updates are “forwarded” to the event bus: zha-device-handlers/ts0601_trv.py at 5272b4c3e180f1c0c43c2a412c8428c12549e772 · zigpy/zha-device-handlers · GitHub . So still not set as entity values.

In fact, the state are handled in ZHA, for instance in light.py which calls on async_write_ha_state which essentlially relies on core/entity.py at 398c7be85046c74574bad266c2129d2de49340f6 · home-assistant/core · GitHub .
(Light.py: core/light.py at dc53a39b0ffcaf8da10b99df5161aa1d539f27c2 · home-assistant/core · GitHub ) .

I do not know what the intended method is by developers of HA and ZHA, but it seems that in the general case one needs to listen for zha_events and track the messages related to the attribute that you are interested in, and then act on that.
That means that you would need to be capable (skills + device functionnality) to setup attribute reporting for the attribute that you are interested in.

Or, if you are in luck, or update the zha-quirk, you can listen to specific events provided by a quirk.

I added the solution that you are using in zha-tool exactly because of all of this:

  • It was unclear how quirks help me to get the state in an attribute;
  • Adding events to quirks, listening and setting up the code to get the date from the event into a state is still unclear (I did not really search how);
  • As I have my hands on zha-toolkit, it was much easier to extend the zha-toolkit read method to put the value directly into a state, which itself has interesting side effects in ZHA (last update time, change detection for automations, history, etc). Pulling is not as efficient as zigbee reports, but so much easier to do in HA at this time.

A nice feature would be to be able to add entities/sensors using the UI by configuring zha-toolkit through entity/endpoint/cluster/attr_id to state (attribute) mapping. And have zha-toolkit listen to the zha_events, see if it matches one of the attributes and update the state (attribute). But I do not have time for that.

1 Like
#3

Regarding extending UI features you might also find the following discussion between ZHA developers:

https://github.com/zigpy/zigpy/discussions/934

The idea discussed there is about exposing more configuration attributes by default in ZHA’s UI.

#4

It is this type of thing more than anything else that caused me to move mostly to zigbee2mqtt.

Once I finally tried it, I was surprised how many more attributes and cofig options z2m exposed as entities.

I don’t know about this particular device though.

It’s an odd dichotomy, ZHA is easier to install and get started with, but Z2M is easier for day to day usage.

#5

I think i will have to either switch to z2m again or wait for the developers to expand zha.

For some reason when I last tried z2m my danfoss ally thermostats would randomly get “out of sync” with HA in the sense that I could adjust them using the z2m web interface but they could not be updated from my lovelace dashboard and would not be updated from z2m.

#6

ZHA also has the drawback of not running remotely. Many of us run HA on virtualized servers in our basements, and prefer to keep the zigbee and zwave radios on PI’s that can be placed more optimally for coverage in the home.

I have been relatively pleased with deconz, but I went that approach because I was coming from Homeseer where I used deconz already, and I didn’t have to re-pair any devices. If I was building from scratch I would probably go with zigbee2mqtt, as it has better device support.

I confess I don’t understand why the HA folks are reinventing the wheel with ZHA - there is already great zigbee support with deconz and z2m that have greater functionality and device support.

1 Like
#7

FYI, ZHA devs are working on a websocket server + addon solution for that (similar to Z-Wave JS), see:

https://github.com/zigpy/zha-websocket-server

and

https://github.com/zigpy/zhaws-addon

There are also other workarounds for that:

  • Using either very long USB extension cables (like USB over Ethernet cable adapters),
  • Using a serial server like ser2net on ex. a RaspBerry Pi Zero (with socat on recieving end).
  • Then there are also finished network-attatched Zigbee adapters like “Tube’s Zigbee Gateways” and “ZigStar LAN Gateway”.

You have to consider that Home Assistant core developers and founders will have other concerns for the long-term sustainability of their code-base for native integration without need for third-party application.

But that is thinking from an end-user perspective and from Home Assistant core developers perspective.

For starters, none of the other Zigbee solutions integrates natively with Home Assistant as ZHA does.

Home Assistant founders (and Nabu Casa founders) have also mentioned that they may want to try to get both Z-Wave and Zigbee certifications, and for Zigbee that certification has to be for the whole Zigbee solution, meaning that Home Assistant can only be do so for nativly integrated solutions like ZHA, (sure you could also cerify Zigbee2MQTT alone as a stand-alone solution without Home Assistant but that would not make “Home Assistant” Zigbee certfied, just like using a Philips Hue Bridge).

ZHA also depends on zigpy libraires which is also used as a dependecy by other open source home automation software (like Domoticz and Jeedom) for their native integrations, so can reuse that code.

zigpy recent development also shows they are aiming for a more purist Zigbee implementation approach that will better comply with the official Zigbee specifications, meaning it should in theory get better out-of-the-box compatibility support in the future, meaning greater change to get Zigbee certificated with it.

As I undersatand it, Zigbee2MQTT is not purist Zigbee implementation and converters are needed for each and every device. The reason, why it supports more devices, is that the project has existed longer and has more developers and a larger community that help make converters for each and every device.

deCONZ/Phoscon is not open-source so that is really relative to a free and open source projekt like HA. You can almost compare that using a propriatory Philips Hue Bridge which is still also very popula.

#8

It’s tough to try and build a new system for zigbee. Many popular devices are not very standards compliant, and therefore will probably all need tweaking, ala z2m. But users won’t move until there support and functionality is there (z2m has more functionality for many devices that are actually supported in ZHA, but need the tweaks to provide that functionality).

Perhaps if folks could write code that translates all the tweaks from z2m to ZHA and fix the compatibility issue in one feel swoop?

Lots of zigbee vendors cut corners on compatibility, so there will always be a ton of tweaks required, at least if you want all the functionality a device provides. That’s just the swamp that’s there, and you will not build a competitive alternative to z2m and deconz if you think you can avoid all that tweaking.

Agree about deconz, as I said I would not have picked that out the gate in a fresh configuiration.

#9

I did look at the remote support being put in to ZHA - that’s a good step forward! I’m not just I want to try that yet, but it’s a good sign.

On the various serial remoting technology, I am running HA in a VM and so running HAOS. Is there a standard integration or addon that can support the remoting of the zigbee stick without needing to monkey with the OS? I am normally fine with making linux changes, but want to be careful with the HA system.

#10

If you assume that the majority of devices are of common device types (like relays, buttons, motion sensors, binary sensors, and temperature) then those should really not need quirks/converters as they should use Zigbee standard clusters and attributes, but yes less common types of devices will probably always need quirks if they are made by manufacturers that do not follow Zigbee specifications.

Though take note that ZHA has relatively recently seemed major updates to make it better comply with the official Zigbee specifications (Zigbee Cluster Library R7 and R8 specifically), and I have also read that developers have also very recently started work on exposing more attribute configurations by default (similar to Zigbee2MQTT) has which should, in theory, make compatible with more devices out-of-the-box as long as the manufacturers is not using wrong clusters, attributes, and/or values.

You can read moe on that developer discussion here:

and

only need quirks when manufacturers deviate from the standard and using wrong clusters, attributes, or values:

That could probably be done by an interested developer, though it would probably not be a good idea to try to make it fully automatic and instead maybe as a conversion script which makes translating converters from zigbee-herdsman-converters into quirks for zha-device-handlers.

See example:

https://github.com/zigpy/zha-device-handlers/pull/1487

https://www.home-assistant.io/integrations/zha#zha-exception-and-deviation-handling

Which came first, chicken or the egg, users or developers. Regardless I do not think that you should talk down disregard ZHA as an option. Also consider that competition is good and breeds innovation.

Home Assistant analytics statistics show that almost 16% of the userbase is using ZHA, and hopefully, as Home Assistant userbase grows over time with more users will also mean more developers join too.

Even an infamously vocal naysayer of Home Assistant and Zigbee was converted after trying ZHA (though looking at his video he thinks that it the ConBee Zigbee adapter that does all the magic :stuck_out_tongue: ):

Before demanding too much, please always remember that ZHA + its dependencies like zigpy and quirks (ZHA Device Handlers) is currently only developed by volunteers in their spare time as an unpaid hobby.

#11

Yes both ZHA and Zigbee2MQTT support ser2net (serial to network proxy server) type remote Zigbee adapter via socat (socket cat) out-of-the-box by pointing the application (client) to the server.

If you have already a pre-made network-attached Zigbee Coordinator LAN adapter or setup your own serial proxy server the “server-side” with a Zigbee Coordinaor USB adapter athen you basically only need to replace the local serial port device path with the IP-address and TCP-port in the ZHA or Zigbee2MQTT application configuration on the “client-side”.

e.i. tcp://192.168.1.99:20108 for Z2M or socket://tcp://192.168.1.99:20108 for ZHA.

https://www.zigbee2mqtt.io/advanced/remote-adapter/connect_to_a_remote_adapter.html

https://www.home-assistant.io/integrations/zha#zigate-or-sonoff-zbbridge-devices

ZigStar LAN Gateway and Tube’s Zigbee Gateways come pre-configured but relatively not inexpensive.

https://community.home-assistant.io/t/zigstar-zigbee-coordinators-and-routers/338586

https://community.home-assistant.io/t/tubes-zb-coordinators-and-routers-was-zigbee-router-on-steroids/280896

DIY approaches include either making own Zigbee serial stream server with ser2net on a Raspberry Pi or hack a hackable Zigbee gateway, neither of which are as complicated as it sounds as there’s guides:

https://community.home-assistant.io/t/zb-gw03-ewelink-ethernet-zigbee-gateway-now-hacked-with-tasmota-and-esphome-so-can-be-used-via-mqtt-or-as-a-remote-zigbee-lan-adapter-with-home-assistants-zha-integration-or-zigbee2mqtt/341223/

https://github.com/AlexxIT/XiaomiGateway3#zigbee2mqtt-mode

https://paulbanks.org/projects/lidl-zigbee/

https://github.com/zigpy/zigpy/discussions/584

https://github.com/tube0013/tube_gateways

https://github.com/MattWestb/IKEA-TRADFRI-ICC-A-1-Module

#12

This is all very helpful. Thank you!

I saw Paul’s video too, and he makes the case that the companies who make all these low cost zigbee gadgets don’t want them to be interoperable. I completely agree with that, which is why zigbee is a lot more like IR remotes (every one having a different IR code) than a a simple interoperable standard. Everything basically needs some customization to use all the functionality because the vendors don’t want to be commoditized. That means anyone doing a new zigbee driver needs to be prepared for endless amounts of customization (quirks) to get full functionality out of these products.

Z2M currently is the gold standard for support, which is why I was looking at that rather than ZHA. What is not clear is how fast ZHA is coming up to parity. Aside from the blakaddr site, there aren’t a lot of explicit comparisons being made, and there is certainly no tracker that is tracking change rates to tell how fast it’s converging with where Z2M is. That would be very useful I think for both devs and users.

Surely it’s good for ZHA to be more compliant with standards, but the problem is that doesn’t buy you a lot given the issues. And ZHA today still lags behind in functionality even for well supported devices like the aqara magic cubes. I also don’t think it’s maps as as useful as Z2M’s, but I’m sure this will get better over time. As they used to say at NASA, with enough thrust even a brick can fly.

I’m not trying to trash the ZHA devs at all, but that’s a lot of work that could have been spent on making Z2M even better. And Zwave still needs work, esp the Zwavejs2mqtt whose map and diagnostics are often just plain wrong in my experience.

#13

This us super useful! Thanks for pointing this out!!

The documentation talks about using socket: for a number of specific devices, but doesn’t mention ser2net compatibility or the ability to use it generically with all adapters remotely using socat! It would be nice is this was more explicit, as it would enable us virtualized HA users an easy path to avoid USB device mapping in the VM’s.

At least for vmware, if you passthrough a USB device to a VM, you break the ability to migrate it or do high availability failover.

Again, thanks for making ser2net support clear!

#14

But that is not how any free and open-source projects work or at least not how the minds of their volunteering developers work.Other than the few hired by Nabu Casa they are not paid employees.

This is their hobby which they do for fun, so you can not really ask a volunteering developer to stop spending their spare time improving the code of ZHA/zigpy and instead start spending time coding Zigbee2MQTT/zigbee-herdsman.

That to be compared would be like asking a person to stop playing tennis in their spare time and instead start playing football instead just because it is the most popular sport in the world.

Different people will find different things interesting and challenging, so while some individuals might personally choose to develop a larger project because it is practical and find it fun to collaborate with others some other individuals will find it more challenging to code alone or on a smaller project.

If you disregard the fact that these projects are all coded in different programming languages which alone might cater to different skills, the personal reason why individuals like different things and choose to spend their free time differently is why there are several popular home automation projects, including Home Assistant, OpenHab, Domoticz, and Jeedom, as well many commercial products which are customizable enough to also have code development for customizations and extensions in their communities too (such as example, SmartThings, Vera, and Homey).

If your coding skills are not good then you can always help with project’s documentation and guides :wink:

https://github.com/home-assistant/home-assistant.io

https://github.com/Koenkk/zigbee2mqtt.io

#15

Btw, for multi-button/function remotes like Aqara/Xiaomi Magic Cube you should really use Blueprints.

https://www.home-assistant.io/blog/2020/12/13/release-202012/#blueprints

https://www.home-assistant.io/docs/automation/using_blueprints/

The same really goes for any complex devices that feature more than lights control or basic sensors, and that goes not only for ZHA but also for Zigbee2MQTT, deCONZ/Phoscon, and Z-Wave JS as well.

https://community.home-assistant.io/c/blueprints-exchange/53

For the Aqara Magic Cube when specifically using ZHA check out these Blueprints:

https://community.home-assistant.io/t/zha-aqara-magic-cube-57-actions/297012

https://community.home-assistant.io/t/aqara-magic-cube-zha-51-actions/270829

https://community.home-assistant.io/t/aqara-cube-zha-simplified-face-based-device-control/388850

For the Aqara Magic Cube when specifically using Zigbee2MQTT check out these Blueprints:

https://community.home-assistant.io/t/zigbee2mqtt-xiaomi-cube-controller/393203

https://community.home-assistant.io/t/z2m-xiaomi-cube-controller/263006

For the Aqara Magic Cube when specifically using deCONZ/Phoscon check out these Blueprints:

https://community.home-assistant.io/t/mi-magic-cube-deconz-45-actions/256656

https://community.home-assistant.io/t/deconz-xiaomi-aqara-mfkzq01lm-cube-controller/255988

Here is a video by Jeff Geerling which clearly show experience before and after discovering Blueprints:

https://www.youtube.com/watch?v=gJFsZL5CTgM&ab_channel=JeffGeerling

PS: The real major difference between Zigbee based proprietary commercial gateways and an open-source gateway like ZHA and Zigbee2MQTT is that Zigbee based proprietary commercial gateways will come with preconfigured from that company with both custom device handlers as well as blueprint automations for the specific devices that they officially support as those have been specifically tested for full compatibility with that Zigbee gateway and its dedicated application, (which unfortunately usually is only their own brand of devices). So the developers of Zigbee based proprietary commercial gateways are normally trying to make their product interoperable with Zigbee devices from third-parties.

2 Likes
Smoke Sensor Heiman HS1SA Zigbee Control/Configuration settings
#16

I haven’t read all the above in detail, but here are my two cents.

I would split up Zigbee compatibility in two main groups: “zigbee management” (“low level”) and Zigbee clusters (+profiles, etc).

Low Level

  • Devices that pass certification are not necessarily compliant because the certification is limited to certain types of tests;
  • The Zigbee stacks from the manufacturers are not tested a lot for interoperability. Some stacks seem to be more permissive than others or cope better with deviations. (To compare with bluetooth: theu have “UnPlugFests” aimed at testing interoperability.
  • Low Level interoperability tends to improve - IMHO because zigbee is actually used.
  • Low Level interoperability is determined essentially by the stack from the IC manufacturer, and in part by how good it is configured by the developer.

Zigbee Clusters

  • Interoperability may be limited because the manufacturer wishes a closed system, but in my experience it is more that they are more than happy when it works with their system and do not want to spend more on making sure it works elsewhere.
    Another reason is that developers do not understand the zigbee specification, are resistant to changing an existing implementation (admitting they are wrong, spend more time than planned, …). I remember though discussions with pairs.
    • One was about a timestamp. The zigbee specification “clearly” says the reference (epoch) is 1/1/2000 but the implementation uses the more classic 1/1/1970. So all dates are by about 30 year wrong. It is still so in the end product for the following reasons forwarded by the subcontractor: this feature was not requested by the customer and they do not agree with my interpretation of the specification.
    • Too many developers tend to implement features in custom clusters or attributes even though the equivalent feature exists in the standard Zigbee Cluster Library specification.
    • Testing is mainly done against the internal coordinator or a single selected coordinator (cost reasons related to time) and improved only when there is a market need (a customer using another coordinator wants to integrate your end device).
    • The market is driven by (B2B agreements of) selling systems, not by B2C sales of end devices.
    • Some companies will deviate from the specification improving the user experience with their coordinator by fixing certain values in the end devices. In one instance I found that an end device can only report to endpoint 1. And I still believe this is more a choice of the developer than a choice of the company.
1 Like
Smoke Sensor Heiman HS1SA Zigbee Control/Configuration settings
#17

I have been using ZHA for a while, and it has worked reliably for mains powered devices. When using it for battery powered devices like thermometer/hygrometers and motion detectors, they seems to unpair after a few hours or days and show as unavailable. What are the first things I should be checking?

#18

I think first things should be to upgrade Zigbee Coordinator firmware and follow these best practices:

https://github.com/zigpy/zigpy/wiki/Generic-best-practice-tips-on-improving-Zigbee-network-range-and-general-stability

https://www.home-assistant.io/integrations/zha#using-router-devices

https://www.home-assistant.io/integrations/zha#best-practices-to-avoid-pairingconnection-difficulties

If still have issues after that you need to enable debug logging for used ZHA libraries and check the debug logs after replicating the issue, and as noted in the last of those best practice guidelines linked above; there can some Zigbee devices that have poorly written firmware from the manufacturer and are therefore not fully compatible with all other brands of Zigbee router devices as they should be. See:

  • Check that you have enough Zigbee router devices (also known as Zigbee signal repeaters or range extenders) and if you do not have any, invest and add some mains-powered devices that will work as Zigbee routers.
    • Aim to start out with mains-powered devices before adding battery-operated devices as a “weak” Zigbee network mesh (e.g., the device is too far from the Zigbee coordinator or a Zigbee router) may prevent some devices from being paired. Zigbee router devices are also needed to increase the maximum of devices that can be connected to your Zigbee mesh network.
    • Note that some Zigbee devices are not fully compatible with all brands of Zigbee router devices. Xiaomi/Aqara devices are for example known not to work with Zigbee router devices from Centralite, General Electrics, Iris, Ledvance/OSRAM, LIGHTIFY/Sylvania, Orvibo, PEQ, Securifi, and SmartThings/Samsung. Better results can usually be achieved by using mains-powered devices IKEA and Nue/3A Home or dedicated DIY routing devices based on Texas Instruments CC253x/CC26x2 and XBee Series 2/3 Zigbee radios.
1 Like