Problem with ZHA network visualization

drcoolzic · February 9, 2021, 4:43pm

Actually, your problem raises interesting questions. Normally a router is supposed to be always power on in order to listen to broadcast commands that affects the network. I will not go into details, but, under certain circumstances, it seems possible that a router will not be able to rejoin the network if it has been down for some times. Furthermore, it does not seems that the behavior of a router when it restarts is fully defined (e.g. rejoin command not mandatory). So different brand will behave differently.

However your setup is quite common particularly for bulb lights. Personally, I will put some switches on the bulb lines so that if the home automation system is down I can at least turn on my lights.

Philips is definitively a major player in Zigbee and it making very good products, but that does not mean that it provides 100% Zigbee compliant devices. For example, according to @MattWestb some old Philips bulbs (but not so old) do not implement neighbor aging algorithm correctly. Therefore, you can be left with long time gone devices still reported by the router.

You may be tempted to keep Philips devices all grouped under a Hue hub and the rest of your Zigbee devices connected to the ConBee. This will work but by doing so you are removing many routers from your network which seems a bad idea as it seems that the more routers you have the more robust your network is (especially to interference).

I just power down few hours ago one of my Hue bulb router with an Aqara end device directly connected to it. I will wait until tomorrow to see what is the behavior of the bulb when I turn it back on. Normally it is possible to specify the behavior of the bulb when the power comes back (on, off, same state). This is easy with the Hue application but hard with ZHA.

One thing I do not like in HA is the fact that when a Zigbee device is gone (or down) there does not seem to be any indication in HA. For the Philips bulb (down) if I try to turn on the bulb with an HA button after a few seconds the button goes back to off state without any indication (error message). I suppose the command is sent but obviously not received by the device and HA must try to read the status back and not receive anything. In such a case, it would be nice to “grey” the device to indicate that it can’t be reached. oops it works as it should - just have to wait that it gets detected as not available

Note that the Zigbee graphs (default visualization or Zigzag) do not reflect the fact that the device is off. This kind of make sense as the graphs are updated every 3-4 hours and furthermore the device is probably still in the neighbor tables but it would be nice to get “real time” information. Even zha-network-card display the device as “online = true”?

drcoolzic · February 9, 2021, 5:12pm

FYI after few hours the zha-network card and Zigzag report the device as offline. This means that a network scan has been done.

Zigzag reports the device in red = not available
According to @Adminiuga : not available =

last_seen > 2h (for powered dev)
last_seen > 8h (for battery operated dev)

image874×583 44.7 KB

r00nster · February 10, 2021, 10:38am

ok, so this has just gone to a whole new level of “weird”

Yesterday evening whilst thinking of resetting everything it occurred to me that I hadn’t seen (and no one had moaned) that any lights had flashed. So I tested a couple and no flash.

So I didn’t change anything and let everything carry on as normal (lights being turned off overnight etc.) then this morning thus far… No flashes ???

I have 2 bulbs that for the last month or so you could guarantee will flash after power on yet this morning neither flashed.

I’m loosing it

So I did some digging around last night and whilst it might have been obvious to others in this thread it wasn’t to me but seems the only way to update Hue bulb firmware is via the hue hub/hue app which I was hoping to do away with using the ZHA/Conbee II stick.

So that leaves me with deciding whether

Stay as I am (ZHA/Conbee II), figure out all the glitches and never update firmware ?
Replace ZHA/Conbee II with my Hue hub move all devices to it and use the HA Hue integration ?
Run dual Hue & ZHA/Conbee II zigbee networks ?

I suspect not all devices will be hue hub compatible which is why I put in option 3.

Any one got comments or good/bad experiences of the above to share ?

walt · February 10, 2021, 8:56pm

You can update the firmware with ZHA:

drcoolzic · February 11, 2021, 9:19am

What is the procedure?
Can you detail please

ViperRNMC · February 11, 2021, 9:52am

What I did, install deconz add-on and start it with the conbeeII as device. In the web UI you can update the stick. After this delete the add-on en reload ZHA.

Using home assistant OS 5.11 and conbee II. I agree it’s not the way you want, but it works.

Tismo · November 18, 2021, 12:35pm

So what’s the consensus to take away from this thread?
to have an optimum zigbee mesh in a house , use routers/repeater plug ins?
do you recommend any particular reliable brands?
I have conbee 2 stick and use ZHA integration with 90% of my end devices from aqara.
Thank you for all the research and testing you did OP

Hedda · December 6, 2021, 2:19pm

If many devices and/or large house then get at least a few known devices working as Zigbee routers:

https://www.home-assistant.io/integrations/zha#best-practices-for-avoiding-pairing-difficulties

https://www.home-assistant.io/integrations/zha#using-router-devices

https://github.com/home-assistant/home-assistant.io/pull/18864

ddeconin · February 24, 2022, 9:54am

Hi,
To get the right steps

Disable ZHA add-on
Install Deconz add-on
attach conbee stick to deconz
update firmware
disabl deconz add-on
enable ZHA add-on
Is he ZHA zigbee network still intact after this procedure?

ddeconin · February 24, 2022, 9:56am

Is it normal some edge devices don’t show the connection in the ZHA map? Some that are connected to routers show, but some others that are identical devices don’t. (perhaps these are directly connected to the controller?)

drcoolzic · February 24, 2022, 2:00pm

Normally you just have to be patient. Usually devices should show up after few hours but I had cases where I had to wait for days

khvej8 · February 25, 2022, 5:36am

It should work. I know it works if you do like this, sorry for the formatting, on a phone.
Disable ZHA, reboot, all the deconz stuff to upgrade, remove deconz, reboot, enable ZHA, reboot.

I have done that with success.

khvej8 · February 25, 2022, 5:42am

I have a lot of Aqara devices and use conbee2. I use Aqara plugs working as repeaters and as plugs at the same time, providing a good mesh. Read somewhere that Aqara devices work best with Aqara repeaters (the plugs) . Not sure it is true, works for me

khvej8 · February 25, 2022, 5:53am

I’m using option 3, mainly do to the missing upgrade in ZHA of hue bulbs. Further, I started with using hue before my HA setup. The hue integration in HA is working well, so I’m not removing it.

Main drawback is the missing bulbs from the ZHA zigbee mesh, hence I have some additional plugs used as repeaters in ZHA. In the basement I even put the hue bulbs on ZHA.

I do not plan to move the rest of the hue bulbs, as they work very well on the hue bridge.

Alamapstieks · February 25, 2022, 9:26am

I guys, not sure this is 100% the right spot to ask but you all seem very knowledgeable. I have a tonne of Hue and Zwave in the house. Recently I got a Sonoff USB 3.0 dongle so I can add some Aqara sensors as they are much cheaper than Zwave.

This works ok, but some devices don’t get picked up if too far from the hub.

So the question(s) are:

How do I add HUE to the mesh to help overcome this
I’ve read some posts of people suggesting to flash the Sonoff, I am unsure of the benefit?

Thanks

danhiking · February 27, 2022, 6:19am

I’ve waited patiently for a week now and the graph is still incomplete:

Screen Shot 2022-02-27 at 1.14.07 AM

It’s always the two Aqara nodes that fail to show an edge to any other node. The Sonoff and Jasco nodes appear correctly.

Despite this, the Aqara do have normal function in HA.

However, I’d really like to know where in the graph the Aqara are connected, especially whether either is connected via one of the relays, or directly with the controller.

How can I find that out, with or without the network graph?

drcoolzic · February 27, 2022, 10:53am

You have several options:

If you want to get a list of devices, you can use the ZHA network card: https://github.com/dmulcahey/zha-network-card
This card will display a lot of ZHA network information. As you are interested in displaying the neighbor’s information, you should use a configuration that contains the undocumented “neighbors” attribute see the example below taken from my Lovelace configuration:

- title: ZHA
    path: zha
    panel: true
    icon: mdi:zigbee
    badges: []
    cards:
      - clickable: true
        columns:
          - name: Name
            prop: name
          - attr: available
            id: available
            modify: x || "false"
            name: Online
          - attr: rssi
            name: RSSI
          - attr: lqi
            name: LQI
          - attr: last_seen
            name: Last Seen
          - attr: manufacturer
            name: Manufacturer
          - attr: model
            name: Model
          - attr: device_type
            name: Type
          - attr: ieee
            name: IEEE
          - name: NWK
            prop: nwk
          - attr: power_source
            name: Power Source
          - attr: quirk_class
            name: Quirk
          - name: Neighbours
            attr: neighbors
            modify: >-
              x.map((n, idx) => { let s = ""; if (idx > 0) s = "<br />"; s = s+
              n.relationship + " " + n.ieee; return s; })
        sort_by: name
        type: custom:zha-network-card

This will give you this kind of output:

if you want a nice graphic representation, you should use the ZigZag custom panel. It gives you a much much nicer representation of your network. You can find the instruction here: https://github.com/Samantha-uk/one/tree/main/home-assistant/zigzag-panel
Just follow the instructions and you will get a graph like this:

image1213×790 57.7 KB

CAVEAT:
All these solutions use the same information accessed from ZHA database and therefore if one fail most probably all will fail in other words if the neighbors don’t show in one display, they probably won’t show in all of them.
If you are using a ConBee II I have found that this coordinator seems to not always return the right information to ZHA or at least can get desynchronized with ZHA. You have to know that even if you restart from scratch a new network, the ConBee II keeps a lot f information in its cache from previous configuration and sometimes that confuses the overall system because the ConBee knows about some devices from a previous configuration but ZHA are not aware of them. One way to overcome this problem is to flush the ConBee II caches. For that matter I think that you only have to change the network id. I did that long ago and I do not remember exactly how to do that but someone can probably explain better than me. For me resetting the caches in the ConBee II and rebuilding the network from scratch solved the problem and now all my devices are shown correctly.
One thing to know about the Aqara devices is that once they are connected to a parent they almost never change even if the quality of the communication becomes bad. For example, I have a router very close to an Aqara device but originally it connected to the coordinator with a relatively bad signal and the Aqara device never changed to this close router. Therefore, the procedure I recommend is the following: When you add an Aqara device (actually I recommend doing this for any passive device) DO NOT add the component using the “add device” from the integration unless you want to connect it to the coordinator. Instead, you should use the “add device via this device” command. For that you have to go to the device description of the router close to the device you want to add and use the command mentioned above. Note that if you see that a device is not connected to the closest router what you can do is “forget” the device and add it again following the described procedure.

I hope this help

drcoolzic · February 27, 2022, 2:21pm

I have found the information about resetting ConBee II internal tables:

There’s no specific command in the protocol. You could change the PanID, Extended PanID and the network key, that would equate to a brand-new network.

To get a better view of what is going on you should turn on the debug flags. I recommend the following in the configuration.yaml:

logger:
  default: info
  logs:
    homeassistant.core: debug
    homeassistant.components.zha: debug
    zigpy: debug
    zigpy_cc: debug
    zigpy_deconz.zigbee.application: debug
    zigpy_deconz.api: debug
    zigpy_xbee.zigbee.application: debug
    zigpy_xbee.api: debug
    zigpy_zigate: debug
    zigpy_znp: debug
    zhaquirks: debug

Not all of them are necessary but …
In the debug log you should look for zdo requrest sent to 0x0000 NWK address with Mgmt_lqi_req commands and you should see in response the neighbors for each router (including coordinator). Something like this:

2021-01-08 18:51:59 DEBUG (MainThread) [**zigpy.neighbor**] [0xabdc] request status: Status.SUCCESS. response: Neighbors(entries=3, start_index=0, neighbor_table_list=[Neighbor(extended_pan_id=00:21:2e:ff:ff:06:19:bc, ieee=00:21:2e:ff:ff:06:19:bc, nwk=0x0000, packed=36, permit_joining=<PermitJoins.Unknown: 2>, depth=0, lqi=184), Neighbor(extended_pan_id=00:21:2e:ff:ff:06:19:bc, ieee=00:15:8d:00:04:50:6c:e9, nwk=0x6C5B, packed=18, permit_joining=<PermitJoins.NotAccepting: 0>, depth=2, lqi=48), Neighbor(extended_pan_id=00:21:2e:ff:ff:06:19:bc, ieee=00:15:8d:00:05:6a:18:9d, nwk=0x7317, packed=18, permit_joining=<PermitJoins.NotAccepting: 0>, depth=2, lqi=159)])
2021-01-08 18:51:59 DEBUG (MainThread) [zigpy.neighbor] [0xabdc] Done scanning. Total 3 neighbours

Remember that:

ConBee does not retain children information (like a regular Zigbee device should do) on a power cycle. If the child was joined directly to ConBee, prior the version when zigpy-deconz was storing that information, then it won’t be reported in the topology, because ConBee does not report it.

For the topology map, zha gets information from zigpy, zigpy gets the information by querying each router in the network.

So if you do not see the children after about 3 to 4 hours in the log file again I would suggest you reset the ConBee II

danhiking · February 28, 2022, 3:14am

Thanks! First things first, I installed the zha-network-card and the ZigZag panel, and they both confirm what you predicted: the missing links appear to be missing in the ZHA database. All visualizations show the same thing.

I am indeed using the ConBee II, so next I’m going to try looking at the debug logs.

What does “resetting” the ConBee II mean? And does that imply reattaching (and renaming??) every entity from scratch? Some of these devices are in very inaccessible places, so I’d rather not!

BTW, thanks for the advice about the “add device via this device” – I may need to re-add the Aqara devices, just as soon as I figure out where they are currently attached.

drcoolzic · February 28, 2022, 10:25am

Resetting as I mentioned means changing the the PanID, Extended PanID and the network key.
Unfortunately reseting the ConBee means that you need to start from scratch as all the devices will be “forgotten” in the operation.
It took me some time and effort to get everything right but I do not remember all details.
Have you done some tests before building the final network? Because I believe that your problem comes from the fact that the ConBee knows about some devices that zigpy-deconz do not know.

Even though your devices are not showing in the graphs are they visible in the device list? If that the case then you can try to “forget” them and “add them again”. The goal is to resynchronize what are stored in the devices internat tables and zigpy-deconz. This may work without the need to rebuild from scratch.