Problem with ZHA network visualization

drcoolzic · February 3, 2021, 10:46am

From what I understand ZHA scan the network for neighbors every four to five hours. It is planned to add, in a future release, a service call like the zha_map.scan_now to ZHA to force an immediate scan. Meanwhile if you want to force more frequent scans you can add this to configuration.yaml file

zha:
  zigpy_config:
    topology_scan_period: 20 # set the scan interval to 20 mn

according to @Samantha (developer of Zigzag) interval value should not be less than 20

Normally zha-map is not used anymore by any of the “new” visualization tools (including Zigzag). So apart from may be waking-up some sleepy devices it should not make any differences.

Currently, in my tiny test network, all my Aqara devices are correctly linked to either the ConBee II coordinator or to my two Philips Hue routers.

Foxxy · February 3, 2021, 11:44am

Small typo
Best, JR

drcoolzic · February 3, 2021, 1:29pm

Yes you’re right!
Corrected

Quatuor · February 3, 2021, 8:18pm

You don’t need zha-map anymore with the latest (2020.12.x and newer) releases, which have either built in one, or the pretty one - ZigZag from Samanha.
There’s no “scan now” service ATM, instead users are urged to practice patience till next scheduled network scan, which occurs either 3 or 4 hours, don’t remember exactly.

dproffer · February 3, 2021, 8:30pm

Thanks for the info and update. The repository did seem ‘dated’. Yes patience, I like to toss my unattached devices around the ZHA Visualization while waiting for them to ‘connect’ with a ‘pal’

mrneutron · February 4, 2021, 5:56pm

Zigbee2MQTT sounds like it has benefits that ZHA does not - stability and OTA firmware update capability.

Is it possible to run both ZHA and Zigbee2MQTT?

If not, what system changes would I have to make to switch to Zigbee2MQTT?
I’ve got a Sonoff Zigby Bridge running with Tasmota firmware.

On another related topic, I just updated HA to:
core-2021.2.0
supervisor-2021.01.7

The first thing I notice is that in the ZHA Visualization map, Zigbee device addresses are reported in hex again.

m0wlheld · February 4, 2021, 5:58pm

Not on the same coordinator device and therefor not on the same Zigbee network.

dproffer · February 4, 2021, 7:37pm

Pretty sure the Sonoff Zigbee Bridge Tasmoto’ed or not is NOT supported as coordinator by Zigbee2MQTT.

mrneutron · February 4, 2021, 11:05pm

What are people using for a Zigbee coordinator, with Zigbee2MQTT?

dproffer · February 4, 2021, 11:14pm

mrneutron · February 4, 2021, 11:22pm

Thanks. All these appear to be host-controlled devices, rather than a stand alone coordinator.

dproffer · February 4, 2021, 11:38pm

I believe that the Sonoff Zigbee unit is about the only ‘open’ device I’ve seen that is not USB or direct bus connected to a host CPU. And the experts do not think that the wifi tcp socket connection that is used between the Sonoff Zigbee bridge and the host CPU is reliable.

sk229 · February 6, 2021, 5:45pm

And, we’re back:

r00nster · February 8, 2021, 7:19pm

Sorry to post a problem here but this seems a very “ZigBee Educated” thread.

I posted this in another thread but didn’t really get an answer.

My setup
Home Assistant docker (latest version)
Conbee II (latest firmware)
ZHA (I recently switched from DeCONZ to see if this would help the problem but it didn’t)
Philips Hue bulbs (mainly)
Philips Hue motion sensors (x4)
Philips Hue smart button (only 1)
Aquara Temperature sensor
Aquara Switch sensor
Trafri GU10 bulbs (only 2)
Tradfri mains outlet socket
Innr mains outlet socket

All has worked fine for over a year and I have been slowly growing my zigbee devices. Then about 2 months ago (just before I started adding bulbs that were non-Philips Hue) the following issue started.

Power-on a light from a physical light switch that had been previously switched off at the physical light switch. The light would switch on as expected but then the following would happen.
After a small amount of time being switch on (say 30 seconds) the light that was switched on would flash (off then back on) in about half a second. Then anything between 5 and 10secs after that it would do it again. It would repeat this a random amount of times. Sometimes once, sometimes 2,3 4, 5 and so on. Sometime this might go on for 5 mins repeating flash, wait, flash etc…
Also a nearby light that was already on may also emit the off/on flash (normally only once but sometimes multiple times).

This was initially just 1 or 2 lights then more started to do this.

I’ve been racking my brain Googling and trying to figure it out and come to the conclusion this is some sort of mesh reconfigure/reset that is going on when the light is powered-on from previously being powered-off. First I noticed lights that had been powered-off (via a physical switch) overnight would do this (others in the family haven’t got out of the habit of switching the light off before bedtime).

Hence a light that had been off for a number of hours was re-announcing itself to the mesh. What’s strange is the “powering-off lights” from a physical switch has always been there but the flashing only started recently,

Also until recently this theory held but then lights that may have been powered off for 20 or so minutes might emit the problem.

Another thing, some light regularly do this from power-on (like almost every time) and other much less frequently and even never.

I switched from DeCONZ to ZHA over the weekend as read somewhere someone saying DeCONZ previously had a similar issue with flashing lights as a light discovered sensors in the mesh. But it still happens in ZHA.

This is obviously very annoying and pretty much halted my HA project as its driving the household mad.

So…

Is this normal behaviour for the lights to flash and under what circumstance might this happen ?

Do I have some sort of issue somewhere ?

How do I troubleshoot and find the problem using ZHA, logging, visualisation etc…

Any help appreciated.

PS. I’ve still got my Philips Hue hub and may move all devices to that temporarily to see if it happens there also.

dproffer · February 8, 2021, 7:52pm

Toggling the power to a zigbee device of any kind is really not the ‘best proactice’ for the use of a zigbee network. I know you say it ‘work’ at times. But basically everytime a device joins (and leaves) the network a somewhat chaotic period begins for the next several hours as all the devices decide on new neighbor connections. Yes, the network should handle this without flashing bulbs, but devices like the Aqara’s and maybe others don’t seem to do as good a job of conforming to zigbee standards as others.

I know my above ‘opinion’ does not help you in any way. Maybe useful questions:

did problems start after some particular device was added? think about the Innr socket as it probably a router and I don’t know much about if anyone else is reporting compatibility issues to level of Aqara devices.

firmware version on all devices, especially all the bulbs? with my Ikea bulb ‘flashing’ issue on my Hue zigbee network, I suspect a firmware change on Hue bulbs may have changed the occurrences of this occurring. I’m pretty sure the only way I got rid of the problem was removing Ikea bulbs from Hue network.

painful, but you might need to segregate your devices to separate zigbee network. About the only way to do this is multiple zigbee2mqtt networks, or separate proprietary Hue and Ikea and ZHA networks.

any chance one or more of your bulbs or devices has a direct control, or grouping setup in it at the zigbee level? For example a Hue motion sensor that has had a config saved in it to turn on a zigbee group of individual bulb?

Hope there is some here with good knowledge to help you!

r00nster · February 8, 2021, 8:37pm

Thanks for the reply. I agree powering down a zigbee device isn’t the best of ideas but find it surprising the response to this when the device is powered on is to flash bulbs. I can’t see how that as “a standard” would catch on as “the user” would find it most annoying after a time.

I started my config on deCONZ with a few hue bulbs and a single Aqara temp sensor. Then slowly added more hue bulbs and hue motion sensors. That config was rock solid and never had any issues (where the bulbs were powered off or not). Then I added more hue bulbs and motion sensors as part of a Black Friday binge. So end of Nov 2020 all was ok.

Then I started to notice a single light doing the flashing very infrequently. When the flashing started I only had the Aqara temp sensor that was non-hue which has been in my zigbee setup since day one.

Then I noticed the Ikea Tradfri bulbs were half the price of hue bulbs so thought I’d give them a try (bad idea!). I got GU10’s and E14’s. The E14’s were a nightmare (saying they were on when they were off etc.) so got rid of them after a day or so but the GU10’s I only had 2 of and seemed ok so I kept them. I bought the Tradfri sockets and Innr Sockets about the same time to control some Xmas lights so all were added around the same time, Since then the problem has probably worsened.

I pretty sure which lights will do the flashing thing but there are some that have never done it. For instance in my hallway I have 2 lights near the back in separate fittings but 3 at the front in a single fitting (all hue). The 3 at the front are grouped in HA but also all hallway lights are grouped. All of these are controlled by a single physical switch so all switch on at the same time. When I switch on the 3 lights at the front randomly flash but to date the 2 at the rear I don’t think have ever flashed.

One other light I have pretty much always flashes. Power it off for 20 mins or so then power it on and it flashes for about 5mins before it settles. Very annoying but at least I have 1 light I know I can recreate the issue with some monitoring (if I only know what to look for). This is also probably now the nearest light to my HA/Conbee box. Although to troubleshoot the issue I moved my HA box as well and the light flashed as bad before I moved it.

I thought it was light groupings but I have other lights not in groups that flash but its funny you mentioned about groupings as when I moved to ZHA I noticed in the Configuration, Integrations, ZHA, Configure, Groups screen there were 4 groups. One called “Default Lightlink Group” and 3 others

“No name group 0x0005”
“No name group 0xFFF0”
“No name group 0xE646”

The first 2 contain bulbs and the last the Conbee device. I didn’t create these, they just appeared in ZHA but they seem to follow my HA light groups I created so thought it had something to do with that.

My next plans are to go through the firmware revisions of all my devices as you suggested and update where possible and also to teardown the whole zigbee setup and start with my original setup on ZHA and work on slowly from there.

But whilst its in the “broken” state it would be nice to troubleshoot and find the issue.

dproffer · February 8, 2021, 9:38pm

Well, to quote Ringo, unscrewing all those bulbs… ‘I GOT BLISTERS ON MY FINGERS!!!’"

Your right, random flashing is not a ‘standard’, however this does seem to be the way manufactures do communicate with us humans for the blubs.

One thing I thought about with the flashing on power on, however perhaps your last post threw water on, what that the bulbs that flash do so because they cannot find a coordinator. This might happen if the flashing bulb had it ‘save’ best next hop as a bulb that was also off. But I’m way out of my depth on how the zigbee mesh network sets up.

Do you have ZHA network diagram of how your network graph looks when every thing is truned on? This might shed some light on how the devices are connecting between themselves.

Again, beyond my paygrade, but having those groups hanging around don’t see right, especially after a fresh setup of everything.

Looking that ZHA config page for one of my devices, in this case an OSRAM bulb, you might try some combo of ‘reconfigure device’ (I’m still not sure exactly what this does) and ‘manage clusters’, see if the bulbs have any groups or devices that don’t look like defaults (for example OTA is a buiild in group you want), and get rid of them.

drcoolzic · February 9, 2021, 10:19am

You have to know that the devices keep information about the network. So if you take a device and add it to a “new” network it will still remember things from the past.

I did some test: I moved from a ZHA network to a deCONZ network and back to ZHA. I was surprised that routers and coordinator would remember some neighbors that were not recognized by ZHA. @Adminiuga told me that this was because the ConBee II was aware of devices in neighbors table that ZHA/Zigpy did not know as the devices where not yet discovered. It seems that by default when you start a new network the coordinator uses the last PAN ID and therefore do not flush its internal tables (same for connected routers). You can find lot of useful information if you take time to read this https://github.com/home-assistant/core/issues/44954

The group information is located inside the Zigbee devices and therefore it make sense that they are found even though ZHA is not aware of them.

For your problem of flashing bulbs, I think that @dproffer is right the light are flashing probably to indicate that they are joining the network. On my test network, I only have hue bulb and Aqara devices. I can do a test to see if light flashes when rejoining the network.

If you have a ConBee coordinator, you have to know that ZHA is using the deCONZ Zigbee stack and therefore it is normal to find information from previous experiment. If you want to start from scratch, you could change the PanID, Extended PanID and the network key, which would equate to a brand new network.

r00nster · February 9, 2021, 4:24pm

Thanks for the info…
How do I reset/change the PAN ID ?
There doesn’t seem to be an option in the GUI or configuration.yaml options
I plan to reset everything tonight and start again, slowing adding devices to see if the issue comes back.
I’ll start with the original setup I had and run that for a number of days before adding anything else.

Also how can I check the device firmware versions via ZHA ?
In deCONZ I could find these but can’t see how to do so in ZHA.

drcoolzic · February 9, 2021, 4:42pm

To change the PAN ID I do not know the answer.
You will get a lot of useful information by turning on the debug Flags. Follow this LINK

Once you have turned on the debug flag look for the following in the HA log file
2021-01-08 18:31:54 DEBUG (MainThread) [zigpy_deconz.api] Command Command.version (0,)
2021-01-08 18:31:54 DEBUG (MainThread) [zigpy_deconz.api] Version response: 26580700

This is suppose to be the latest. Go to deconz-firmware to get the latest firmware