1% Failure Rate - Help!

Configuration
1

I setup HA last year and added a few HUE motion sensors via node-red and HA UI. I couldn’t quite dedicate the time to learn so I gave up on HA. One thing I noticed is that motion detection and lights turning on was near instantaneous.

This year, I dove in and setup HA and learned all about it. I noticed there is a small delay in motion detection and lights turning on. Not a HUGE deal but definitely noticeable. I thought maybe it was because I am not using node-red anymore and only HA

The problem then started getting worse…

I am dealing with a weird “issue” of sluggish performance and failed automations. When I say failed, there is absolutely nothing in the logs, logbooks or traces of the automations.

The weird part is I can’t figure out a pattern or issue. Sometimes, they just don’t work. If our front door opens 100 times a day, the TTS announcement happens probably 95 times. During those 5 times, I look at the logs, the sensor, the speaker. All seem fine but no mention of the trigger or door opening.

I do not have many complicated automations or an excessive amount of devices/entities. I assume most people here have a lot.

1.) Motion sensor at the entrance turns on lights (fails 1/10 times)

2.) Front door open = TTS announcement on a nearby speaker (fails randomly too)

3.) Stairs motion sensor, turns on light for 30 seconds, then off. (fails sometimes as well)

Setup details:
HA on Unraid VM
The server itself is quite beefy. The home network is also quite overkill too.

My question is, what do I do to troubleshoot? No info in the logs helps, logbook just doesn’t show those devices as triggering. The devices aren’t losing connection to wifi or their hubs.

My unraid VM Is used as a plex server running more than few docker containers. It is filling up on space - could this be an issue? (I can’t remember what I allocated HA, but HA UI says 50% of storage is used up)

I am lost and need some guidance

2

i think there’s a bunch of noise that may be causing you more confusion. you’re talking about failed automation, node red, etc.

focus on this. if the devices/entities events aren’t happening, the automation can be perfect and they’ll still never work.

you didn’t say much about the devices you have. and how do you know that they aren’t having temporary disconnects. you said “hub” so does this mean zwave or zigbee?

2 Likes
3

More detail needed. Presumably you’re using Zigbee, if the motion sensors are Hue? What integration?

Intermittent failures are often associated with too few mains-powered router devices. All the devices you mention are sensors, so not routers. You say the home network is “quite overkill” - is this the Zigbee network? Hue motion sensors have nothing to do with wi-fi.

Lots of good advice here:

2 Likes
4

Thanks for the replies. My post was rushed and lacking technical information. Sorry for that.

Integrations (not everything is listed):
Hue lights and motion sensors = hue hub
Lutron lights = lutron hub
Aqara fp2 presense sensors = wifi/homekit(bridge?)
Aqara door sensors = aqara hub via matter integration
Esphome
Mqtt
Etc etc

I have about a dozen other integrations but this is the meat of the hardware.

I dont have zigbee or zwave integrations. I found it too technical. I live in a 3 story house with the unraid server in the bottom floor in a mechanical room. Adding peripherals didnt seem efficient and i had hubs for all of these devices before I joined HA.

In terms of the issues, its hard to describe. The hue motion sensor will detect motion because of the red light. But the lights wont turn on, or off.

This happens maybe 1% of the time and i cannot repeat it. If i restart HA, works fine. If i wait 5 minutes and try the automation again, works fine.

However, its annoying because these 3 automations are critical.

Lastly, my post is more about finding ways to troubleshoot an issue I cant re-create. Or “good habits”

What should i be checking? More ram? Is the activity on my unraid server affecting the HA VM?

Thanks

5

There are three main components to your lighting automations:

  1. INPUT: Hue Mottion Sensor → Hue Bridge → Home Assistant
  2. The Automation/Rule in HA: what code is being triggered by #1 above?
  3. OUTPUT: Home Assistant → Lutron Hub → Lutron Switch

What is not clear from what you’ve shared thus far is where in the chain of this lighting automation are things going sideways.

Run a test to verify that for EVERY Hue motion sensor event, that you see a corresponding Home Assistant event on the corresponding Device.

Verify that for every Hue Motion event, that your HA Automation is being properly triggered/run.

Verify that every time Home Assistant tries to turn on/off your Lutron light, that the actual light works as expected.

Somewhere within this chain of events, something is either not configured correctly, or a device is not communicating correctly.

It will be up to you to determine what is going on by breaking the problem down into its 3 main pieces, and then work within the one area where the problem is, to fix the root issue.

3 Likes
6

I highly recommend that you reconsider moving all your Zigbee devices to the built-in native ZHA integration with Zigbee Coordinator USB dongle, it is really easy to setup and use days + the best practice is regardless to put all Zigbee devices on the a single Zigbee network/gateway/coordinator including adding a bunch of Zigbee mains-powered devices that act as ”Zigbee Router” (Zigbee repeater/extender) devices on that same network. Check out these community guides which also explains why you should want to consolidate:

1 Like
7

There are by the way many step by step ”ZHA beginner” videos about ZHA setup from scratch on YouTube. Like for example these:

8

Thanks! I have followed that standard proecedure a few times and can’t figure out where its going wrong. Again, it barely happens

  1. INPUT: Hue Mottion Sensor → Hue Bridge → Home Assistant
  2. The Automation/Rule in HA: what code is being triggered by #1 above?
  3. OUTPUT: Home Assistant → Lutron Hub → Lutron Switch

It just happened again, I checked everything and it seems like motion wasn’t detected, so the automation never ran. Usually what I would do to test this, is go wave infront of the motion sensor, very easy to see if its working or not - right?

But as soon as I do, works fine. Its just failing once or twice a day.

The sensor log is not reporting any unknown/unavailable state. A motion event just doesn’t exist for that time

I guess I was hoping maybe I had some glaring holes in my setup and I had allocated too few resources to HA and that would at times cause some kind of lag/downtime - but I guess thats not the case…

9

Aquara hub, Hue hub, Lutron hub, Homekit bridge…

I agree with Hedda, get a Zigbee stick and connect any of those devices directly to it.

Your Hue lights and motion sensors are Zigbee, as are the Aquara door sensors. Zigbee and WiFi signals overlap, so that might explain the intermittent interference.
You could try changing your 2.4 Ghz channel on your router to see if that helps.
I’m not sure if Lutron is affected by WiFI signals.

10

I can’t speak from first-hand experience, but could your Zigbee devices be “hub hunting” because you have so many?

One hub to rule them all.

When I added one of these in every room, my Zigbee network became much more reliable.

11

Sounds to me like root cause is likley either EMI interference or too few Zigbee Router devices on each Zigbee network, or a combination of those two, which is the two most common problems with Zigbee networks/devices and the reason I wrote this guide with the best practice tips that should be followed:

12

Thats absolutely perfect. Thank you for your contribution in writing the guide and helping me with this issue.

I have had the hubs before i got into HA and getting a dongle for my server in a closed mechanical room seemed like a recipe for disaster but after reading your guide and watching a few videos I think I will take the plunge.

Thanks again!

1 Like
13

@BlackMamba08 Good for you! Personally I would like to recommend that you do this:

  1. Buy a Zigbee Coordinator USB radio dongle that is well supported by the community, like either:
    a): The CC2652P-based “Sonoff Zigbee 3.0 USB Dongle Plus (which was later renamed to “ZBDongle-P” by ITead), and flash latest Z-Stack 3.x.0 community firmware on it using ZigStar Multi Tool on a different computer before getting started with it. With a CC2652P-based Zigbee Coordinator you can begin by using Home Assistant’s built-in native ZHA (Zigbee Home Automation) integration but still leave the option to later reuse it if you even want to move to the stand-alone Zigbee2MQTT (a.k.a. Z2M) Zigbee gateway application. Another bonus with this “ZBDongle-P” dongle is the external antenna which makes it get better reception, (not having an external antenna should only be a problem if you have not add enough Zigbee Router devices as Zigbee repeaters/extenders).
    b): The EFR32MG21-based official Home Assistant SkyConnect USB radio dongle/stick (which was later renamed to “Home Assistant Connect ZBT-1” by Nabu Casa) if you are sure that you will only use Home Assistant’s built-in native ZHA (Zigbee Home Automation) integration, but if you do make sure to first update to the latest EmberZNet (NCP) firmware using the official Web-Flasher on a different computer before getting started with it, that will make it into a dedicated Zigbee Coordinator and disable the multi-PAN/multiprotocol support that is not longer recommended. The downsides to Silicon Labs based radios are not recommended for Zigbee2MQTT and as the SkyConnect model does not have an external antenna it is extra important to start adding enough Zigbee Router devices as Zigbee repeaters/extenders (especially before adding any battery-powered devices). The upsides is that it is very easy to update firmware on it from Home Assistant, and even easy migrate to a different Zigbee Coordinator adapter using the ZHA integration if you ever want to repurpose this SkyConnect dongle hardware for Matter by reflashing it into a dedicated Thread radio adapter using OpenThread firmware instead.
  2. Since you want to have the dongle far way from the computer running Home Assistant I suggest buying a “USB Ethernet RJ45 Extender Adapter” converter-kit and and a single long CAT5e/CAT6 shielded Ethernet cable with RJ45 connectors as such a converter will allow you to convert any CAT5e/CAT6 Ethernet cable into a very long dumb USB extension cable (note that 30 meters or 100 feet is the recommended maximum length for USB 2.0 data traffic over such a passive cable). There are many inexpensive variants of such USB Extender Over RJ45 adapters available, see example this → https://www.amazon.com/Male-RJ45-Female-Extension-Adapter/dp/B083W3D65G
    a): An alternative could be to instead get a network-attached Zigbee Coordinator adapter but that increases the complexity of the setup and makes it more complicated if you ever want to upgrade.
  3. Connect the Zigbee Coordinator USB radio dongle you bought via the long shielded USB extension cable to a native USB 2.0 port on your computer/server. Do not connect it to a USB 3.0/4.0 port as those are infamously well known to generate serious EMI/RMI/EMF interference to all low-power 2.4GHz devices (including Zigbee, Thread. and Bluetooth).
    a: If your computer/server does not have a native USB 2.0 port then buy and connect a powered USB 2.0 hub (with an external-supply) and then connect the USB extension cable + Zigbee Coordinator USB radio dongle via that powered USB 2.0 hub, (note that is needs to be power USB 2.0 hub with external-supply and not a USB 3.x hub / USB 4.x hub or no external-supply). See example → https://www.amazon.com/AmazonBasics-Port-USB-Power-Adapter/dp/B00DQFGJR4/
  4. Place your Zigbee Coordinator radio adapter in a location where it is not too close to any known sources of EMI/RMI/EMF interference. That includes all types of electronics and appliances or cables/wired with electricity. Seriously, all Zigbee radios are extremely sensitive to electromagnetic interference and electromagnetic fields, to the point of it being ridiculous sometimes.
    a): Preferably also place your Zigbee Coordinator radio adapter so that it is not too close to a wall, ceiling, floor, etc. as just getting it 25-centimeter/10-inches away from any building surface will make it get better reception.
  5. After you are done initial installation/configuration of Home Assistant’s built-in native ZHA (Zigbee Home Automation) integration, then start adding all mains-powered Zigbee Router devices, beginning with those that are closest to the Zigbee Coordinator. The best process is if you finish doing this 24-hours before start adding battery-powered Zigbee devices.
    a): If you do not have many mains-powered Zigbee Router devices or you do not want to start by moving such devices from their existing Zigbee network(s) then you need to consider buying a few dedicated mains-powered Zigbee Router devices to connect first. Regardless, be aware that adding a few “known good” dedicated mains-powered Zigbee Router devices to your Zigbee network will make it more stable and robust. See → Zigbee networks: how to guide for avoiding interference + optimizing using Zigbee Router devices (repeaters/extenders) to get best possible range and coverage
    b): Pro-tip is that you can easily convert inextensive Zigbee Coordinator USB radios adapter like the “Sonoff ZBDongle-P” and “Sonoff ZBDongle-E” models into very powerful “known good” dedicated mains-powered Zigbee Router devices by simply re-flashing them with Zigbee Router firmware and then powering them via USB-charger adapters to make the best permanent Zigbee repeaters/extenders that you can currently buy. The alternative is just buy a few pre-made dedicated Zigbee Router products, like the “IKEA Tradfri Signal Repeater ” and/or the “Aeotec Range Extender Zi ” (which work very well out-of-the-box but are not as good as the above mentioned DIY solution).
  6. If you have problems pairing a device, do not move it closer to the Zigbee Coordinator USB radio adapter. Even though that approach is recommended by some manufacturers, the general best practice if that Zigbee devices should be paired in the location where it’s going to be used.
    a): If your paired/joined Zigbee device is not showing all expected entities (attributes) in the ZHA integration then first try re-pairing it a few times, it is likely that the device interview process was interupted for some reason, and some battery-powered devices might have to have their batteries replaced and/or manually be kept alive by activating a change state on the device (like for example triggering a motion sensor with motion).
    b): if your paired/joined Zigbee device still is not showing all expected entities (attributes) in the ZHA integration after being re-paired then chances are it will need custom ZHA Device Handlers (also known as “zha-quirk”) for that specific device. This is more common if it is a newly released or brand new device that no one added support for previously or an odd device with complex features that almost no one else have ever bought due to it being a niche product with a very small userbase. See → Zigbee Guide: How-to setup local custom device handler quirks in ZHA integration
  7. When in doubt, keep add more mains-powered Zigbee Router devices! :wink:

Additional note: If you have any Zigbee lightbulbs/lights that are Zigbee Router devices (which most are) then you need to make sure that the power for those are not connected via a dumb switch (like an easily accessible wall-switch) that people that flick ON and OFF. Zigbee Router devices are meant to bve always connected and always powered, and if they constantly and unessesary become unavailable on a regular basis then that will mess up your Zigbee network mesh as device will need to keep finding alternative routes. And understand that while it might alleviate the situation a little by adding a few “known good” dedicated Zigbee Router devices in strategic locations it is not a permanent alternative to leaving your Zigbee lightbulbs/lights connected to dumb switches if they are Zigbee Router devices.

If you do that it should not only be more or less plug-and-play but you avoid the most common pitfalls.

PS: Again, highly recommend you also read and do your best to follow all the best practice tips here → Zigbee networks: how to guide for avoiding interference + optimizing using Zigbee Router devices (repeaters/extenders) to get best possible range and coverage

14

FYI, I decided to rewrite that post and re-posted it as a new comment post to the existing communty guide about migrating from the Philips Hue Bridge to the ZHA integration so that it can be useful to more people that are considering migration away from their commercial Zigbee gateway/hub/bridge → Migrating from a Philips Hue Bridge to the ZHA (Zigbee Home Automation) integration - #11 by Hedda