All my hue lights randomly becoming unavailable

[Mariusthvdb (Marius)]

@balloob

are having performance issues with their python scripts with just normal commands, we might need to evaluate that component and see where things go wrong.

Hey all, @balloob

back again, after extensive testing, both on Hassio and Hassos, I can report that the flapping issue seems to be caused by several ‘listeners’, rather than the python_script.

If I run the python scripts per time period, now set to each 15 seconds, the flapping stops, and responses of the Hue system seem alright. On the Hassos system, where none of these scripts is installed, but the Hue Custom component is, Hue system is dead right, and no issues arise.

That being said, one needs to be able to trigger from light state changes now and then to automate the Home, and thats when the issues arise. I have tested the following automations, which in them selves works fine, but cause the Hue system to go flapping, ie, show the lights as unavailable for a certain time, and then come back again. Also the disconnecting to the Hue ip address is shown in the logs.

Automations are:

based on event call_service in domain light:

- alias: 'Call Service Event Light'
  id: 'Call Service Event Light'
  trigger:
    platform: event
    event_type: call_service
    event_data:
      domain: light
  action:
    - delay: 00:00:02
    - service: python_script.whats_on
    - delay: 00:00:02
    - service: python_script.summary

based on individual light state (with added condition not being in home mode party, because thats another listener to the lights):

- alias: 'Sense Lights change'
  id: 'Sense Lights change'
#  hide_entity: True
#  initial_state: 'on'
  trigger:
    platform: state
    entity_id:
#Hue lights
      - light.dining_1
      - light.table_lamp_1
      - light.table_lamp_2
      - light.table_lamp_3
      - light.5
      - light.6
      - light.home_theater
      - light.frontdoor
      - light.9
      - light.10
      - light.hue_go_1
      - light.driveway
#Tradfri lights
      - light.chair_long
      - light.drawer
      - light.hobby_tafel
      - light.tv_backlight
      - light.bureau_left
      - light.bureau_right
      - light.inside
      - light.outside
      - light.pantry
  condition:
    condition: state
    entity_id: input_boolean.home_mode_party
    state: 'off'
  action:
    - delay: 00:00:02
    - service: python_script.whats_on
    - delay: 00:00:02
    - service: python_script.summary

and one on state change event, with template selecting the appropriate lights, rather a short and elegant automation:

- alias: 'State changed Light'
  id: 'State changed Light'
  trigger:
    platform: event
    event_type: state_changed
  condition:
    condition: template
    value_template: >
      {{ trigger.event.data.entity_id in state_attr('group.all_lights_only','entity_id') }}
  action:
    - delay: 00:00:02
    - service: python_script.whats_on
    - delay: 00:00:02
    - service: python_script.summary

There is some difference in system response, with the latter being the most robust, or better said least causing issues.
Still, they all are causing the Hue system to disconnect and reconnect after a short while, making this an unreliable source for further Home-automation ( because after reconnecting all last_changed attributes restart, and after disconnecting the condition for a light being in a certain state is not valid anymore) . which is what we all are here for aren’t we.

That the above is only happening in the Hue system implementation could be established by this automation, checking only my Tradfri lights:

- alias: 'State changed Ikea Light'
  id: 'State changed Ikea Light'
  trigger:
    platform: event
    event_type: state_changed
  condition:
    condition: template
    value_template: >
      {{ trigger.event.data.entity_id in state_attr('group.ikea_tradfri_lights','entity_id') }}
  action:
    - delay: 00:00:02
    - service: python_script.whats_on
    - delay: 00:00:02
    - service: python_script.summary

Immediate response, no hubs disconnecting.

Somehow the Hassio - Hue system (integration) doesn’t like listeners

Hope this helps in further exploring this issue, and be glad to help in any way.
before you ask: of course I did not run these automotations simultaneously…

Cheers,
Marius

[ptdalen]

I thought I had gotten past this as well. Running 75.3 now. Created this automation the other day. Worked great for one day, then this morning, it was stuck, could not get it to work, took me a few minutes to realize that sure enough, several of my hue bulbs were unavailable. I’ll double check, but I don’t have many conditions based on hue, I have several triggers based on other things, that have actions with Hue (pretty standard stuff). Definitely no python scripts to check status.

- alias: Set alcove light color based on Traffic to work
  trigger:
  - platform: state
    entity_id: input_select.house_mode
    from: 'Night'
  - platform: state
    entity_id: sensor.person_traffic_density_to_work
  - platform: time
    at: '07:35:00'
  condition:
  - condition: state
    entity_id: binary_sensor.workday_sensor
    state: 'on'
  - condition: state
    entity_id: input_select.house_mode
    state: Home
  - condition: time
    after: '07:35:00'
    before: '08:35:00'
  action:
  - service: light.turn_on
    data_template:
      entity_id: light.alcove_light
      brightness_pct: 100
      color_name: >
        {% set map = {'Normal': 'green', 'Moderate': 'orange', 'Heavy': 'red'} %}
        {% set state = states('sensor.person_traffic_density_to_work') %}
        {{ map[state] if state in map else 'white' }}

This bulb and another bulb physically nearby, but not part of any recent automation were suddenly unavailable. Just passing along for a data point. I will see how this runs tomorrow. After restarting HA it worked as expected. Also I had restarted HA yesterday, so it was not up and running for days before this happened

[ptdalen]

Just wanted to add this too, doubt it’s related, but much of my lighting for hue (not the example above), is controlled by calling hue scenes with scripts. So morning, runs two scenes for two groups of lights, turn on tv after dark, runs a different set of scenes, come home after dark, etc. Anyone else with issues, calling hue scenes?

[Mariusthvdb (Marius)]

ive been digging some further, and found out that not only does the Hue integration suffer from lights related automations calling the python scripts, but other processor heavy automations (listeners) cause the Hue lights to go unavailable immediately and start the ‘flapping’

I have an automation that triggers on each automation being triggered (of course without being triggered by itself…or several other often triggered ones) and another automation triggered by state change of my power sensors.

enabling either of these throws the Hue into ‘flapping’.

And only the Hue hub, the rest of the system runs fine. This is a timing issue leading to frustration. I can prevent it from happening (well, almost, even without these automations and pythons the hue shows unavailable now and then) by disabling these automations. But, these automations are there to monitor the House and its main systems. Exactly what HomeAssistant should be doing.

Hope the dev’s can optimize the Hue integration (before the asyncio move this didnt happen) and stop this unstable Hue behavior.

these are the 2 automations.

- alias: Automation ran
#  hide_entity: True
  initial_state: 'off'
  trigger:
    platform: event
    event_type: state_changed
  condition:
    condition: template
    value_template: >
      {% set skip_list = ['automation_ran', 'count_warnings', 'count_errors',
                           'activate_map_sensors_actueel'] %}
      {{ trigger.event.data.entity_id.startswith('automation.') and
         trigger.event.data.entity_id.split('.')[1] not in skip_list and
         'old_state' in trigger.event.data and 'new_state' in trigger.event.data }}
  action:
    - condition: template
      value_template: >
        {{ trigger.event.data.new_state.attributes.last_triggered !=
           trigger.event.data.old_state.attributes.last_triggered }}
    - service: notify.filed_automations
      data_template:
        message: >
         {{ as_timestamp(trigger.event.data.new_state.last_updated)|timestamp_custom("%d %b: %X") }}: {{ trigger.event.data.new_state.name }}
    - service: python_script.last_automation
      data_template:
        event: '{{ trigger.event }}'

- alias: 'Activate Map sensors actueel'
  id: 'Activate Map sensors actueel'
#  hide_entity: True
  initial_state: 'off'
  trigger:
    platform: event
    event_type: state_changed
    event_data:
      domain: sensor
  condition:
    condition: template
    value_template: >
      {{ trigger.event.data.entity_id.endswith('actueel')}}
  action:
    service: python_script.map_sensors_actueel

[ptdalen]

I’m running 76.2 and am still having this issue from time to time. Recently I added a new automation that uses one of my hue bulbs and have noticed that the issue is much, more common now. To be fair, I’m not always looking and the UI, or logs, so It’s possible they are “unavailable” but on other times during the day.

Here’s my usual Hue light control. Wake up in the morning, turn on a couple of hue light strips. Leave the house, turn off all lights (inlcuding those strips). Come home turn on the light strips again, if its dark, also turn on a couple other strips and a bulb. Turn on TV after dark, change the color/brightness of a some of the strips. Turn off the TV, change brigtness/etc. all of the lights are changed by calling Hue Scenes via a script. The lights at night are 99% stable. I cannot remember the last time they did not work.
The lights in the morning used to be stable, until I added this recent automation. It is more “intensive”. I’ll post in a second. Just to be clear, and to avoid useful recommendations, I’ll add that this has not always been an issue, its not my wi-fi, or channel, etc. Been there done that.

So here is what happens in the morning, and this is what has seemed to be causing my lights to become unavailable

- alias: Good Morning Routine
  trigger:
  - platform: state
    entity_id: binary_sensor.motion_sensor_living_room_motion
    from: 'off'
    to: 'on'
  - platform: state
    entity_id: binary_sensor.motion_sensor_kitchen_motion
    from: 'off'
    to: 'on'
  condition:
  - condition: state
    entity_id: input_select.security_system
    state: 'Armed (Home)'
  - condition: time
    after: '05:30:00'
    before: '11:30:00'
  action:
  - service: script.turn_on
    entity_id:
      - script.ecobee_resume_mode
      - script.kitchen_evening_lights_hue_scene
  - service: lock.unlock
    entity_id: lock.lock_garage_door_lock
  - service: input_select.select_option
    data_template:
      entity_id: input_select.security_system
      option: Disarmed
  - service: input_select.select_option
    data_template:
      entity_id: input_select.house_mode
      option: Home

Here is the script for the scene
kitchen_evening_lights_hue_scene:
  alias: Hue Scene Evening Lights (Kitchen)
  sequence:
    - service: hue.hue_activate_scene
      data:
        group_name: "Kitchen"
        scene_name: "Evening Lights"

Additional Automations
- alias: Set alcove light color based on Traffic to AVIS
  trigger:
  - platform: state
    entity_id: input_select.house_mode
    from: 'Night'
  - platform: state
    entity_id: sensor.tracys_traffic_density_to_avis
  - platform: time
    at: '06:35:00'
  condition:
  - condition: state
    entity_id: binary_sensor.workday_sensor
    state: 'on'
  - condition: state
    entity_id: input_select.house_mode
    state: Home
  - condition: time
    after: '06:35:00'
    before: '07:34:00'
  action:
  - service: light.turn_on
    data_template:
      entity_id: light.alcove_light
      brightness_pct: 100
      color_name: >
        {% set map = {'Normal': 'green', 'Moderate': 'orange', 'Heavy': 'red'} %}
        {% set state = states('sensor.tracys_traffic_density_to_avis') %}
        {{ map[state] if state in map else 'white' }}

- alias: Set alcove light color based on Traffic to Base
  trigger:
  - platform: state
    entity_id: input_select.house_mode
    from: 'Night'
  - platform: state
    entity_id: sensor.pauls_traffic_density_to_base
  - platform: time
    at: '07:35:00'
  condition:
  - condition: state
    entity_id: binary_sensor.workday_sensor
    state: 'on'
  - condition: state
    entity_id: input_select.house_mode
    state: Home
  - condition: time
    after: '07:35:00'
    before: '08:35:00'
  action:
  - service: light.turn_on
    data_template:
      entity_id: light.alcove_light
      brightness_pct: 100
      color_name: >
        {% set map = {'Normal': 'green', 'Moderate': 'orange', 'Heavy': 'red'} %}
        {% set state = states('sensor.pauls_traffic_density_to_base') %}
        {{ map[state] if state in map else 'white' }}

Additional details. Looking through the logs for time stamps as I was writing this, I see the three lights in my kitchen hue group, starting going unavailable, available at 5am exactly. Which does not seem to tie into this after all, but it is weird that it’s just this group of lights. The other 4-5 hue lights in different groups did not go unavailable at all during that time.

Well, I guess I believed the traffic automation was the cause, but I think that it just brought it more to my attention. It is weird that it is only one hue group though.

[Mariusthvdb (Marius)]

just a heads-up the dev’s have now enabled debug on the hue component to see if updating causes these disconnects. Ive posted some of my findings on the Github issue:

Philips Hue changed to unavailable

please have a look if you could do so too, so the dev’s have as much useful info as possible to be able to mitigate the issue here.
thx

[Mariusthvdb (Marius)]

binary sensors ping (and mqtt) causing Hue unavailable? please see https://github.com/home-assistant/home-assistant/issues/16689#issuecomment-442094226 and check if this would help in your setups.

cheers.

[Jonde]

Is there any update on this? I started to have the same problem yesterday when couple of my Hue lights started becoming unavailable suddenly. Noticed this when one of my automations was not working even when HA showed that it triggered normally. And when investigating I noticed that everything works but some of the lights are becoming unavailable and then suddenly available again.

This started when I added device_tracker and couple of automations that use that.

@Mariusthvdb are you still suffering from this or did you solved it by changing the python script and/or binary_sensor ping?

According to the Github issue, someone did a reset on their Hub and it started working. I actually plan on doing this since there are so many useless scenes etc. created by 3rd party andoird and iOS apps but don’t know if that is going to help with this.

Edit: Sorry for tagging you Mariusthvb

[Mariusthvdb (Marius)]

no this still prevails, and reseting the hub wont work…
there are many many posts on this.
that being said, I have 2 instances and the one which has my mqtt broker on it (the hassio add-on) suffers most. I can not have any automation rely on lights state, because of that.
It apparently has to do with I/O going on. And we must see the issue as a signal of something else in the system taking processor time/goin wrong. thats what the HA founder told me.

If you add show_last_changed to the lights, you can exactly see for how long they have been seen by the system.
On my Mqtt system that mostly is about a few minutes, and if I am lucky, several 10’s of minutes up to an hour.
On my 2nd system, which I use to show in the app and use as a front-end, light are available most of the time. But reloading Lovelace heavy cards (weather custom cards etc) might again cause Unavailable.

So long answer short: its still there.

see below, showing the same lights/groups in my

mqtt system:

and main system:

[Jonde]

Well this is too bad. It is actually kind of breaking thing and might move people away from Home Assistant in the end if it is not fixed.

I can investigate this on my side but most likely nothing new and useful will be found

[jono]

As a final thing to try take a look here

Philips Hue changed to unavailable

…so after my Hue lights were often showing as unavailable I also started seeing quite a few of my TP-Link smart plugs as unavailable in HA too. It seemed that over time more and more of them were showing as unavailable until I had the same problem with most of them (I have quite a few of them). The ones that started showing as unavailable first were downstairs, and quite a distance from our router upstairs. So I found out an old router and set that up downstairs to boost/extend the wifi. Sinc…

Since I boosted the wifi signal in my house the lights haven’t shown as unavailable once.

[Jonde]

Thanks, I will take that into account. Most likely that has nothing to do with my lights going unavailable since I have a powerful WIFI AP that is only a WIFI AP so it has pretty strong signal everywhere in my house. But this is still a good point to look

[Mariusthvdb (Marius)]

@jono

s a final thing

well, not ‘final’ as in this solved it…

this has been discussed before, as has changing the zigbee channels. It might solve some issues in particular for some users,because of their specific home settings. It isnt a solution to the issue in itself though.

It has to do with I/O on your local system and the Hue needing to be polled regularly, as it is implemented now, rather than receiving pushed changed from the Hue side. (like eg the Tradfri system does)

To stress the importance of this issue, we could however all hop over on the Hue developers forum and support @balloob’s call for a push API on the HUE system: https://developers.meethue.com/forum/t/2018-and-2019-hue-api-and-sdk-features/5996/21

[smart (Ron)]

I am having this issue also. I don’t think it is related to interference by WiFi, wouldn’t that make the lights unavailable on all platforms? I have eero’s mesh wifi and you can’t select channel in the config so I can’t really test that. My hue lights work perfectly with Amazon Echo, They work perfectly with the App and they work perfectly with two Flic Button Hubs I have on my system. Only HA has them bouncing between unavailable and active. I have two lights which are the worst and I have them automated to turn on when I arrive home and when a zigbee button is pressed to toggle. One or the other fails almost all the time. It’s rare that both are online at the same time.
I am on 0.81.0 but see no reason to upgrade for this issue since people are reporting same problem with 0.86.3 which is the current version as of this post.

[ptdalen]

Yeah, there seem to be two common causes of “unavailable”, but often people get them confused.

One possible common cause, is wifi/channel/network related, this will affect all apps, controllers, etc

The issue that this thread is trying to resolve is why HA is marking some lights as unavailable even when they seem to be available to other systems.

I will say that there has been a lot of back and forth, and no resolution. For me and many others my lights go unavailable one or two at a time for a minute or so, throughout the day. But 95% of the time it does not affect me as I dont turn my lights off and on every 5 minutes, etc. For people who have the hub going off line, or all lgihts going offline for a long period of time, you are probably dealing with the 1st issue.

[smart (Ron)]

Mine are offline much more often. Checking right now. Two are offline

stairs_top and stairs_bottom are offline every morning which is when I use them. I no longer expect them to work at all.

A few seconds later now 4 are offline

I pretty much can’t control Hue with HA anymore.

few seconds later and different 4 offline now.

Few seconds later different set again (still 4 offline).

Basically 2-4 of my 12 Hue lights are offline at any given moment.
I can just watch this panel and they rotate around continuously. I never see them all online.

[ptdalen]

I cannot say for sure, but to me it does sound like a wifi/channel/HA/network issue. Most of my issues (and several others like me), are an occasional drop out of a bulb at a time. During a normal day, even in the middle of the night when nothing is going on, I’ll see a light or two go unavailable, and then back available in the next minute.

Are you running on a RPI? Do you have a lot going on? One of the comments above seemed to point to the HA server being too busy to verify connectivity to the hue entities

All my hue lights randomly becoming unavailable

well, not ‘final’ as in this solved it… this has been discussed before, as has changing the zigbee channels. It might solve some issues in particular for some users,because of their specific home settings. It isnt a solution to the issue in itself though. It has to do with I/O on your local system and the Hue needing to be polled regularly, as it is implemented now, rather than receiving pushed changed from the Hue side. (like eg the Tradfri system does) To stress the importance of this issu…

[smart (Ron)]

@ptdalen

Are you running on a RPI? Do you have a lot going on? One of the comments above seemed to point to the HA server being too busy to verify connectivity to the hue entities

This is the one I suspect. I am running on RPI but I have nothing else running on this RPI. I have MQTT on it’s own PI, I have several other RPI’s about doing various choirs.

I just tried disabling the recorder, I do have mysql running on this RPI so maybe that is an issue. I will disable the mySQL server and see if that helps. I am using an external drive NOT SD card so performance is pretty good on this RPI which is a 3b+

On a side note I found that my recorder has never purged. Documentation says default is to keep 10 days worth and purge every day. I have found 505,858 events in the DB with a min created date of Sept 6 up until today. Can I just delete all the data in each table to manually purge?

I still say if it was WiFi interference my other methods of connection would be failing. Alexa is 100% never fails.

UPDATE: disabled mysql cpu at 3% max lights still bouncing offline/online in HA only.

[Anwen]

I have had the bouncing happen with my Hue lights at times, just in HA and nowhere else. I have found that rebooting my Pi - not just restarting HA - often makes the problem go away. I’m using HassOS/Hass.io on a RPi 3B. Even though everything looks fine on the Pi in regards to cpu usage, memory usage, etc, rebooting the host makes a real difference.

HA on my Pi had been running great on the last release of 0.82 for quite some time. I don’t like to get too far behind, so this weekend I upgraded HA version by version to 0.85.1 and started seeing those Hue unavailable errors again. I rebooted the Pi and it’s been running beautifully ever since, no Hue errors at all. I’m sure this won’t solve everyone’s problems, but maybe it will help someone.

[Mariusthvdb (Marius)]

@ptdalen et all,

Please allow me to add to you analysis of 2 reasons. I think there are 3

1- interference with other wireless topologies. Can’t say I’ve been hampered by that, but some are, and solutions have been suggested. Even Hue has that builtin their app, changing to another Zigbee channel could solve that, as does changing router channels.

2- Don’t forget Hue lights due really turn unavailable sometimes. I see that happening too, but, that is on an individual light basis, and doesn’t cause the Hue hub to go unavailable. In fact, it should show the lights unavailable, because they are.

3- The main issue.Tthis is the most unclear and most frustrating reason, and has been so for almost a year now… Still not solved.
It seems to boil down to I/O happening in your system, which causes the Hue Hub to go unavailable and hence show all lights unavailable. Why only the Hue integration suffers from that is beyond me or even the dev’s for that matter, because no answers are given when asked. Seems to also have to do with the fact HA needs to poll the Hue, versus components changes pushed.

Solution for now:
My Mqtt broker (which I had on my main system) is one of those I/O troublemakers, so I moved that to another RPi. Ive had my share of binary_sensors, deleted these, and that gave some relieve. Bluetooth is another known factor. If you only check the startup log, and see how long it takes for that component to setup, alarm bells should ring. Taken that out too. (Maybe better filed under 2… still seems a HA heavy I/O thing)

All in all, I now have 2 systems: a HA Mqtt ‘hub’, with bluetooth and some other administrative things, and my main system, that subscribes to the mqtt topics I need (my Mqtt broker receives many more topics, that are now not overflowing the main systems statemachine.

Only times when my main system is showing unavailable now, is when I do some system checking, or reloading/freshing Lovelace. Regular Ha seems steadier than ever.

I can now wake up and see my lights having a last_changed of over 6 hours While on the Mqtt system last_changed of over more than half an hour is a rarity.

As a side note: if individual lights show unavailable, try switching the group, and see the lights toggle… show the lights are reachable and HA incorrectly states they aren’t.

Hope this helps.
cheers.