Insteon device status not updating correctly

Third party integrations
1

I have two FanLincs each linked to and controlled by two 6-button wall keypads. I then have scenes created in the Insteon app that tie main button to the lights and the four smaller buttons to the fan’s four speeds. Scenes were created in the Insteon app to keep everything in sync. From in the Insteon app, I can turn on the “both fans to high” scene and four devices will respond at the same time – both fans go to high and both wall keypads light up the ‘high’ button. The hardware all responds nicely and the scenes in the Insteon app seem to be working as expected. But…

When I try bring all of this into HA, things are not responding as expected.

I tried recreating via HA scenes and groups the same scene controls. What I found was that devices would react one at a time with as a full second or longer lag between. So, firing a “both fans to high” scene would turn one fan on, then a light on a keypad, then the other fan, then the other keypad. Super annoying.

So, then I created a series of automations that would fire the Insteon scenes. The hardware responds appropriately with every device reacting simultaneously as expected. The problem I’m seeing is that for several seconds after that there are still events scrolling along in the HA log and the states of the various entities will start toggling. After 15-20 seconds, everything will settle down. But the states of the entities in HA will often not align with the state of the device.

What can I do make it all more efficient so it consistently and accurately picks up the state and does so within a second or two instead of 10-20 seconds? I’m so confused by all the traffic in the logs that I’m not even sure where to begin.

2

For example, I triggered the “both kids lights off” scene…
All four devices physically changed state within 1/2 second.
State in HA caught up one device at a time almost a minute later.

2020-02-17 13:40:25 INFO (MainThread) [homeassistant.helpers.script] Script Kids Room Lights Off: Running script
2020-02-17 13:40:25 INFO (MainThread) [homeassistant.helpers.script] Script Kids Room Lights Off: Executing step call service
2020-02-17 13:40:25 DEBUG (SyncWorker_0) [insteonplm.plm] Queueing msg: {'code': 0x62, 'address': 00.00.05, 'flags': 0xcf, 'cmd1': 0x13, 'cmd2': 0x00, 'acknak': 0xNone}
<<< 2312 lines removed >>>
2020-02-17 13:41:56 DEBUG (MainThread) [insteonplm.plm] Last item in self._recv_queue reached.
2020-02-17 13:41:56 DEBUG (MainThread) [insteonplm.plm] Processing _acknak_queue acknak: 139917818958096 msg: 139917844709456
2020-02-17 13:41:56 DEBUG (MainThread) [insteonplm.plm] ACK or NAK received

That is a full 91 seconds for HA to react to an Insteon scene. I assume having debug logging on slows this down a bit, but I can’t image it is adding over a minute to the execution time. If I try to interact with anything Insteon through HA during that minute and a half, those commands sit in queue and happen eventually.

3

I ended up scraping the UI card that fired insteon scenes.

Instead, I have an entity card that lets me directly control the lights and fan speeds with a dozen automations that update the LEDs on the wall keypad to match the state of the fanlinc when it detects a state change to either light or fan speed. There is a significant lag of 1-2 seconds when I change state before the wall pad updates. For instance, changing fan from high to off kills the fan motor, turns off the ‘high’ button on the wall pad, turns on the ‘off’ button on the wall pad, with a 1-2 second delay between each of those three actions. The other down side is the scene-based control allowed me to turn both lights on with a single button, but now I have two separate buttons for the two lights.

If the lag was in milliseconds I wouldn’t care, but in seconds it is super annoying and in minutes it is unusable. I still feel like it shouldn’t be responding this slow and that I have something broke in my config that is causing it to run this way.

This lag and the inability to turn on multiple devices simultaneously was what I was trying avoid by having the UI trigger insteon.scene_on instead of having HA interact with all the lights individually. But doing it that was lead to hardware reacting instantly and UI lagging by as much as a minute, and other problems. What am I doing wrong here?

4

Based on your description I assume you are using an Insteon Hub. The core issue is the way the hub works under the hood and the fact that scenes do not give feedback on a per device basis. Every time a device changes state manually or via a scene, the Hub poles EVERY device to see if it is a responder to that original device. I don’t know how many devices you have but the more you have the worse this problem becomes. It gets even worse if you have the Insteon app running as well since it refreshes too. So you get a double refresh response which backs up other commands. Currently HA does not see the Insteon App commands but the Insteon App sees the HA commands. This is being addressed in a future release that I currently have working in my test environment. This will speed up the process somewhat but the underlying issues still persist so you will not get instant response in HA but it will be as fast as the Insteon App updates.

5

I thought that was done as a single packet broadcast to every device. Does it really have a separate conversation with every device? That seems an inefficient way for the protocol to be designed.

What is the implication of this? Using the Insteon App will cause state in HA to be incorrect?

I REALLY appreciate all the work you put in on this Insteon integration. If there is anything I can do to help you test, troubleshoot, or otherwise, please let me know.

6

So the message to trigger the scene on/off is a single message. This is why you see the devices change state quickly and synchronously. What happens after that is three fold:

  1. HA sends a ‘cleanup’ message to each device in the scene that it knows about. When I say “that it knows about”, this means the All-Link Database of the device needs to be loaded for HA to know that the device is a responder to the scene. This is part of the Insteon design and cannot change but this process can take up to 2 seconds per device. So in your case 8 seconds.
  2. HA then sends a request to get the status of each device in the scene. This also can take 2 seconds per device, so another 8 seconds. This is because Insteon does not really give feedback regarding the device state after the scene is triggered so you have to ask the device for its status. This is “pessimistic” message handling. Some packages do optimistic handling, meaning they assume the device responded as it was told to and don’t ask the device for status. I designed this Insteon module for accuracy vs speed. It is a design decision but that could change.
  3. On a periodic basis or whenever the Insteon App sees a change to a device’s state, it will ask every device for status. I don’t just mean the devices in the scene. I mean all devices in the Hub All-Link Database. The Hub appears not to use its information about an All-Link Database and so it queries all devices on a regular basis to ensure it has the current state of each device. It feels very silly to me but I cannot fix that. This also takes 2 seconds per device so if you have 20 devices this can easily be 40 seconds. This occurs more often if you have the Insteon App open.

Regarding the Insteon module not seeing the Insteon App commands, yes, this means that the App will see state changes triggered by HA but HA does not see state changes triggered by the app, causing HA states to be incorrect.

7

I went away to work on something else for a bit and finally came back to this today.
After a bit of stress, I think I have everything playing nicely. I wanted to record what I did here for posterity in case anyone else wants to see the example (and so that if I ever have to recreate the config, there’s a backup here).

To recap: I have a 6-button keypad in wall and a Fanlinc in the fan linked together and also controlled via a Hub. I have a bunch of Scenes programmed in Insteon to keep the state of the two devices in sync.

The problem is that in HA I’m interacting with the light directly, which results in the keypad not matching the fan’s state. So, to keep the two in sync when controlling then from HA I created a set of automations to keep the state of the two devices in sync. There’s a 1-2 second lag before the keypad changes, but at least it catches up eventually.

#########
##  These automations update the state of the keypad to match what the HA just did to the light
##  Controls in HA interact with the light directly. 
##  These automations mirror the commands to the keypad so both change together.
#########
- alias: update keypad to match left light
  trigger:
    - platform: state
      entity_id: light.fanlinc_X_light 
      from: 'off'
      to: 'on'
  action:
    - service: light.turn_on
      entity_id: light.keypad_with_dimmer_X_main

- alias: update keypad to match left light
  trigger:
    - platform: state
      entity_id: light.fanlinc_X_light 
      from: 'on'
      to: 'off'
  action:
    - service: light.turn_off
      entity_id: light.keypad_with_dimmer_X_main

From the keypad, pressing ‘on’ and ‘off’ turns the light on the fan on and off quite nicely. The problem I ran into is that pressing the button on the keypad would send out a message that HA would intercept and update the keypad’s state, but the fanlinc as the responder either wasn’t sending out a message that it responded or HA just wasn’t getting it. The result was that using the keypad to turn the light on would show in HA as the keypad being on but it wasn’t showing that the actual light being controlled was on. So these automations update the state of the light in HA assuming that the light actually turned on when the keypad said it was pressed.

#########
##  These automations update the state of the light in HA to match what the hardware just did
##  The keypad is linked to light at hardware level, so when it is pressed the light turns on.
##  Insteon reports to HA that the keypad was pressed, but doesn't update HA that the light changed
#########
- alias: update light to match left keypad being turned off
  trigger:
    - platform: state
      entity_id: light.keypad_with_dimmer_X_main
      from: 'off'
      to: 'on'
  condition:
    # trigger only if light is not already on
    # this is necessary otherwise the action here triggers the other automation below
    - condition: state
      entity_id: light.fanlinc_X_light
      state: 'off'
  action:
    - service: python_script.set_entity_state
      data:
        entity_id: light.fanlinc_X_light
        state: 'on'      
        brightness: 255
        
- alias: update light to match left keypad being turned on
  trigger:
    - platform: state
      entity_id: light.keypad_with_dimmer_X_main
      from: 'on'
      to: 'off'
  action:
    - service: python_script.set_entity_state
      data:
        entity_id: light.fanlinc_X_light
        state: 'off'   
        brightness: 0

For the actions in the above, I just wanted to update the state rather than trigger a light.turn_on since the light was already on. I also ran into an issue where I needed to specify the brightness otherwise the entity-slider-card in the UI got jacked up when this automation turned the light on without also setting the brightness. So, the action that gets called is this script below, stolen from elsewhere in this forum.

#########
#  python_scripts/set_entity_state.py 
#  Set the state or other attributes for the entity specified in an Automation Action
#########
inputEntity = data.get('entity_id')
if inputEntity is None:
    logger.warning("===== entity_id is required if you want to set something.")
else:    
    inputStateObject = hass.states.get(inputEntity)
    inputState = inputStateObject.state
    inputAttributesObject = inputStateObject.attributes.copy()

    for item in data:
        newAttribute = data.get(item)
        logger.debug("===== item = {0}; value = {1}".format(item,newAttribute))
        if item == 'entity_id':
            continue            # already handled
        elif item == 'state':
            inputState = newAttribute
        else:
            inputAttributesObject[item] = newAttribute
        
    hass.states.set(inputEntity, inputState, inputAttributesObject)

Then there are these. They completely duplicate the scenes programmed in Insteon. When I push a button on keypad, the Insteon scene fires and does all this so that the four scene buttons on the keypad never show that the fan is in more than one state. But in HA I’m interacting directly with the fan, so I have a series of automations, one for each fan speed, to set the keypad to match the state we just set the fan to.

#########
##  These automations update the state of the keypad to match what the HA just did to fan
##  Four buttons on keypad map to fan speeds: off, low, medium, high. 
##  Scenes in insteon do this same action when button is pressed on keypad, but HA interacts with fan directly
##  so these automations replicate those scenes to make button lights match fan speed.
#########
- alias: Keypad to match left fan speed off
  trigger: 
    # when the button in HA is pressed to set fan speed to off
    platform: template
    value_template: "{{ is_state_attr('fan.fanlinc_X_fan', 'speed', 'off') }}"
  action:
    # set the state of each of the four keypad scene button lights so the off button is only one lit
    - service: switch.turn_off
      entity_id: switch.keypad_with_dimmer_X_button_a
    - service: switch.turn_off
      entity_id: switch.keypad_with_dimmer_X_button_b
    - service: switch.turn_off
      entity_id: switch.keypad_with_dimmer_X_button_c
    - service: switch.turn_on
      entity_id: switch.keypad_with_dimmer_X_button_d

# and then this repeats four times, once for each button with the on/off states changing as appropriate

There is undoubtedly a cleaner way to do all this. I went went brute force. I suspect through templates and scripts there is probably a way to combine multiple of the above automations together. But at this point it works and I have other things to work on. Also, above is for one fan. Everything repeats for each fan.

8

@taylormade I have released a new beta version of the Insteon component as a custom component. It is available for download and installation if you have the time to test it. Details can be found in this post:

9

This is awesome!

I’m running HA in Docker. The absence of gcc is preventing the install of pyinsteon. I’m investigating potential ways around this. As soon as I get a plan, I’ll get it installed and let you know how it looks.

Thanks for doing this! This is great.

10

This was fixed earlier this week. I was using a module called aiofile and now I am using aiofiles which does not require gcc to install it.

11

Tom,

First off thanks for all your work on integrating Insteon with Home Assistant!

I just installed your insteon2 beta plugin in a docker on a Synology Diskstation. The install went just fine.

I wanted your input on updating lights from triggering a scene. I have 3 dimmers as part of a scene. Once I use the scene to turn them off, the status of the individual lights do update, but they do it sequentially over the coarse of a few seconds. The Insteon app has all 3 lights updated as soon as I switch from the scene tab to the device tab. Is that all expected? Is there a way to get the HA app to update as fast as the Insteon app does?

Also, if I execute my scene from the Insteon app, the light status in HA will update after a few seconds. If I execute my scene from HA, the lights will change but they will not update in HA. If I open view the room in the Insteon App, the status of the lights will be wrong, but if I open the detail view of the light it appears to query the current status and it displays properly in both the App and in HA.

Is there a command I could add to my automations to make it go query specific light status so they will be updated correctly?

Thanks!
John Vickers

12

@jgvicke, first, welcome to Home Assistant.

There are a couple of items here so I will address them individually:

I have 3 dimmers as part of a scene. Once I use the scene to turn them off, the status of the individual lights do update, but they do it sequentially over the coarse of a few seconds.

I want to separate the physical lights and the status in HA. I would expect the physical lights to update all at the same time. I would then expect HA to update the status of the lights one at a time over the course of a few seconds (1 - 3 seconds per device). Is that what you are also seeing?

If I execute my scene from HA, the lights will change but they will not update in HA.

This is not what I would expect. Please check if the All-Link database is loaded for each of the devices in question. If the ALDB is loaded the device status should update in HA.

If I open view the room in the Insteon App, the status of the lights will be wrong, but if I open the detail view of the light it appears to query the current status and it displays properly in both the App and in HA.

This is all as expected (sort of). When ever the Insteon App queries a device for its status, HA will also see that status request and the response. So this way HA and the App should always be in sync. What is surprising, as stated above, is that HA is out of sync to begin with. The Insteon App does not try to keep up with device changes. It queries the device for status any time the user wants to know the device status. HA listens for device changes and updates based on what it hears. Scenes are a little different in that the device never tells anyone that it responded to the scene so HA sees that the scene was triggered, knows what devices should respond then asks the device if it did respond as expected.

Hopefully that all makes sense.

13

I think the biggest problem I was having was from using an older version of HA as a Synology package. To get the latest Insteon integration I moved to using the docker package and it is working much better. When I send the scenes now they do turn off as a set and will update inside the HA app within a few seconds.

I am going to try using the USB modem to see how that responds with scenes and I will post results here. Probably going to be about a week.

14

@teharris1 I was going to create a new thread, but I think my problem is very similar to what @taylormade was describing, but perhaps not exactly.

I was actually trying to do some work with my FanLinc as well, a device that uses 5 Insteon scenes. But it doesn’t seem to be the FanLinc that is giving me the biggest issues.

I’ve noticed over the last few days that Home Assistant isn’t reliably updating the status of buttons A-D on my 6 button KeypadLincs. Sometimes it’s instant, sometimes there’s a delay, and sometimes it takes so long to respond that I forget it even happens. I’ve noticed that any buttons directly connected to a load are pretty much instant when updating. It’s very strange. For example the keypad in my office A is lazy, B is instant, and C and D never trigger status in Home Assistant. I can toggle the LEDs on the button by switch.turn on and switch.turn off and it’s instant and visible in HA, but again, HA sometimes never sees the changes happening when you press them manually.

I am using a Hub, version 2. Still configured in Configuration.Yaml, I am worried that if I switch to the UI integration config that I will have to go through and rename 15 devices/38 entities.

insteon:
host: 192.168.9.91
ip_port: 25105
username:
password:
hub_version: 2

I’m wondering if switching to the hub out for PLM would fix this behavior. I’m a little scared about having to redo all my scenes, since the insteon-only scenes are working great, it’s just using buttons that aren’t assigned to scenes to control my other Home Assistant Entities that isn’t working. Also my HA runs in Hyper-V and I can’t pass through USB devices… so I’d probably have to either build a new server or plug the PLM into my rPI and use Ser2Net/Socat to connect.

Have you ever written something down, which makes you think about it in a different way? I just added a bunch of keypad buttons to scenes where they are the only member. Since doing that, they seem to respond in Home Assistant nearly instantly. I think this may be because adding the scene makes them talk to the hub in a more timely manner.