New Insteon PLM modem integration option via MQTT

I have taken the (stupid?) plunge and am in the process of factory-resetting every single device. There were too many out of date links, and being the geek that I am I felt the need to simplify, consolidate and start over…

My first step was to factory reset my PLM:
> insteon-mqtt config.yaml factory-reset

Following this, I am resetting each device one-by-one with the following procedure:
> insteon-mqtt print-db aa.bb.cc #want to see before state
reset device
> insteon-mqtt refresh -f aa.bb.cc #force db update on device
> insteon-mqtt join aa.bb.cc
> insteon-mqtt pair aa.bb.cc

At this point, everything appeared to be good. Doing a “print-db” on the PLM I see the recently paired device added to group 1, all good?

The problem is that when I manually push a switch (on or off), after about 2s the bottom LED flashes red. I believe this means it is unable to communicate with its controller? What am I doing wrong?

Here is the output of print-db on my PLM followed by a switch:

pi@Protocol-Pi:/opt/insteon-mqtt $ ./bin/insteon-mqtt config.yaml print-db modem
Commanding Modem device 49.f9.87 (modem) cmd=print_db
49.f9.87 modem database
ModemDb:
  ID: 3e.12.c0  grp: 1  type: CTRL  data: 0x01 0x00 0x00
  ID: 3e.12.c0  grp: 1  type: RESP  data: 0x01 0x00 0x00
GroupMap
  1 -> ['3e.12.c0']
pi@Protocol-Pi:/opt/insteon-mqtt $ ./bin/insteon-mqtt config.yaml print-db 3e.12.c0
Commanding dimmer device 3e.12.c0 (bar under cabinet) cmd=print_db
3e.12.c0 (bar under cabinet) device database
DeviceDb: (delta 5)
  0fef: 49.f9.87 grp:   1 type: CTRL data: 0x03 0x00 0x01
  0ff7: 49.f9.87 grp:   1 type: RESP data: 0xff 0x00 0x01
  0fff: 49.f9.07 grp:   1 type: RESP data: 0xff 0x1c 0x01
Unused:
Last:
  0fe7: 00.00.00 grp:   0 type: RESP data: 0x00 0x00 0x00 (UNUSED) (LAST)
GroupMap
  1 -> ['49.f9.87']

A follow-up question is where is the entry for ‘49.f9.07’ coming from? I have no such device on my network nor in my config.yaml.

Please disregard both of the above comments… found a possible error on my side. Will report back on confirmation.

As a follow-up to the above, I made a very stupid error. In reding the hex address of my PLM, I mistook the 0 for an 8. When I entered the incorrect address in the config.yaml file, insteon-mqtt was creating errant links as I had confused it.

Moral: read carefully. 8 <> 0

I’m finally testing the USB PLM and works.

I’m stuck and i fell dumb for not find a clear solution for me in two things:

  1. I don´t know how to create a scene, the documents didn’t explain it clearly (to me), so i have 2 switches manually linked but i don’t know how to invoque that scene from MQTT cos i don’t have the scene number (was manual setup) (update, noob comment, the scene number it’s only for the modem, i already know how they work in the device side only).

I need to know how link the scene with the PLM so the 3w can report to it when triggers.

  1. I have Insteon Covers, and that basically on/off switch latch with a define time (2 channels), i don’t know how to control them and setup the latching time. Anyone with this experience?

Thanks for teaching me.

update:

lucky me i tried with the scene/1 topic and works :expressionless:. But i need to know how to create them and define a group number.

Greetings all,

Thanks to each of you that have contributed to the development of this add on. I edited my request since it had changed and wouldn’t have been helpful as it was.

EDIT-
I just had another go at getting hass.io installed. I have it and Node-red and this integration running and talking together at lease some. That is progress. I went this way because there was a quickstart guide to get insteon-mqtt plug in going - assumption being that the plugin is available for the most recent / 0.95.4 hassio release. No suprise - it is not. It took quite a while to figure out what I needed to change to get this integration to work in with the newest hassio release.

looking at the issues on github for insteon-mqtt and figuring out that the path specified here:

https://github.com/TD22057/insteon-mqtt/blob/master/docs/hassio_quick_start.md
> Create a directory for insteon-mqtt in your /addons folder: mkdir /addons/insteon-mqtt

needs to be:

> mkdir /addons/local/insteon-mqtt

At this stage I still have nothing in terms of controllable devices showing up in HA. I think that is because I have not understood the process to construct a connection to a device through mqtt. It is probably obvious to everyone else, but to me it is not clear whether the information HA needs for insteon devices must be in HA’s configuration.yaml or will it parse what it needs from the config.yaml in this component? And what do I need to construct in HA to have ‘handles’ in the browser interface to actually control a dimmer or switch?

A few well chosen words and pasting some code would help greatly!

Thanks everyone.

Q

See if the following helps…
Think of HA and Insteon-mqtt as two separate systems with their own config files…and the two systems talk to each other via MQTT.

You tell Insteon-MQTT (via config.yaml) about Insteon Devices:

  devices:
    # On/off switch devices (outlets, wall switches, appliance modules, etc).
    switch:
      - 12.34.56: 'My Insteon Switch'

and…you tell HA about MQTT devices (via HA’s configuration.yaml):

switch:
  - platform: mqtt
    name: "My HA Switch"
    state_topic: "insteon/12.34.56/state"
    command_topic: "insteon/12.34.56/set"

If you look further in insteon-mqtt’s config.yaml, it explains how to setup HA’s configuration.yaml

HA doesn’t know anything about Insteon in this case. It only knows about MQTT topics to ascertain the state of some device and command the device to do something.

Insteon-MQTT likewise doesn’t know anything about HA, it only knows that if it receives a command topic it in turn is to perform an action to the Insteon device, and send a state topic to the broker to update a device’s state.

Thanks Willy,

I did read ‘further’ and I guess some of it hasn’t sunk in. It was clear that mqtt is a dfferent route and since I understand a little more about the direct communication with plm and a whole lot less about HA I was uncertain and no doubt will have more difficulty making sense of it all as I progress.

Thanks for the clarification and example. This will help a lot.

There is one other question that comes to mind though. Your example is for a “switch:” This is not distinct from a “dimmer:” correct? “My HA Switch” could be a dimmer that would have additional valid states it will accept or report -eg 70 % vs just 0/1 or off/on for a simple switch. Though on and off are valid for a dimmer as well. So I might call a dimmer “My HA Dimmer” but it would still need to be in under the “switch:” category correct?

Regards,

Q

Just to follow up, thanks again Willy, I was able to turn on and of a light connected to a keypadlinc with your help! It is so great to be making progress!

configuration.yaml

light:
  - platform: mqtt
    schema: json
    name: "Desktop_KP_w_DIM_A"
    state_topic: "insteon/1e.f8.83/state/1"
    command_topic: "insteon/1e.f8.83/set/1"

and config.yaml

    keypad_linc:
      - 1e.f8.83: 'Desktop_KP_w_DIM_A'

Thanks so much! Not just to you but also to TD22057 and lnr0626 for giving us this integration!

Regards to all.

Q

Be aware that you shouldn’t use “/1” (or anything else) at the end of the state and command topics in the configuration.
HA and Insteon-MQTT take care of that part. For example If you were to monitor MQTT and have HA turn on a device, you’ll see HA sends out the “set” topic and fills it out with “ON” and Insteon-MQTT respond with the “state” topic and it fills it out “ON”

MQTT messages:

insteon/12.34.56/set ON
insteon/12.34.56/state ON

Thanks again Willy.

You say:

More confusion for me as I strugge through the config.yaml file. What I tested and posted was lifted from the keypadlinc section round about line 700 (here):

#   switch:
  #     - platform: mqtt
  #       state_topic: 'insteon/aa.bb.cc/state/1'
  #       command_topic: 'insteon/aa.bb.cc/set/1'

Were you not taking into account that I was dealing with a keypadlinc? Or am I even more confused than I think?

The instructions in the file there are a bit confusing because they indicate that - all other buttons except “1” are “2” (paraphrase). I don’t think this is correct. Any value from 2 to 8 would be more correct, or at least clearer, with the value corresponding to the desired button. Have I got this right? It seems to work this way for me.

There is another issue to possibly run afoul of here when dealing with the 6 button keypads if memory serves. A mode where the big buttons top and bottom correspond to on and off or the bottom button can be independent. I will have to do some testing on that because I have several 6 button keypads…

I am glad to have your help and the explanations as they are. I truly understand that there is a tension between code and the documentation thereof, especially in this setting. So it is all appreciated - especially the help over the rough patches that can be had here!

Thanks again.

Q

A little follow up. Part of my confusion is the format “insteon/aa.bb.cc/state/1” It would seem ‘more correct’ and understandable to have “insteon/aa.bb.cc/Button number/state” instead of putting the “state” in the middle… Confused!!

Sorry, No I was not considering Keypadlinc. My mistake.
Yeah, I can see why you would want button before state and not after.

I was away for a week and when I came back I found that none of my Insteon devices were responding anymore. Taking a look at the insteon-mqtt logs, I see repeated entries like:

2019-08-26 21:12:54 WARNING Base: Handler timed out 1 of 3 sent: Std: 42.79.2c, Type.DIRECT, 19 00 
2019-08-26 21:12:54 DEBUG Base: Increasing max_hops to 3
2019-08-26 21:12:54 INFO Protocol: Write message to modem: Std: 42.4a.8a, Type.DIRECT, 19 00 
2019-08-26 21:12:54 DEBUG Protocol: Write bytes to modem: b'\x02bBJ\x8a\x0f\x19\x00'
2019-08-26 21:12:54 DEBUG Serial: Wrote 8 bytes to serial /dev/ttyUSB0

Looking at my PLM, I see that the LED is red.

Does this mean the PLM is hosed? Is there another explanation?

Does this PLM Modem integration support the X10 protocol also or only the insteon protocol?

Yes, the PLM can speak both Insteon and X10.

I ended up getting a new PLM to replace my (broken?) one. When I plugged it into the wall socket, the LED was green. I’m going to take that as meaning the previous one died.

I’m now trying to figure out how to migrate all of my Insteon devices over to the new PLM. Here are the steps I’ve taken:

  1. Pull USB cable from old PLM, then disconnect from wall socket.
  2. Verify using journalctl -f that the PLM was disconnected
  3. Plug new PLM into wall socket, then existing (same) USB cable.
  4. Verify using journalctl -f that the new PLM was discovered.
  5. Update the modem address in config.yaml
  6. For good measure, reboot system

All of my devices are of course linked with the old PLM. I would like to pair them with the new one without resetting them all as many of them are inaccessible (fanlincs). However, when I issue the following commands, nothing happens:

pi@Protocol-Pi:/opt/insteon-mqtt $ ./bin/insteon-mqtt config.yaml join 42.72.3f
Reply timed out
pi@Protocol-Pi:/opt/insteon-mqtt $ ./bin/insteon-mqtt config.yaml pair 42.72.3f
Reply timed out

Same thing if I try any of the other commands:

pi@Protocol-Pi:/opt/insteon-mqtt $ ./bin/insteon-mqtt config.yaml refresh 42.72.3f
Reply timed out

Please help!

Solved! I forgot that you need to have the daemon running to issue the individual commands… Now off to re-pair all of my devices.

You can probably fix your old PLM. The caps in there were garbage, I’ve repaired mine, have a article on it: https://1projectaweek.com/blog/2017/6/2/repairing-a-failed-insteon-powerlinc-power-line-modem-plm-2413u

1 Like

(SOLVED see bottom of this post and an update later) My problem, briefly stated is that I cannot control 2 plugin modules from the default view. They are present but status from and commands to the modules are ineffective. … I am sorry if this post is unnecessarily long but it is tough to trim to only ‘relevant’ parts with my current level of understanding.

For testing purposes I am focusing on using 2 lamp plugins, a keypad dimmer in a desktop enclosure and the PLM. All devices I have were set up with the now abandoned HouseLinc software and working with it. …snip…

…snip… HA is showing devices in overview as expected and for at least a few, status and control are working. However I am having trouble with the lamp modules I have chosen for testing. This is the error I get in the ad-don log:

2019-10-13 14:42:26 DEBUG Dimmer: Dimmer message insteon/1a.e0.bd/set b'{"state": "ON"}'
2019-10-13 14:42:26 DEBUG MsgTemplate: Input template render: '{ "cmd" : "{"state": "on"}" }'
2019-10-13 14:42:26 ERROR MsgTemplate: Invalid JSON message { "cmd" : "{"state": "on"}" } from template { "cmd" : "{{value.lower()}}" }
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/insteon_mqtt/mqtt/MsgTemplate.py", line 177, in to_json
    return json.loads(value)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 14 (char 13)
2019-10-13 14:42:26 INFO Dimmer: Dimmer input command: None
2019-10-13 14:42:26 ERROR Dimmer: Invalid switch on/off command: None
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/insteon_mqtt/mqtt/Dimmer.py", line 224, in _input_on_off
    is_on, mode = util.parse_on_off(data)
  File "/usr/lib/python3.6/site-packages/insteon_mqtt/mqtt/util.py", line 30, in parse_on_off
    cmd = data.get('cmd').lower()
AttributeError: 'NoneType' object has no attribute 'get'

Prior, I went tried using the directions here:https://github.com/TD22057/insteon-mqtt/blob/42f3933566fb2bcdbe80de7e6f5ff8ff35ecdea0/docs/mqtt.md and went to my HA at: …:8123/developer-tools/mqtt page. I sent;

insteon/command/1f.15.3a
{ "cmd": "join" }

Join gives the following result:

2019-10-09 11:16:42 INFO Mqtt: MQTT message insteon/command/1a.e0.bd b'{ "cmd": "join" }'
2019-10-09 11:16:42 UI Mqtt: Commanding dimmer device 1a.e0.bd (plugin_a) cmd=join
2019-10-09 11:16:42 INFO Base: Join Device 1a.e0.bd
2019-10-09 11:16:42 DEBUG CommandSeq: Running command 1 of 2
2019-10-09 11:16:42 INFO Base: Device 1a.e0.bd (plugin_a) cmd: get engine version
...snip...
2019-10-09 11:16:49 INFO Protocol: Read 0x53: All link done: 1a.e0.bd grp: 1 Cmd.CONTROLLER cat: 0x01 0x0e 0x41
2019-10-09 11:16:49 UI ModemLinkComplete: Modem db entry added via linking: 1a.e0.bd grp 1 

This all appears ok and has been done numerous times.

Then pair:

insteon/command/1a.e0.bd
{ "cmd": "pair" }

Results:

...snip...
2019-10-09 11:28:54 INFO Protocol: Write message to modem: OutAllLinkUpdate: 1a.e0.bd grp: 1 Cmd.UPDATE
2019-10-09 11:28:54 DEBUG Protocol: Write bytes to modem: b'\x02o \xe2\x01\x1a\xe0\xbd\x01\x00\x00'
2019-10-09 11:28:54 DEBUG Mqtt: MQTT writing
2019-10-09 11:28:54 DEBUG Serial: Wrote 11 bytes to serial /dev/ttyUSB0
2019-10-09 11:28:54 INFO Protocol: Read 0x6f: OutAllLinkUpdate: 1a.e0.bd grp: 1 Cmd.UPDATE ack: False
2019-10-09 11:28:54 DEBUG Protocol: Passing msg to write handler: ModemDbModify handler
2019-10-09 11:28:54 ERROR ModemDbModify: Modem db updated failed: OutAllLinkUpdate: 1a.e0.bd grp: 1 Cmd.UPDATE ack: False
2019-10-09 11:28:54 ERROR Mqtt: Modem database update failed
2019-10-09 11:28:54 DEBUG Protocol: Write handler finished

I see a “fail” so I reasonably conclude there is a problem here - but what do I know? I am a noob. Restarts,reboots and retries do not improve this. So I push on.

Note: Folowing the documentation here: https://github.com/TD22057/insteon-mqtt/blob/42f3933566fb2bcdbe80de7e6f5ff8ff35ecdea0/docs/mqtt.md Which says to do this for every device (no exceptions) so :

insteon/command/1a.e0.bd 
 { "cmd": "sync" }

This fails. “sync” is indicated to be an invalid command. Later in the same document I read:

Sync Device Links (Placeholder)
Supported: modem, device

This command is not yet supported.

Perhaps it would be better to remove the initial reference to an apparently unsupported requirement to reduce the confusion factor for newbs like me since some of us do try to RTFM even though we don’t understand a lot of it.

2019-10-09 11:53:32 INFO Mqtt: MQTT message insteon/command/1a.e0.bd b'{ "cmd": "sync" }'
2019-10-09 11:53:32 UI Mqtt: Commanding dimmer device 1a.e0.bd (plugin_a) cmd=sync
2019-10-09 11:53:32 ERROR Mqtt: Unknown command 'sync' for device type dimmer.  Valid commands: dict_keys(['db_add_ctrl_of', 'db_add_resp_of', 'db_del_ctrl_of', 'db_del_resp_of', 'print_db', 'refresh', 'linking', 'join', 'pair', 'get_flags', 'get_engine', 'get_model', 'on', 'off', 'set', 'increment_up', 'increment_down', 'scene', 'set_flags'])

Back to the core issue. Attempting to turn on lamp module from HA overview page fails.

light:
  - platform: mqtt
    schema: json
    name: "Desktop_KP_w_DIM_A"
    state_topic: "insteon/1e.f8.83/state/"
    command_topic: "insteon/1e.f8.83/set/"

Solution at least in part: command_topic: “insteon/1e.f8.83/set/” is incorrect. Set should be level for a dimming lamp module. As well, scrupulous attention to case. I still had a few devices with upper case address characters!

Thanks to all.

Q

@Quanta: Look at the first error - it says that the JSON is invalid - it has nothing to with joining or pairing. HA sends an MQTT message to IM which passes it through Jinja templates to get a json command structure. There is a mismatch in what HA is sending vs what the IM templates are expecting. It could be the examples are wrong w/ the new json output mode in HA or your config is incorrect. I haven’t upgraded in awhile (busy with other jobs) but start there. Look at the IM config and see what it’s expecting, then look at the MQTT message that HA is sending. Either adjust the HA config or the IM config to match. I’ll try and find time later this week to upgrade to the latest HA and retest things.