Insteon devices got slow / stopped responding

Hardware
1

Hello
I have been using Insteon via Home Assistant for a long time, and all this time it has worked quite well. (Fun note: I did not notice Insteon servers going down until I read it in news, thanks to Home Assistant.)

My Insteon setup, using docker and USB modem, has started giving me some trouble lately. Specifically the lights are responding either very slowly or not at all (aka slower than my patience allows me to wait). This has been going on for a few weeks now.

I looked at the logs today, and looks like I am getting a bunch of NAKs from almost all lights. Here is beginning of a log:

2023-04-30 19:07:02.629 DEBUG (MainThread) [homeassistant.components.insteon] Insteon device count: 34
2023-04-30 19:07:02.629 DEBUG (MainThread) [homeassistant.components.insteon.utils] Insteon Services registered
2023-04-30 19:07:02.648 DEBUG (MainThread) [pyinsteon.topics] Topic: send.status_request.direct data: {'address': 43XXe5, 'status_type': 2}
2023-04-30 19:07:02.663 DEBUG (MainThread) [pyinsteon.messages] TX: msg_id: 0x62, address: 43XXe5, flags: 0x0f, cmd1: 0x19, cmd2: 0x02
2023-04-30 19:07:02.701 DEBUG (MainThread) [pyinsteon.messages] RX: msg_id: 0x62, address: 43XXe5, flags: 0x0f, cmd1: 0x19, cmd2: 0x02, ack: 0x06
2023-04-30 19:07:02.701 DEBUG (MainThread) [pyinsteon.topics] Topic: ack.43XXe5.status_request.direct data: {'cmd1': 25, 'cmd2': 2, 'user_data': None}
2023-04-30 19:07:08.702 DEBUG (MainThread) [pyinsteon.topics] Topic: send.status_request.direct data: {'address': 43XXf6, 'status_type': 2}
2023-04-30 19:07:08.703 DEBUG (MainThread) [pyinsteon.messages] TX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02
2023-04-30 19:07:08.719 DEBUG (MainThread) [pyinsteon.messages] RX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02, ack: 0x15
2023-04-30 19:07:08.719 DEBUG (MainThread) [pyinsteon.topics] Topic: nak.43XXf6.status_request.direct data: {'cmd1': 25, 'cmd2': 2, 'user_data': None}
2023-04-30 19:07:08.921 DEBUG (MainThread) [pyinsteon.topics] Topic: send.status_request.direct data: {'address': 43XXf6, 'status_type': 2}
2023-04-30 19:07:09.506 DEBUG (MainThread) [pyinsteon.messages] TX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02
2023-04-30 19:07:09.519 DEBUG (MainThread) [pyinsteon.messages] RX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02, ack: 0x15
2023-04-30 19:07:09.519 DEBUG (MainThread) [pyinsteon.topics] Topic: nak.43XXf6.status_request.direct data: {'cmd1': 25, 'cmd2': 2, 'user_data': None}
2023-04-30 19:07:09.924 DEBUG (MainThread) [pyinsteon.topics] Topic: send.status_request.direct data: {'address': 43XXf6, 'status_type': 2}
2023-04-30 19:07:10.307 DEBUG (MainThread) [pyinsteon.messages] TX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02
2023-04-30 19:07:10.319 DEBUG (MainThread) [pyinsteon.messages] RX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02, ack: 0x15
2023-04-30 19:07:10.320 DEBUG (MainThread) [pyinsteon.topics] Topic: nak.43XXf6.status_request.direct data: {'cmd1': 25, 'cmd2': 2, 'user_data': None}
2023-04-30 19:07:10.921 DEBUG (MainThread) [pyinsteon.topics] Topic: send.status_request.direct data: {'address': 43XXf6, 'status_type': 2}
2023-04-30 19:07:11.109 DEBUG (MainThread) [pyinsteon.messages] TX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02
2023-04-30 19:07:11.120 DEBUG (MainThread) [pyinsteon.messages] RX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02, ack: 0x15
2023-04-30 19:07:11.120 DEBUG (MainThread) [pyinsteon.topics] Topic: nak.43XXf6.status_request.direct data: {'cmd1': 25, 'cmd2': 2, 'user_data': None}
2023-04-30 19:07:11.923 DEBUG (MainThread) [pyinsteon.topics] Topic: send.status_request.direct data: {'address': 43XXf6, 'status_type': 2}
2023-04-30 19:07:11.923 DEBUG (MainThread) [pyinsteon.messages] TX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02
2023-04-30 19:07:11.936 DEBUG (MainThread) [pyinsteon.messages] RX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02, ack: 0x15
2023-04-30 19:07:11.936 DEBUG (MainThread) [pyinsteon.topics] Topic: nak.43XXf6.status_request.direct data: {'cmd1': 25, 'cmd2': 2, 'user_data': None}
2023-04-30 19:07:12.938 DEBUG (MainThread) [pyinsteon.topics] Topic: send.status_request.direct data: {'address': 43XXf6, 'status_type': 2}
2023-04-30 19:07:12.939 DEBUG (MainThread) [pyinsteon.messages] TX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02
2023-04-30 19:07:12.960 DEBUG (MainThread) [pyinsteon.messages] RX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02, ack: 0x15
2023-04-30 19:07:12.960 DEBUG (MainThread) [pyinsteon.topics] Topic: nak.43XXf6.status_request.direct data: {'cmd1': 25, 'cmd2': 2, 'user_data': None}
2023-04-30 19:07:14.161 DEBUG (MainThread) [pyinsteon.topics] Topic: send.status_request.direct data: {'address': 43XXf6, 'status_type': 2}
2023-04-30 19:07:14.162 DEBUG (MainThread) [pyinsteon.messages] TX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02
2023-04-30 19:07:14.176 DEBUG (MainThread) [pyinsteon.messages] RX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02, ack: 0x15
2023-04-30 19:07:14.176 DEBUG (MainThread) [pyinsteon.topics] Topic: nak.43XXf6.status_request.direct data: {'cmd1': 25, 'cmd2': 2, 'user_data': None}
2023-04-30 19:07:15.537 DEBUG (MainThread) [pyinsteon.messages] RX: msg_id: 0x50, address: 43XXe5, target: 49f922, flags: 0x2f, cmd1: 0x19, cmd2: 0x00
2023-04-30 19:07:15.537 DEBUG (MainThread) [pyinsteon.topics] Topic: 43XXe5.status_request.direct_ack data: {'cmd1': 25, 'cmd2': 0, 'target': 49f922, 'user_data': None, 'hops_left': 3}
2023-04-30 19:07:15.578 DEBUG (MainThread) [pyinsteon.topics] Topic: send.status_request.direct data: {'address': 43XXf6, 'status_type': 2}
2023-04-30 19:07:15.578 DEBUG (MainThread) [pyinsteon.messages] TX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02
2023-04-30 19:07:15.600 DEBUG (MainThread) [pyinsteon.messages] RX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x19, cmd2: 0x02, ack: 0x06
2023-04-30 19:07:15.601 DEBUG (MainThread) [pyinsteon.topics] Topic: ack.43XXf6.status_request.direct data: {'cmd1': 25, 'cmd2': 2, 'user_data': None}
2023-04-30 19:07:15.985 DEBUG (MainThread) [pyinsteon.messages] RX: msg_id: 0x50, address: 43XXf6, target: 49f922, flags: 0x2f, cmd1: 0x19, cmd2: 0xff
2023-04-30 19:07:15.985 DEBUG (MainThread) [pyinsteon.topics] Topic: 43XXf6.status_request.direct_ack data: {'cmd1': 25, 'cmd2': 255, 'target': 49f922, 'user_data': None, 'hops_left': 3}

(I redacted part of the device ID; I also have a lot more log, if needed)

What could be the issue?

Thanks!

P.S. I “THINK” that the beginning of issue coincides with me getting some emails for development going on in pyinsteon repo; I can’t be sure.

2

So I reset my entire Insteon network and added switches one by one.
The issue is now gone, but 1 of the switch is not reporting its status because I am unable to “Add Default Links”.

Anytime I try to read the ALDB for this switch, pyinsteon throws the following error:

2023-05-19 18:16:50.711 DEBUG (MainThread) [pyinsteon.topics] Topic: send.set_address_msb.direct data: {'address': 43XXf6, 'high_byte': 0}
2023-05-19 18:16:50.711 DEBUG (MainThread) [pyinsteon.messages] TX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x28, cmd2: 0x00
2023-05-19 18:16:50.732 DEBUG (MainThread) [pyinsteon.messages] RX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x28, cmd2: 0x00, ack: 0x06
2023-05-19 18:16:50.732 DEBUG (MainThread) [pyinsteon.topics] Topic: ack.43XXf6.set_address_msb.direct data: {'cmd1': 40, 'cmd2': 0, 'user_data': None}
2023-05-19 18:16:50.732 DEBUG (MainThread) [pyinsteon.topics] Topic: send.set_address_msb.direct data: {'address': 43XXf6, 'high_byte': -1}
2023-05-19 18:16:51.034 DEBUG (MainThread) [pyinsteon.messages] RX: msg_id: 0x50, address: 43XXf6, target: 49f922, flags: 0x2f, cmd1: 0x28, cmd2: 0x00
2023-05-19 18:16:51.034 DEBUG (MainThread) [pyinsteon.topics] Topic: 43XXf6.set_address_msb.direct_ack data: {'cmd1': 40, 'cmd2': 0, 'target': 49XX22, 'user_data': None, 'hops_left': 3}
2023-05-19 18:16:51.088 DEBUG (MainThread) [pyinsteon.topics] Topic: handler.43XXf6.set_address_msb.direct data: {'high_byte': 0}
2023-05-19 18:16:51.191 DEBUG (MainThread) [pyinsteon.topics] Topic: send.peek.direct data: {'address': 43XXf6, 'lsb': 0}
2023-05-19 18:16:51.513 DEBUG (MainThread) [pyinsteon.messages] TX: msg_id: 0x62, address: 43XXf6, flags: 0x0f, cmd1: 0x28, cmd2: 0x-1
2023-05-19 18:16:51.513 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pyinsteon/protocol/protocol.py", line 235, in _write_messages
    await self._transport.async_write(msg)
  File "/usr/local/lib/python3.10/site-packages/pyinsteon/protocol/serial_transport.py", line 69, in async_write
    self.write(data)
  File "/usr/local/lib/python3.10/site-packages/pyinsteon/protocol/serial_transport.py", line 64, in write
    msg_bytes = bytes(data)
  File "/usr/local/lib/python3.10/site-packages/pyinsteon/protocol/messages/__init__.py", line 36, in __bytes__
    return vars_to_bytes(data)
  File "/usr/local/lib/python3.10/site-packages/pyinsteon/utils.py", line 104, in vars_to_bytes
    msg.extend(bytes([val]))
ValueError: bytes must be in range(0, 256)

Any ideas why it is trying to send a negative 1 for command?