UPB lighting

[123 (Taras)]

Can I just remove the part that renames the event or do I have to close this PR and submit a new one?

[gwww (Glenn Waters)]

I don’t know as I rarely use the edit function. I suspect just hit the edit button, make the change, and everything will work as hoped. Generally speaking, it won’t let you do anything “stupid”, which is why I think that the edit button will just work.

[123 (Taras)]

I believe I have found where I can edit the file but are you certain I still need to revert it? Your PR appears to be progressing; Martin made the following comment several hours ago:
https://github.com/home-assistant/core/pull/35644#discussion_r425612962

[gwww (Glenn Waters)]

Good point, I misread his comment. I’ve made the change. Guess you have nothing more at the moment.

[gwww (Glenn Waters)]

The code PR is merged! Just waiting for the docs. Yipee! Congrats @123 for your first PR!

[123 (Taras)]

Well thank you but it took far more hand-holding than I anticipated and the resulting PR still managed to target the wrong branch. Sheesh!

Thank you for your assistance. I look forward to testing the UPB integration in 0.110. It represents the first major integration that I will be moving from Premise to Home Assistant.

---

UPDATE

Documentation PR was merged.

[jameshilliard]

I now have full register decoding working(on one device so far but building out for the remaining devices should be fairly straight forward).

DEBUG:__main__:Device 1:5 registers: 
01:05:00:00:00:01:00:04:00:1d:01:08:00:00:00:00
20:20:20:20:20:20:20:20:20:20:20:20:20:20:20:20
4f:66:66:69:63:65:20:20:20:20:20:20:20:20:20:20
52:69:67:68:74:20:20:20:20:20:20:20:20:20:20:20
01:64:ff:02:00:ff:03:32:ff:04:4b:03:05:19:ff:ff
64:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff
ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:f1:64:ff
ff:00:ff:ff:00:ff:ff:00:ff:ff:00:ff:ff:00:ff:ff
ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:fc:09:0b:f1:fe:fa
f1:66:88:33:44:f1:55:77:22:44:ff:ff:ff:ff:ff:ff
ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff
ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff
22:00:ff:22:64:ff:23:00:05:23:64:05:24:ff:ff:21
ff:ff:20:ff:ff:22:00:00:22:64:00:22:00:01:22:64
01:22:00:08:22:64:08:25:1e:ff:00:00:00:00:00:00
00:00:00:00:00:00:00:00:ff:00:a6:00:ff:55:00:00

DEBUG:__main__:Device 1:5: {'device_name': b'Right           ',
 'dim_options': 241,
 'firmware_major_version': 1,
 'firmware_minor_version': 8,
 'led_options': 9,
 'link_ids': [1, 100, 255, 2, 0, 255, 3, 50, 255, 4, 75, 3, 5, 25, 255, 255],
 'manufacturer_id': 4,
 'module_id': 5,
 'net_id': 1,
 'network_name': b'                ',
 'password': 0,
 'preset_fade_table': [255,
                       255,
                       255,
                       255,
                       255,
                       255,
                       255,
                       255,
                       255,
                       255,
                       255,
                       255,
                       255,
                       241,
                       100,
                       255],
 'preset_level_table': [100,
                        255,
                        255,
                        255,
                        255,
                        255,
                        255,
                        255,
                        255,
                        255,
                        255,
                        255,
                        255,
                        255,
                        255,
                        255],
 'product_id': 29,
 'reserved1': [255],
 'reserved2': [255, 255, 255, 255, 255, 255, 255, 255, 34],
 'rocker_action': [{'bottom_rocker_double_click': 119,
                    'bottom_rocker_hold': 34,
                    'bottom_rocker_release': 68,
                    'bottom_rocker_single_click': 85,
                    'bottom_rocker_tid': 241,
                    'top_rocker_double_click': 136,
                    'top_rocker_hold': 51,
                    'top_rocker_release': 68,
                    'top_rocker_single_click': 102,
                    'top_rocker_tid': 241},
                   {'bottom_rocker_double_click': 255,
                    'bottom_rocker_hold': 255,
                    'bottom_rocker_release': 255,
                    'bottom_rocker_single_click': 255,
                    'bottom_rocker_tid': 255,
                    'top_rocker_double_click': 255,
                    'top_rocker_hold': 255,
                    'top_rocker_release': 255,
                    'top_rocker_single_click': 255,
                    'top_rocker_tid': 255},
                   {'bottom_rocker_double_click': 255,
                    'bottom_rocker_hold': 255,
                    'bottom_rocker_release': 255,
                    'bottom_rocker_single_click': 255,
                    'bottom_rocker_tid': 255,
                    'top_rocker_double_click': 255,
                    'top_rocker_hold': 255,
                    'top_rocker_release': 255,
                    'top_rocker_single_click': 255,
                    'top_rocker_tid': 255},
                   {'bottom_rocker_double_click': 255,
                    'bottom_rocker_hold': 255,
                    'bottom_rocker_release': 255,
                    'bottom_rocker_single_click': 255,
                    'bottom_rocker_tid': 255,
                    'top_rocker_double_click': 255,
                    'top_rocker_hold': 255,
                    'top_rocker_release': 255,
                    'top_rocker_single_click': 255,
                    'top_rocker_tid': 255}],
 'rocker_config': 11,
 'rocker_options': 250,
 'rocker_transmit_options': 252,
 'room_name': b'Office          ',
 'serial_number': 0,
 'transmission_options': 254,
 'upb_options': 0,
 'upb_version': 1}
DEBUG:__main__:manufacturer = SimplyAutomated, product = SA_US2_40

[jameshilliard]

I think I’ve now got all the register maps extracted from upstart, I’ve pushed an update of my library to pypi.

It would be helpful if some people can verify it decodes the registers on different device types, as before it can be installed and run via:

python3 -m pip install upb
python3 -m upb.tools.dumpreg --host=192.168.1.32 --network=1 --device=5

The device register maps are here while the mappings for the manufacture/product ID’s are here.

[jameshilliard]

@123

Automatic password discovery

I figured out how premise is doing this, it’s quite clever, they are exploiting a flaw in the UPB protocol which makes it possible to massively reduce the amount of requests required to brute force the network password.

Reading the UPB documentation one would assume there are a maximum of 65536 passwords you would have to try, which would take a very long time with how slow the UPB protocol is.

However by using the getdevicesignature command one can get the checksum for the registers which surprisingly is computed based on all register values even when not in setup mode, including the password registers which are not returned by the getregistervalues command unless in setup mode.

Because of this design flaw we can compute the sum of the two bytes that contain the 16 bit password by subtracting the computed register checksum(which doesn’t contain the password registers unless in setup mode) from the real checksum(which always covers the password registers) returned by getdevicesignature, this reduces the amount of possible passwords we have to try from 65536 to at most 256(in practice it should be much less than this most of the time).

I’ve currently implemented the checksum computations needed for this and should have this feature fully implemented shortly, I just need to make it brute force the possible passwords obtained by subtracting the computed register checksum from the real register checksum.

[123 (Taras)]

Great detective work!

It’s becoming increasingly clear that the unknown author of Premise’s UPB driver (only known by the moniker “GeorgeS”) had keen insight into the UPB protocol (back in 2006). Fortunately for us, that knowledge wasn’t lost, just tucked away in many lines of C++ code, waiting to be re-discovered.

When I have a chance, I’ll try your library and report my results.

[jameshilliard]

@123

It’s becoming increasingly clear that the unknown author of Premise’s UPB driver (only known by the moniker “GeorgeS”) had keen insight into the UPB protocol (back in 2006). Fortunately for us, that knowledge wasn’t lost, just tucked away in many lines of C++ code, waiting to be re-discovered.

Yep, the premise code is very professionally written, it’s cleaner than the upstart code and extremely well commented, the upstart code release is not quite as clean or well commented but I’ve now gotten fairly familiar with it so that I can use it to fill in various knowledge gaps not covered by the premise code. I’ve been using both extensively as a reference for understanding how to best implement certain features and serialize/parse various UPB data.

I think I know why the premise developers decided to automatically crack the network passwords, due to the same checksum bug that makes cracking the password fast it’s not actually possible to validate the register checksum unless you also know the password. I bet the premise devs were trying to figure out why their register checksums weren’t validating correctly and ended up implementing the password crack capability as a workaround in order to make it possible to validate the register checksums. From my reading of the upstart code it doesn’t appear upstart validates the register checksums at all due to this issue.

@webmtn

So, I’m revising my earlier comment to say that there will probably be some pretty good commonality in the memory registers, but be careful as all the vendors have slightly different feature sets that will impact the memory map.

So there were a lot more different register device memory maps than I originally expected, this would have been a big problem if it were not for the upstart code release that has a header file that contains what appears to be the complete memory map tables for every single production UPB device ever created(along with some that may not have even made it to production).

In addition there is also seemingly complete MID/DID mappings for the manufacturer id and device id pairs which allows associating devices to their correct device class specific register memory map table.

I was able to import all these memory mapping tables into my python library using ctypes which provides a nice clean abstraction layer for reading/writing to them, in total I currently count 18 distinct memory map tables used by different UPB devices(there are many more UPB devices than this as a lot of these maps are shared across various devices and manufacturers).

[jameshilliard]

I now have automatic password discovery working as part of the register dump process, there’s a few optimizations I still need to do to speed it up but it’s relatively fast even when brute force searching nearly all 256 possible passwords.

[bbrendon]

Is anyone running the UPB code in Core? I’m tempted but nervous about the changes I’ll have to make to my automations.

[123 (Taras)]

@gwww
I have tried the release version of the UPB integration and have a failure to report. Shall I post it in this thread or start a new one?

[gwww (Glenn Waters)]

Start with posting here.

[123 (Taras)]

I’m testing the UPB integration on a fresh installation of Home Assistant 0.110.0 on an RPI3 (i.e. clean test-bed; default configuration; no other integrations). I’m using a ‘known good’ serial-to-USB cable and a Simply Automated Serial PIM (in Message mode). The cable and PIM were used for testing the beta version of the UPB integration.

Upon plugging the cable into the RPI3, it is detected by HassOS and reported in Supervisor > System > Hardware as two devices (by name and by port number):

serial:
    /dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_D-if00-port0
    /dev/ttyUSB0

I copied the UPB configuration file to the RPI3 in /config/upb.upe.

Here’s a screenshot of the initial config flow screen:

After submitting the data, several seconds pass and then it returns with the following error:

Failed to connect to UPB PIM, please try again.

Screenshot:

The connection failure is reported in the log:

2020-05-22 12:42:27 WARNING (MainThread) [upb_lib.upb] Could not connect to UPB PIM ([Errno 2] could not open port /dev/ttyUSB0: [Errno 2] No such file or directory: '/dev/ttyUSB0'). Retrying in 1 seconds
2020-05-22 12:42:28 WARNING (MainThread) [upb_lib.upb] Could not connect to UPB PIM ([Errno 2] could not open port /dev/ttyUSB0: [Errno 2] No such file or directory: '/dev/ttyUSB0'). Retrying in 2 seconds
2020-05-22 12:42:30 WARNING (MainThread) [upb_lib.upb] Could not connect to UPB PIM ([Errno 2] could not open port /dev/ttyUSB0: [Errno 2] No such file or directory: '/dev/ttyUSB0'). Retrying in 4 seconds
2020-05-22 12:42:34 WARNING (MainThread) [upb_lib.upb] Could not connect to UPB PIM ([Errno 2] could not open port /dev/ttyUSB0: [Errno 2] No such file or directory: '/dev/ttyUSB0'). Retrying in 8 seconds
2020-05-22 12:42:42 ERROR (MainThread) [homeassistant.components.upb.config_flow] Timed out after 15 seconds trying to connect with UPB PIM at serial:///dev/ttyUSB0

I repeated the test using the serial port’s alternate name:
/dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_D-if00-port0
but the result was the same.

2020-05-22 13:08:40 WARNING (MainThread) [upb_lib.upb] Could not connect to UPB PIM ([Errno 2] could not open port /dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_D-if00-port0: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_D-if00-port0'). Retrying in 1 seconds
2020-05-22 13:08:41 WARNING (MainThread) [upb_lib.upb] Could not connect to UPB PIM ([Errno 2] could not open port /dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_D-if00-port0: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_D-if00-port0'). Retrying in 2 seconds
2020-05-22 13:08:43 WARNING (MainThread) [upb_lib.upb] Could not connect to UPB PIM ([Errno 2] could not open port /dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_D-if00-port0: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_D-if00-port0'). Retrying in 4 seconds
2020-05-22 13:08:47 WARNING (MainThread) [upb_lib.upb] Could not connect to UPB PIM ([Errno 2] could not open port /dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_D-if00-port0: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_D-if00-port0'). Retrying in 8 seconds
2020-05-22 13:08:55 ERROR (MainThread) [homeassistant.components.upb.config_flow] Timed out after 15 seconds trying to connect with UPB PIM at serial:///dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_D-if00-port0

I disconnected the cable and PIM from the RPI3, connected it to the laptop I had used (0.108.5, docker) with the beta UPB integration (configured in YAML) and it found the PIM and worked correctly. This re-confirms the cable and PIM are functional.

[jameshilliard]

From my understanding by default the RPI images for home assistant run home assistant in a container which doesn’t have access to USB devices unless configured a certain way.

[123 (Taras)]

I’m unfamiliar with that constraint. It’s my understanding that whatever is reported in the Supervisor > System > Hardware list is readily accessible.

Here’s a post that says the same:

Hass.io generic Linux install - how to map the USB port for the ZWave stick?

In the hassio installer, every USB is passed through. You don’t do anything to pass it through.

Otherwise this alleged USB-device constraint would affect all RPI3 users who plug in coordinators for Zigbee and zwave.

[gwww (Glenn Waters)]

Hmmm. I don’t know how to fix this. I’ve never used a USB connected device in hassio. When I did ElkM1 it was network connected. The UPB is USB connected to a windows box that runs a serial to TCP proxy, so UPB is network connected.

What I have tested and what I know works is a USB connected UPB PIM using a serial to USB adapter. That was tested on a MacBook. Similar to your hardware. The difference is Docker.

I’ll do some searches to see what I can uncover. I’d recommend you do the same. Let’s see what we can come up with.

What is clear from the error message is that the device name is not understood. Perhaps poke on the hassio forums or Discord chat.

[123 (Taras)]

A few more details regarding the system used for the beta version (i.e. the one that works). It’s running Home Assistant Core 0.108.5 in Docker. I’m using docker-compose and here’s the relevant portion that maps the serial-to-USB cable from its “real-world” device name to the port name visible within the Docker container.

    devices:
      - "/dev/serial/by-id/usb-Prolific_Technology_Inc._USB-Serial_Controller_D-if00-port0:/dev/ttyUSB0"

In configuration.yaml, I specify the port name visible within the Docker container, namely /dev/ttyUSB0.

upb:
  url: serial:///dev/ttyUSB0:4800
  file_path: ./upb.upe

All of this plumbing works correctly on this system running the beta version.

Home Assistant 0.110.0 on an RPI3 (a.k.a. the old “hassio”) includes Home Assistant Core as a docker container. However, there’s no user intervention required to map “real world” ports to how they are seen within the container. Theoretically, this is done for the user and revealed in the Hardware list.