My winter project: UPB to MQTT

[123 (Taras)]

I’m using this thread like a diary to document my project. I encourage anyone with an interest, or expertise, in UPB to jump in and help out.

This winter’s project is to create a UPB to MQTT converter (upb2mqtt). Any home automation software than supports MQTT will be able to take advantage of it. For example, Home Assistant’s MQTT Light integration will be able to use it.

To accelerate development, I’ve chosen to create it as a Node-Red flow. After about a week’s worth of UPB (and Javascript) research and experimentation, I’ve created a flow that receives, validates, and interprets UPB messages then converts them to appropriate MQTT topics and payloads. Before moving on to the task of transmitting UPB messages, I need to test and ensure I’m properly handling received UPB messages.

UPB’s computer interface device is called a PIM (Powerline Interface module). I have one connected via a USB-RS232 converter to a Raspberry Pi 3B hosting Node-Red and an MQTT Broker. A PIM has two modes of operation:

1. Pulse Mode
   In this mode, it reports all UPB activity. However, the information, although very comprehensive, is reported as tiny packets. The burden of assembling these packets into meaningful UPB messages is left to the software developer.

2. Message Mode
   In this mode, it reports a subset of all UPB activity. Only complete UPB messages are reported. This mode makes the software developer’s life easier. Most UPB integrations I’ve found use Message Mode (DaAwesomeP’s UPB for node-js, cvanorman’s UPB for openHAB, and Jason Sharpee’s UPB for Pytomation).

I currently use Premise which has an excellent UPB integration that uses Pulse Mode. It serves as a reference for comparing the differences between what can be accomplished using Pulse versus Message modes (a little foreshadowing here).

The first drawback of Message Mode that I’ve encountered is one involving scenes (a.k.a. links):

Scene members do not report status upon deactivation.

• Upon activating a scene (link), ReportState commands are sent to all scene members.
• They reply with their current state via a DeviceStateReport.
• Upon deactivating a scene (link), no ReportState commands are sent so none of the scene members report their current state.

For example, assume a scene simply turns on/off three devices.

• When the scene is activated, each scene member will report its current state (on).
• When the scene is deactivated, each scene member will NOT report its current state (off).

This is an annoying deficiency (for obvious reasons) and DaAwesomeP mentioned it in this post:

whenever a scene/link change is sent out by a switch, the switches affected by the scene don’t broadcast their states; this is a limitation of UPB

However, I believe that statement requires qualification; it is a limitation of UPB’s Message Mode. The limitation doesn’t appear in Premise’s UPB integration (which uses Pulse Mode). When I deactivate a scene, the new status of all scene members, in Premise, is updated. So at first glance, it appears there’s some kind of enhanced scene reporting mechanism available in Pulse Mode that doesn’t exist in Message Mode. However, the programming challenge of using Pulse Mode is a bit daunting to me.

Here’s another behavioral quirk that, as far as my tests show, occurs in both Message and Pulse modes.

Double Reporting
When manually dimming/brightening a switch (that is already on and configured to report its state), it sends two DeviceStateReports:

1. First Report: False values
   • If you are increasing its brightness, it reports its level is 100 (i.e. turned on).
   • If you are decreasing its brightness, it reports its level is 0 (i.e. turned off).
2. Second Report: True values
   • It reports its actual, true level.

The purpose of the first report is unclear to me. It looks exactly like the report generated when a switch is turned on or off but that’s not what’s actually happening (it’s level is simply changing to a value between 0 and 100). The second report is correct.

• The device does not issue an initial false report if it is initially off and you manually turn it on or if it is initially on and you turn it off.
• The false initial report only occurs when the light is already on and then you manually dim/brighten it.

This quirk produces an undesirable cosmetic defect. For example, if you manually dim a switch from 75 to 50, in Home Assistant the switch’s slider will initially, and very briefly, move from the three-quarter position to zero before moving to the halfway position. That’s because it is responding to the first report (level is 0) before it responds to the second report (level is 50). This can escalate from being a cosmetic defect to a functional one if you have automations triggered by the switch’s state changing to on or, more significantly, to off.

I’ve been thinking of a mechanism (involving the use of a queue) to discard the first report if it is immediately followed by a second one. However, the darn thing will have to be foolproof otherwise it may discard legitimate reports of a switch turning on or off.

tl;dr
Scenes are stunted in Message Mode and switches will briefly report invalid states when operated manually. Using Pulse Mode should fix scenes but at the cost of development complexity.

---

NOTE:
My first attempt to transmit a UPB message was a failure. It not only failed to control the switch, it locked up the serial port node and prevented the flow from receiving anything from the PIM. Deleting and re-adding the serial port node cleared it up (but that’s no way to run a railroad). The UPB message was valid because it was generated by the message-building utility (supplied by the manufacturer). Clearly, there’s something important I’ve missed and will take more time than anticipated. Hopefully, not longer than one winter season …

[david1 (David)]

What is UPB?

[123 (Taras)]

UPB = Universal Powerline Bus

It’s a communications protocol designed for lighting. Whereas ZigBee and Z-Wave are protocols that propagate via RF (radio frequency), UPB is designed to propagate via the building’s existing electrical wiring (i.e. powerline). There’s no hub needed for UPB devices to communicate with one another.

UPB has been around for a long time but is more likely to be used by professional systems integrators than home automation hobbyists. When properly installed and configured, it is reliable and maintenance-free. The software used to configure a UPB system is called UPStart. It lets you configure new and existing devices, build scenes, diagnosis problems, store/restore configurations, etc.

Sample view of UPStart software taken from pcslighting’s UPStart User Guide (pdf):

A UPB dimmer switch can be configured to behave as either a dimmer or a switch (i.e. the same physical device can serve either purpose). The faceplates are removable and can be replaced with another color (without having to uninstall the entire device). There’s one model that supports faceplates ranging from a single rocker button all the way up to four rockers or combinations of rockers and pushbuttons. It means you can reconfigure the device to serve a new purpose by simply changing its faceplate, and its programming, without having to physically uninstall it (or replace it). This is important to professional integrators (time = money).

Unlike ZigBee and Z-Wave’s pairing procedure, to join a new device to a UPB network you tap on its rocker 5 times to put it in setup mode. The new device will be recognized in UPStart and then you can configure it. Each device stores its name, room, scene memberships, etc.

[123 (Taras)]

I sorted out the problem I had with transmitting a UPB message. The combination of this Cocoontech post and PCSlighting’s Application Note 126 (pdf file) paved the way to a solution.

The following UPB message was produced by the UPB Command Wizard. In a nutshell, it turns on a light at a specific rate.

09049609FF224003F0

The Command Wizard led me to believe that I only had to append appropriate delimiters (Ctrl-t = hex 14 and a carriage return = hex 0D) and it was good to go. Like this:

1409049609FF224003F00D

Wrong. It’s a bit more complicated than that.

The UPB message produced by the Command Wizard is comprised of stringified hex codes. You first have to convert each character in the string to its ASCII hex equivalent. For example, 0 is 30 and F is 46.

So this:

09049609FF224003F0

becomes this:

303930343936303946463232343030334630

Then you append the delimiters 14 and 0D:

143039303439363039464632323430303346300D

Finally, (yes, there’s one more step) you convert each hex code into a proper hex value before transmitting it to the PIM. That means 14 gets sent as hex value 14, etc.

Here’s the whole process in Javascript (part of a Function node in Node-RED):

var str = '09049609FF224003F0';
// ASCII to hex codes
var arr = [];
for (var i = 0, l = str.length; i < l; i ++) {
    var hex = Number(str.charCodeAt(i)).toString(16);
    arr.push(hex);
    }
// Append delimiters
var cmd = '14' + arr.join('') + '0D';
// Convert to hex values
msg.payload = Buffer.from(cmd, 'hex');
return msg;

Now that it’s resolved I still have the problem of serial-port stability. After transmitting a command via the serial-port node it ceases to receive commands. The serial-port node is the latest version (10), node.js is also very recent (12.13.1) but my version of Node-red is ancient (0.18). I don’t know if that has any bearing but, before I raise this stability issue on the Node-RED forum, I’ll update Node-RED to the latest version and repeat my testing.

[123 (Taras)]

Progress! A showstopper issue has been eliminated. I reached out to the Node-RED community and it’s all documented here:

Serial-port nodes failing to receive and transmit within the same flow

The actual reason why the serialport nodes, serial-in and serial-out, were misbehaving remains unclear but at least now I have a way of making them function as advertised. As a result, my flow can now both receive and send UPB messages without locking up.

Through experimentation, I continue to learn more about the behavior of various UPB options. For example, for each transmitted message you can optionally include:

• ACK Pulse
• ID Pulse
• Acknowledgement Message
• and two more concerning if and how the message should be handled by a repeater

All of the acknowledgement requests don’t do what I expected which is to respond with the device’s current state. What they do is simply acknowledge receipt of the command.

What this implies is that, in practice, it may be advisable to follow up any command, that alters the device’s state (Goto, StartFade, etc), with another command that requests the device’s new state (ReportState). So it’ll be like ‘change state to on’ and then ‘tell me your state’. To implement this, I’m thinking of using a message queue to buffer the transmitted commands. Perhaps node-red-contrib-simple-message-queue or node-red-contrib-queue-gate. The fun continues.