Zwave - cannot read values from Command Class MANUFACTURER_PROPRIETARY (145dec, 91hex)

[ChristianKuehnel]

Home Assistant release with the issue:
dev branch

Last working Home Assistant release (if known):
I already had a working implementation about 2 years ago, but did not merge it. Back then I could just read the values from HA (with a modified open-zwave library).

Operating environment:

• Linux, Python 3.7
• open-zwave, python-ozw: hass branch (from the HA repository)

Component/platform:
zwave.cover

Description of problem:
I want to extend the zwave.cover component to support the proprietary zwave command class “MANUFACTURER_PROPRIETARY” (145 dec, 0x91 hex). This enables an extended support for venetian blinds, where you can set the tilt angle of the blinds.

My steps:

1. I implemented the changes on open-zwave https://github.com/home-assistant/open-zwave/pull/6,
2. I reconfigured the dependencies to use a local python-ozw version with my modified open-zwave as per instructions.
3. ozw_shell and MinOZW show the data from this command class. So the open-zwave changes seem to be working.
4. The new libopenzwave.cpython-37m-x86_64-linux-gnu.so also contains the symbols that I added. HA is using this library.
5. When I try to read the new values in HA, they do not exist. Also the zwave.nodes.ZWaveNode.command_classes() does not list the command class I added.

It looks like this command class is somehow filtered when passing the data from open-zwave to HA. I looked through the code, but I did not find it.

The modified zwave.cover component is here:

ChristianKuehnel/home-assistant/blob/fibaro2019/homeassistant/components/zwave/cover.py

Can anyone tell my why I cannot access the added command_class?
Does anyone know if HA is filtering the list of command_classes for a node?

Best,
Christian

[ChristianKuehnel]

As the setup is non-trivial, here are the instruction to replicate/test this:

configure the fibaro devices

First you need to enable the proprietary venentian blind mode in the Z-Wave device:

• Set 3: Reports type to Blind position reports sent to the main controller using Fibar Command Class
• Set 10: Roller Shutter operating modes to 2 - Venetian Blind Mode, with positioning

Only then will the device output (and accept) the Command_Class_Manufacturer_Proprietary.

open-zwave

To get and build the modified version of open-zwave:

git clone [email protected]:ChristianKuehnel/open-zwave.git
cd open-zwave
git checkout ha_fibaro
make

# required for open-zwave (see below)
export LOCAL_OPENZWAVE=`pwd`

Now you can run the MinOZW binary. If you now toggle the venetian blinds with a normal switch, you should see tilt angle and slat position in the log output.

python-openzwave

Now we need to configure python-openzwave to use the modified open-zwave library. We do not need to change the python library, we just need to use a different C++ library.

git clone [email protected]:home-assistant/python-openzwave.git
cd python-openzwave
git checkout hass

# Load your virtual environment for HomeAssistant then install the modified version there.
# This depends on the LOCAL_OPENZWAVE environment variable
pip install --upgrade --install-option="--flavor=dev" .

You can read and write the slats and tilt positions using the ozwsh tool included in python-openzwave. Here the changes to open-zwave are working.

Links:

• Instruction for the “dev” version of pyhton-openwave are here: https://github.com/home-assistant/python-openzwave
• Instruction for modifying the dependencies of HA: https://developers.home-assistant.io/docs/en/creating_integration_manifest.html

HomeAssistant

Now that we have the ZWave libraries set up, we can get the changes in HomeAssistant:

git clone [email protected]:ChristianKuehnel/home-assistant.git
cd home-assistant
git checkout fibaro2019

# the argument --skip-pip is important to prevent HA from overriding the modified library
hass --skip-pip

[andre-richter (Andre)]

I was able to reproduce the scenario you described. Using MinOZW, I can observe the proprietary message:

2019-06-03 22:29:06.690 Detail, Node028,   Received: 0x01, 0x0e, 0x00, 0x04, 0x00, 0x1c, 0x08, 0x91, 0x01, 0x0f, 0x26, 0x03, 0x03, 0x00, 0x00, 0x58
2019-06-03 22:29:06.690 Info, Node028, Received Fibaro proprietary blind/slat position for node 28: Blinds: 0 Slats: 0
2019-06-03 22:29:06.690 Detail, Node028, Refreshed Value: old value=0, new value=0, type=byte
2019-06-03 22:29:06.691 Detail, Node028, Changes to this value are not verified
2019-06-03 22:29:06.691 Detail, Node028, Refreshed Value: old value=24, new value=0, type=byte
2019-06-03 22:29:06.691 Detail, Node028, Changes to this value are not verified
2019-06-03 22:29:06.691 Detail, Node028, Notification: ValueChanged
2019-06-03 22:29:06.691 Detail, Node028, Notification: ValueChanged

Yet, HA is not showing this command class. Here is a snippet of the output of PRINT NODE from the web interface (beautified):

Summary

{  
   'basic':4,
   'capabilities':{  
      'listening',
      'routing',
      'beaming'
   },
   'command_classes':{  
      0,
      32,
      37,
      38,
      133,
      134,
      43,
      142,
      112,
      49,
      50,
      114,
      115,
      117
   },
   'command_classes_as_string':{  
      'COMMAND_CLASS_SCENE_ACTIVATION',
      'COMMAND_CLASS_METER',
      'COMMAND_CLASS_MANUFACTURER_SPECIFIC',
      'COMMAND_CLASS_POWERLEVEL',
      'COMMAND_CLASS_NO_OPERATION',
      'COMMAND_CLASS_ASSOCIATION',
      'COMMAND_CLASS_BASIC',
      'COMMAND_CLASS_VERSION',
      'COMMAND_CLASS_MULTI_CHANNEL_ASSOCIATION',
      'COMMAND_CLASS_CONFIGURATION',
      'COMMAND_CLASS_SENSOR_MULTILEVEL',
      'COMMAND_CLASS_SWITCH_MULTILEVEL',
      'COMMAND_CLASS_PROTECTION',
      'COMMAND_CLASS_SWITCH_BINARY'
   },

Then again, looking at OZW_Log.txt in the HA folder shows command class 0x91 being issued:

2019-06-03 23:55:58.779 Detail, Node028,   Received: 0x01, 0x0e, 0x00, 0x04, 0x00, 0x1c, 0x08, 0x91, 0x01, 0x0f, 0x26, 0x03, 0x03, 0x00, 0x1f, 0x47
2019-06-03 23:55:58.779 Info, Node028, ApplicationCommandHandler - Unhandled Command Class 0x91
2019-06-03 23:55:58.794 Detail, Node028,   Received: 0x01, 0x0c, 0x00, 0x04, 0x00, 0x1c, 0x06, 0x31, 0x05, 0x04, 0x22, 0x00, 0x00, 0xff
2019-06-03 23:55:58.794 Info, Node028, Received SensorMultiLevel report from node 28, instance 1, Power: value=0.0W
2019-06-03 23:55:58.794 Detail, Node028, Refreshed Value: old value=0.0, new value=0.0, type=decimal
2019-06-03 23:55:58.794 Detail, Node028, Changes to this value are not verified
2019-06-03 23:55:58.794 Detail, Node028, Notification: ValueChanged

Why would the OZW_Log created by MinOZW correctly print the Fibaro tilt/slat message, while the OZW_Log created by HA wouldn’t?

Could there be something afoul with the state of zwcfg_*.xml? The original one did not have entries for the MANUFACTURER_PROPRIETARY command class, so I started MinOZW with a blank config and it created a new config file having the propietary entrries. Feeding the newly created one into HA seems to not work by default.
For example, I am able to control my devices, by querying node config options yields wrong results.

BR,
Andre

[andre-richter (Andre)]

Update:

Yep, it seems to be the zwcfg_*.xml.

Running MinOZW with my old HA zwcfg (which doesn’t have entries for MANUFACTURER_PROPRIETARY), I get the

ApplicationCommandHandler - Unhandled Command Class 0x91

message.

Running MinOZW with the zwcfg it created itself (which has entries for MANUFACTURER_PROPRIETARY), I can see the Fibaro blind/slat message again, and not the unhandled command class error.

[ChristianKuehnel]

Hmm, that’s unexpected. I’ll have a look on the weekend. Thanks for investigating! I’ll keep you posted.

[andre-richter (Andre)]

Hi Christian,
I finally figured this out. Sort of…

First of all, I found various temp files and cache files and pyc files to interfere with the whole endeavor. Hence, the following guide assumes a pristine Ubuntu installation. A Virtualbox VM worked fine for me.

Guide

1. Install dependencies.

# From https://developers.home-assistant.io/docs/en/development_environment.html
sudo apt-get install python3-pip python3-dev python3-venv
sudo apt-get install autoconf libssl-dev libxml2-dev libxslt1-dev libjpeg-dev libffi-dev libudev-dev zlib1g-dev
sudo apt-get install -y libavformat-dev libavcodec-dev libavdevice-dev libavutil-dev libswscale-dev libavresample-dev libavfilter-dev

2. Make a venv for HA.

python3 -m venv hass_venv

3. Install open-zwave with MANUFACTURER_PROPRIETARY Fibaro changes.

git clone -b ha_fibaro https://github.com/ChristianKuehnel/open-zwave.git --depth=1
cd open-zwave
export LOCAL_OPENZWAVE=`pwd`
cd ..

4. Install homeassistant-pyozw using the local open-zwave.

source hass_venv/bin/activate
pip install wheel
pip install PyDispatcher
pip install homeassistant-pyozw --no-cache-dir --no-deps --install-option="--flavor=dev"

5. Install Home Assistant with Christian’s code for Fibaro tilt control.

git clone -b fibaro2019 https://github.com/ChristianKuehnel/home-assistant.git --depth=1
cd home-assistant
script/setup

6. Now, copy your .homeassistant folder to the VM’s ~ if it wasn’t already there.
7. The next step is odd, but needed. We need to let openzwave build a new zwcfg.xml that has entries for the MANUFACTURER_PROPRIETARY command class. Therefore, delete your old config now (absolutely make a backup before deleting, just in case!).

rm ~/.homeassistant/zwcfg*
rm ~/.homeassistant/pyozw.sqlite
rm ~/.homeassistant/options.xml

8. Start Home Assistant.

hass

9. Give HA some time to rediscover all the devices in the network. You can follow the activity live in another terminal:

cd ~/.homeassistant
tail -f OZW_Log.txt

10. After some 10 minutes or so, the home assistant console will show that the Z-Wave network is now ready by printing this message:

INFO (Dummy-3) [homeassistant.components.zwave] Z-Wave network is ready for use. All awake nodes have been queried. Sleeping nodes will be queried when they awake.

11. Close homeassistant once now by pressing CTRL-C. This will trigger the new zwcfg.xml to be written.
12. You can do a first sanity check now by grepping for COMMAND_CLASS_MANUFACTURER_PROPRIETARY in this newly written config.
13. Now, start HA again. If all went good, you can see and control your devices after some time. If you change the slat position now via physical wall switch, the OZW_Log.txt console should show the following message:

Info, Node028, Received Fibaro proprietary blind/slat position for node 28: Blinds: 0 Slats: 18
Detail, Node028, Refreshed Value: old value=0, new value=0, type=byte
Detail, Node028, Changes to this value are not verified
Detail, Node028, Refreshed Value: old value=0, new value=18, type=byte
Detail, Node028, Changes to this value are not verified
Detail, Node028, Notification: ValueChanged
Detail, Node028, Notification: ValueChanged

14. Even the web interface is showing the tilt dialogue now :
    

Observations

1. Slat control via web interface is only working partially. I can open about 50% and back again, but not more. Might be because I don’t have fine-tuned the respective config parameters in the zwave device yet.
2. The newly generated zwcfg.xml shows all my zwave devices in the web interface correctly. However, control is working only partially.
   2.1 For example, I can open a cover, but the device would not receive updates on the position so I can never close it again because the close button is greyed out.
   2.2 Maybe it is needed to create a manual hybrid of the original configuration and the newly created one. E.g. copying over the newly added MANUFACTURER_PROPRIETARY parts (which are definitely needed according to my experiments), and leaving other parts untouched.

BR,
Andre

[ChristianKuehnel]

Andre, you’re right! When I delete the zwcfg.xml and have it added a new entry to the Fibaro devices. And that’s my missing command class! For whatever reason the library is storing a copy of the device configuration and never updates it.

			<CommandClass id="145" name="COMMAND_CLASS_MANUFACTURER_PROPRIETARY" version="1" request_flags="4" innif="true">
				<Instance index="1" />
				<Value type="byte" genre="user" instance="1" index="0" label="Venetian Blind slat position" units="%" read_only="false" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="100" value="0" />
				<Value type="byte" genre="user" instance="1" index="1" label="Venetian blind tilt position" units="%" read_only="false" write_only="false" verify_changes="false" poll_intensity="0" min="0" max="100" value="0" />
			</CommandClass>

Thank you so much! I really missed that point.

On my setup it’s working right away, but there is sometimes a significant delay (~7 sec) between changing something in the UI and the device actually moving. I did some more improvements to the code. What’s still missing is a proper detection of the configuration parameters so that I know when to turn on the Venetian Blind mode: https://github.com/home-assistant/home-assistant/issues/24404

I updated my branch to the latest dev branch. To get things to work again I had to wipe my .homeasssistant folder.

I have my changes now up in a PR for public review and comments:
https://github.com/home-assistant/home-assistant/pull/24405

[andre-richter (Andre)]

@ChristianKuehnel

On my setup it’s working right away, but there is sometimes a significant delay (~7 sec) between changing something in the UI and the device actually moving.

I am observing this too. Following a tilt command, aka MANUFACTURER_PROPRIETARY, any next command is only happening after a few seconds of delay. Be it another tilt command again, or just an open/close blinds command.

Issuing non-tilt commands back-to-back, e.g. close - stop - open blinds, this delay is not there. They work within sub-second intervals.