RPi as Z-Wave/ZigBee-over-IP server for Hass

Project goal

Setup an RPi as a USB-over-IP server. Run Hass on a VM (or wherever) and connect to a Z-Wave/ZigBee/etc dongle physically plugged into an RPi somewhere on your network.

Background

I started working with Hass using an RPi3 and an HUSBZ-1 combo Z-Wave/ZigBee stick which worked well but I have concerns about long-term reliability of SD cards on the Pi and while performance was OK, I have a virtualization farm here that provides me much better performance and availability than an RPi could. The trouble there is that I still needed access to the USB stick, and physically plugging it into one of my virtualization hosts would prevent me from moving the Hass VM around in response to performance or availability situations in my farm.

The goal here is to create a USB over IP service on a Raspberry Pi, plug the USB radio(s) into the Pi, then place that device on your network somewhere close to the controlled devices. Then Hass can run wherever you like while controlling your devices over an IP link to the radio(s).

Setting up the USB/IP server

Requirements

  • Raspberry Pi running Rasbian (or something like it)
  • USB dongle to share

Process

SSH to raspbian and execute the following commands:

sudo -s
lsusb

lsusb should show a list of attached USB devices, hereā€™s what mine looks like:

Bus 001 Device 006: ID 10c4:8a2a Cygnal Integrated Products, Inc.
Bus 001 Device 005: ID 0557:2306 ATEN International Co., Ltd
Bus 001 Device 004: ID 0781:5583 SanDisk Corp.
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Weā€™re looking for the device identifier for the USB radio, which in my case is that Cygnal Integrated Products, Inc. device with an ID of 10c4:8a2a. Weā€™ll then setup a systemd service definition that is going to search for that device string and attach it to the USBIPd service. If youā€™re using a different USB device, change the device ID in the lines below for ExecStartPost and ExecStop

# Install usbip and setup the kernel module to load at startup
apt-get install usbip
modprobe usbip_host
echo 'usbip_host' >> /etc/modules

# Create a systemd service
vi /lib/systemd/system/usbipd.service

Copy and paste the following service definition:

[Unit]
Description=usbip host daemon
After=network.target

[Service]
Type=forking
ExecStart=/usr/sbin/usbipd -D
ExecStartPost=/bin/sh -c "/usr/sbin/usbip bind --$(/usr/sbin/usbip list -p -l | grep '#usbid=10c4:8a2a#' | cut '-d#' -f1)"
ExecStop=/bin/sh -c "/usr/sbin/usbip unbind --$(/usr/sbin/usbip list -p -l | grep '#usbid=10c4:8a2a#' | cut '-d#' -f1); killall usbipd"

[Install]
WantedBy=multi-user.target
# reload systemd, enable, then start the service
sudo systemctl --system daemon-reload
sudo systemctl enable usbipd.service
sudo systemctl start usbipd.service

Setting up the USB/IP client

Requirements

  • Linux server/desktop (Tested on Ubuntu server 17.04. There are some Windows builds but I couldnā€™t get any of them to work reliably under Win10/Server 2016)
  • IP address of your RPi running as a server. Here Iā€™m using 192.168.0.10.

Process

SSH to the Linux server and execute the following commands:

sudo -s
apt-get install linux-tools-generic -y
modprobe vhci-hcd
echo 'vhci-hcd' >> /etc/modules

Much like we did on the server, weā€™re going to need to modify the ExecStart and ExecStop lines below to search for the correct USB device ID thatā€™s being presented by your USB/IP server. Likewise, change the IP 192.168.0.10 to match your RPi USB server.

vi /lib/systemd/system/usbip.service

Copy and paste the following service definition:

[Unit]
Description=usbip client
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c "/usr/lib/linux-tools/$(uname -r)/usbip attach -r 192.168.0.10 -b $(/usr/lib/linux-tools/$(uname -r)/usbip list -r 192.168.0.10 | grep '10c4:8a2a' | cut -d: -f1)"
ExecStop=/bin/sh -c "/usr/lib/linux-tools/$(uname -r)/usbip detach --port=$(/usr/lib/linux-tools/$(uname -r)/usbip port | grep '<Port in Use>' | sed -E 's/^Port ([0-9][0-9]).*/\\1/')"

[Install]
WantedBy=multi-user.target
# reload systemd, enable, then start the service
sudo systemctl --system daemon-reload
sudo systemctl enable usbip.service
sudo systemctl start usbip.service

You should now be able to access the USB device over the network as if the device was plugged in locally, and you have an auto-starting systemd service to control things.

50 Likes

This is really cool!

3 Likes

Nice!!
Is it possible to share more than one device ?

Yup! Just make multiple service definitions (one for each device) on both the client and the server side, and setup the USB device descriptors in each.

2 Likes

Awesome job! MARK & TO DO:rofl:

Great workā€¦how is the stability? Iā€™m in the same boat - hate to rely on a Pi for everything, and Iā€™d like to move everything that I can virtualize onto a proper serverā€¦

I made the pi a zwave-mqtt/http bridge. Trying now to make the image read-only / writable with two scripts. Seemed more reliable than USB over IP.
Anyway, great work and thank you for the tutorial!

1 Like

Iā€™ve been running for about a month with this configuration with no problems thus far. The only complication Iā€™ve run into is that if for whatever reason I need to restart the Pi, on the Hass VM Iā€™ll then need to manually shutdown hass, then usbip, then turn them back on as neither USBIP nor hass will attempt to auto re-connect.

Nice work! Thanks!

Will it work to do the same with RPis built in Bluetooth? Or with an USB BT?

Can you share your implementation, please? Iā€™m looking for a good and reliable bridge between zwave and mqtt. tnx

This script is a work in progress: https://github.com/yahat/home_assistant_mqtt_translation

It takes all the states and attributes from home assistant and publishes them to individual topics.

Itā€™s a work in progress, I need one or two more weeks to finish it (add more features) and add some documentation.

I probably expressed myself wrong earlier, the pi is a zwave->homeassistant ā€œbridgeā€ and this script makes it homeassistant -> mqtt bridge.

This way i get all the web interface / yaml config files from home assistant, and the mqtt reliability.

The script useses http api get state and the mqtt eventstream on the pi to get live data.

You can install pm2 https://github.com/Unitech/pm2 to run nodejs scripts as services easily.

Oh wow - thatā€™s exactly how mqtt_eventstream doesnā€™t work but SHOULD! By putting every device on itā€™s own topic the broker can retain state and itā€™s so much easier to work with then dumping giant JSON objects on a single topic. Great work!

Commands are not supported yet but they can be called directly from home assistant

Nice! I actually just submitted a pull request for a new ā€˜mqtt_statestreamā€™ component (https://github.com/home-assistant/home-assistant/pull/9286). I think itā€™ll be in the 0.54 release. It currently only publishes states, not attributes, though.

I wanted something where I could subscribe some external systems to only the entities that I was interested in, not the whole flood of eventstream.

1 Like

Oh lord I could kiss you right now. I really donā€™t like mqtt_eventstream for all the reasons I just posted in response to your PR. It serves the one purpose of linking together hass instances ok but it really is WAY out of alignment with proper MQTT use.

Using separate topics for every entity_id is The Way To Go and Iā€™m thrilled youā€™ve made it happen! Further, youā€™ve designed the namespace such that it can align with the MQTT Discovery standards which is also really nice.

If I might make some suggestions / feature requests:

  • As youā€™ve noted there are no attributes supported (yet). Maybe add them as subtopics? For example homeassistant/light/master_bedroom_dimmer could have a subtopic homeassistant/light/master_bedroom_dimmer/brightness
  • Could you also publish an autodiscovery config message for each device (maybe on component initialization with retain on)? See examples here. If you published this one message for each device, one could set up another Hass instance with mqtt_discovery enabled and then all of the (supported) objects would automatically appear in the other instance without any further configuration.

Glad this seems useful to you!

I like the idea of adding attributes - Iā€™ll see if I can get a PR ready in time for the release. I think it might be a good idea to publish the state to a ā€˜stateā€™ subtopic rather than using the root (ie. homeassistant/light/master_bedroom_dimmer/state) if weā€™re going to add in all the attributes. It would be good to do that before it gets released rather than needing a breaking change right away.

As for the autodiscovery - it should be doable, but right now it would really only make sense for sensors and binary_sensors. Itā€™s only a one-way component (right now, at least) - it doesnā€™t listen for update being made to push back into home assistant. That may be doable, but itā€™s a lot more involved and will take some timeā€¦

Even as a 1-way component having auto-discovery enabled still would allow the target Hass instance to reflect the state of the object (light/switch/etc). That may be confusing to the user however. Anyway, great work and Iā€™m looking forward to seeing it in the official distribution soon.

Very nice work! Thank you so much! If it would be possible to also add attributes it would be perfect.

FYI - the basic statestream component just went live in 0.54. Iā€™m working on publishing attributes and should be able to get a PR in soon. After that, Iā€™ll see what I can do with autoconfigā€¦

1 Like