[Guide] Separating ZWave to a Separate HASS Instance via MQTT Statestream

Background

I recently completed a major overhaul of my HASS setup. I took advantage of some free time to move off of hassbian to a Docker install on a standalone (pre-existing) server. That process was somewhat challenging but presented a number of opportunities. Most notably the ability to create a number of test instances to play around with without fear of harming my “main” instance.

I had seen several people discuss separating their ZWave network onto two separate instances, but I always pushed it to the side because I had everything running in hassbian and didn’t see a need. Now that I was running in Docker I decided I could test things out. Also, I’d seen some posts about zwave2mqtt so I figured I could experiment and see how things went.

Well, long story short, this was a difficult project for me. I’m not a coding expert and I found a lot of posts talking about the basics of how to separate/control two instances of HASS but not a lot of clear guidance or documentation. So I wanted to share exactly how I got it all working in case somebody else wants to try this.

The major advantage from separating my Zwave instance from the main instance is that I can now restart my main instance extremely quickly and I don’t have to wait for the Zwave network to come back online. I found this most frustrating when I was adjusting things (new scenes, sensors, etc) and would have to wait ~5 minutes between restarts. Now it’s <2 minutes and I can immediately test things.

Setup

So to start, you’ll need MQTT. I’m not going to cover setting up MQTT, there’s plenty of guides out there. To accomplish the syncing, I used MQTT Statestream. I think this part is fairly self-explanatory. You set the base_topic you want (i used homeassistant for my Zwave instance to publish to the main and homeassistant_zwave to publish from the main instance to Zwave). I excluded a bunch of entities/only included what I needed. Up to you if you want all your data streamed, I only needed certain things so that’s what I did.

You’ll also need to enable MQTT Discovery. Again, this part is fairly self explanatory, just set your discovery_prefix to whatever you want to discover.

Now comes the complicated part. To get entities set up in the main HASS instance, I was inspired by RobDYI’s post on Automating the sharing of sensors and switches between HA’s using MQTT Discovery and Statestream. Here are the automations that worked for me:

On the main instance:

- alias: "MQTT Discovery Switch Creator"
  trigger:
    - platform: mqtt
      topic: 'homeassistant/switch/#'
  condition:
    condition: template
    value_template:  "{{ trigger.topic.split('/')[3] == 'state' }}"   
  action:
    - service: mqtt.publish
      data_template:
        topic: "homeassistant/{{ trigger.topic.split('/')[1] }}/{{ trigger.topic.split('/')[2] }}/config"
        payload: "{\"name\": \"{{ trigger.topic.split('/')[2]| replace('_', ' ') | title }}\",  \"pl_off\":\"off\", \"pl_on\":\"on\",  \"stat_t\": \"homeassistant/{{ trigger.topic.split('/')[1] }}/{{ trigger.topic.split('/')[2] }}/state\",  \"cmd_t\": \"homeassistant/{{ trigger.topic.split('/')[1] }}/{{ trigger.topic.split('/')[2] }}/set\" }"
        retain: true

- alias: "MQTT Discovery Light Creator"
  trigger:
    - platform: mqtt
      topic: 'homeassistant/light/#'
  condition:
    condition: template
    value_template:  "{{ trigger.topic.split('/')[3] == 'state' }}"   
  action:
    - service: mqtt.publish
      data_template:
        topic: "homeassistant/{{ trigger.topic.split('/')[1] }}/{{ trigger.topic.split('/')[2] }}/config"
        payload: "{\"name\": \"{{ trigger.topic.split('/')[2]| replace('_', ' ') | title }}\", \"pl_off\":\"off\", \"pl_on\":\"on\",  \"cmd_t\": \"homeassistant/{{ trigger.topic.split('/')[1] }}/{{ trigger.topic.split('/')[2] }}/set\",  \"stat_t\": \"homeassistant/{{ trigger.topic.split('/')[1] }}/{{ trigger.topic.split('/')[2] }}/state\",  \"bri_stat_t\": \"homeassistant/{{ trigger.topic.split('/')[1] }}/{{ trigger.topic.split('/')[2] }}/brightness\",  \"bri_cmd_t\": \"homeassistant/{{ trigger.topic.split('/')[1] }}/{{ trigger.topic.split('/')[2] }}/brightness/set\" }"
        retain: true

- alias: "MQTT Discovery Binary Sensor Creator"
  trigger:
    - platform: mqtt
      topic: 'homeassistant/binary_sensor/#'
  condition:
    condition: template
    value_template:  "{{ trigger.topic.split('/')[3] == 'state' }}"    
  action:
    - service: mqtt.publish
      data_template:
        topic: "homeassistant/binary_sensor/{{ trigger.topic.split('/')[2] }}/config"
        payload: "{\"name\": \"{{ trigger.topic.split('/')[2]| replace('_', ' ') | title }}\", \"pl_off\":\"off\", \"pl_on\":\"on\", \"state_topic\": \"homeassistant/{{ trigger.topic.split('/')[1] }}/{{ trigger.topic.split('/')[2] }}/state\"}"
        retain: true

- alias: "MQTT Discovery Sensor Creator"
  trigger:
    - platform: mqtt
      topic: 'homeassistant/sensor/#'
  condition:
    condition: template
    value_template:  "{{ trigger.topic.split('/')[3] == 'state' }}"   
  action:
    - service: mqtt.publish
      data_template:
        topic: "homeassistant/sensor/{{ trigger.topic.split('/')[2] }}/config"
        payload: "{\"name\": \"{{ trigger.topic.split('/')[2]| replace('_', ' ') | title }}\", \"state_topic\": \"homeassistant/sensor/{{ trigger.topic.split('/')[2] }}/state\"}"
        retain: true

And on the Zwave instance:

- alias: "MQTT Scene"
  trigger:
    - platform: mqtt
      topic: 'homeassistant/scene/#'
  condition:
    condition: template
    value_template:  "{{ trigger.payload == 'on' }}"
  action:
    - service: scene.turn_on
      data_template:
        entity_id: "scene.{{ trigger.topic.split('/')[2] }}"

- alias: "MQTT Script"
  trigger:
    - platform: mqtt
      topic: 'homeassistant_zwave/script/#'
  action:
    - service: script.turn_on
      data_template:
        entity_id: "{{ trigger.topic.split('/')[1] }}.{{ trigger.topic.split('/')[2] }}"

- alias: "MQTT Light and Switch Toggle"
  trigger:
    - platform: mqtt
      topic: 'homeassistant/switch/#'
    - platform: mqtt
      topic: 'homeassistant/light/#'
  condition:
    condition: template
    value_template:  "{{ trigger.topic.split('/')[3] == 'set' }}"
  action:
    - service_template: "{{ trigger.topic.split('/')[1] }}.turn_{{trigger.payload | lower }}"
      data_template:
        entity_id: "{{ trigger.topic.split('/')[1] }}.{{ trigger.topic.split('/')[2] }}"

- alias: "MQTT Light Brightness"
  trigger:
    - platform: mqtt
      topic: 'homeassistant/light/#'
  condition:
  - condition: template
    value_template:  "{{ trigger.topic.split('/')[3] == 'brightness' }}"
  - condition: template
    value_template:  "{{ trigger.topic.split('/')[4] == 'set' }}"
  action:
    - service: light.turn_on
      data_template:
        entity_id: "{{ trigger.topic.split('/')[1] }}.{{ trigger.topic.split('/')[2] }}"
        brightness: "{{trigger.payload}}"

- alias: "MQTT Cover Toggle"
  trigger:
    - platform: mqtt
      topic: 'homeassistant/cover/#'
  condition:
    condition: template
    value_template:  "{{ trigger.topic.split('/')[3] == 'set' }}"
  action:
    - service_template: "cover.{{trigger.payload | lower }}_cover"
      data_template:
        entity_id: "{{ trigger.topic.split('/')[1] }}.{{ trigger.topic.split('/')[2] }}"

- alias: "MQTT Lock Toggle"
  trigger:
    - platform: mqtt
      topic: 'homeassistant/lock/#'
  condition:
    condition: template
    value_template:  "{{ trigger.topic.split('/')[3] == 'set' }}"
  action:
    - service_template: "lock.{{trigger.payload | lower }}"
      data_template:
        entity_id: "{{ trigger.topic.split('/')[1] }}.{{ trigger.topic.split('/')[2] }}"
        code: !secret door_code

- alias: "MQTT Input Boolean"
  trigger:
    - platform: mqtt
      topic: 'homeassistant_zwave/input_boolean/#'
  condition:
    condition: template
    value_template:  "{{ trigger.topic.split('/')[3] == 'state' }}"
  action:
    - service_template: "input_boolean.turn_{{ trigger.payload | lower }}"
      data_template:
        entity_id: "{{ trigger.topic.split('/')[1] }}.{{ trigger.topic.split('/')[2] }}"

- alias: "MQTT Input Text"
  trigger:
    - platform: mqtt
      topic: 'homeassistant_zwave/input_text/#'
  condition:
    condition: template
    value_template:  "{{ trigger.topic.split('/')[3] == 'state' }}"
  action:
    - service: input_text.set_value
      data_template:
        entity_id: "{{ trigger.topic.split('/')[1] }}.{{ trigger.topic.split('/')[2] }}"
        value: "{{trigger.payload}}"

- alias: "MQTT Input Select"
  trigger:
    - platform: mqtt
      topic: 'homeassistant_zwave/input_select/#'
  condition:
    condition: template
    value_template:  "{{ trigger.topic.split('/')[3] == 'state' }}"
  action:
    - service: input_select.select_option
      data_template:
        entity_id: "{{ trigger.topic.split('/')[1] }}.{{ trigger.topic.split('/')[2] }}"
        option: "{{trigger.payload}}"

What this setup does is automate the creation of several entity types on the main instance. After a LOT of trial and error I found this to be the best setup. I particularly struggled with MQTT Lights as I really wanted control of power (on/off), brightness, and transition (ie how quickly to change the brightness). Transition proved to be the largest obstacle and I just couldn’t get it to work despite hours of trying. I also got things working in one way only to realize I had broken it in another way.

Eventually I decided on a work around. I moved all the scenes that relied upon transition to my Zwave instance and used MQTT to call the scene via an MQTT Publish action. I did the same with scripts, hence the two “extra” automations on the Zwave side. It’s not as clean or simple as I would have liked, but it absolutely works and works well.

Two other things that took a little bit of trial and error involved my locks and garage door. Both involved tweaking the value_template for the state of the entity. For example, to close the lock, the command was “lock” or “unlock.” However, the state reported is “locked” or “unlocked.” So to avoid any warnings/errors, I added value_template: "{{ value | replace('ed','') }}". Similarly for the garage, the states reported by the door were “open,” “opening,” “closing” and “closed” while the MQTT Cover component only supports “open” or “closed” so I would get an error with “opening” and “closing” states. The following fixed that issue:

    value_template: >
      {% if value == 'open' or value == 'opening' %}
        open
      {% else %}
        closed
      {% endif %}

I also ran into some issues controlling my door lock codes but that was relatively easy to get working once I got an MQTT Statestream working in reverse to send the states back to the Zwave instance.

Alternate Options

Recently, zwave2mqtt has started being developed. I looked into this option but ultimately chose not to use it, primarily because I wasn’t confident that it could handle some of the automations I had pre-existing for controlling my door lock codes and some other niche things I do. It very well may (and I think it’s a good option for some). But until it’s a little further along I think this is a better option. That’s my own opinion, take it for what it’s worth.

Conclusion

So was it all worth it? Maybe. I do enjoy being able to tweak/update things without having to take down the zwave network constantly and I think in the long term that’s going to be useful. But this project required a ton of time on my part, largely because I couldn’t find a good guide to walk me through it. Anyway, if anybody else is thinking of doing something similar I hope this helps.

I also want to thank the people who helped me in various threads while I was doing this. This really is a terrific community!

1 Like

This seems very complex compared to just running zwave2mqtt on the other computer.

I don’t believe zwave2mqtt can yet do some of the things I need it to, as I said in the last paragraph.

Just to clarify a bit, if I want to forward a switch, what I need is:

  • MQTT Light and Switch Toggle in automations.yaml on the zwave server
  • this, in configuration.yaml on the main server:
mqtt:
  broker: MY_IP
  username: MY_USER
  password: MY_PASS
  client_id: ha_virtualbox
  keepalive: 60
  discovery: true
  discovery_prefix: ha_master_topic
  • MQTT Discovery Switch Creator in automations.yaml on the main server
  • this, in configuration.yaml on the zwave server:
mqtt_statestream:
  base_topic: ha_master_topic
  publish_attributes: true
  publish_timestamps: true
  include:
    entities:
      - switch.fibaro_system_fgwpef_wall_plug_switch_5

Am I missing something? This does make the switches work on the main server, just somewhat unreliably.

Yes, but your state/command topics are going to depend on what your switch reports to MQTT. Use MQTT-Explorer or some other app to see what it’s publishing. You’d also need to change the topic from what i had in my example (homeassistant) as you have ha_master_topic.

But I don’t have state/command topics anymore, after getting the switches discovered via your code above, they generally work, I erased this kind of stuff:

  - platform: mqtt
    name: "Air Humidifier"
    state_topic: "ha_master_topic/switch/greenwave_powernode_6_port_switch/state"
    command_topic: "ha_master_topic/switch/greenwave_powernode_6_port_switch/set"
    availability_topic: "ha_master_topic/zwave/greenwave_powernode_6_port/is_ready"
    payload_on: "on"
    payload_off: "off"
    state_on: "on"
    state_off: "off"
    payload_available: "true"
    payload_not_available: "false"
    optimistic: false
    qos: 0
    retain: true

Do I still need to have this kind of code?

You need to modify my code to work with your switch as my state/command topics were different and i don’t have an availability topic. All my code does is streamline the generation of the config for the switches. So the automation should generate a config that is identical/similar to what you have in your post.

I did modify your code, because I have a different topic I wrote this (here and everywhere else you had the homeassistant topic and I had ha_master_topic):

- alias: "MQTT Light and Switch Toggle"
  trigger:
    - platform: mqtt
      topic: 'ha_master_topic/switch/#'
    - platform: mqtt
      topic: 'ha_master_topic/light/#'
  condition:
    condition: template
    value_template:  "{{ trigger.topic.split('/')[3] == 'set' }}"
  action:
    - service_template: "{{ trigger.topic.split('/')[1] }}.turn_{{trigger.payload | lower }}"
      data_template:
        entity_id: "{{ trigger.topic.split('/')[1] }}.{{ trigger.topic.split('/')[2] }}"

The code I posted above for the Air Humidifier I used before implementing your discovery stuff, right now I don’t have any such entries (and none were automagically generated) in my configuration.yaml, shoud there be such mqtt switches there?

In any case, when I get home I’ll try MQTT-Explorer and see what I can find…

I’d start by trying to get the switch working without auto discovery. Aka manually configure it. You may have to tweak more than just the topic name, there may be extra "/"s in there for you. You can adjust it by changing the [1]/[2] etc after the .split. For example, with:

ha_master_topic/switch/greenwave_powernode_6_port_switch/set
0 is “ha_master_topic”, 1 is “switch”, 2 is “greenwave_powernode_6_port” and 3 is “set”

Interesting info, thanks!

This is the automation that I have in the zwave instance for the above Air Humidifier:

- alias: control_air_humidifier
  trigger:
  - platform: mqtt
    topic: "ha_master_topic/switch/greenwave_powernode_6_port_switch/set"
  action:
  - service_template: "switch.turn_{{trigger.payload | lower}}"
    entity_id: switch.greenwave_powernode_6_port_switch

Thus, 3 seems right for it…and it does work…nearly all the time, but not all the time like when I press the original item itself in the zwave instance.

It may take a few seconds for it to register on the other instance if you’re toggling from the zwave side. It has to poll the state which takes a bit.

FWIW, there’s also a custom component, called Home Assistant Remote, that uses WebSockets to connect multiple instances of Home Assistant:

1 Like

That’s true, I just couldn’t get it to work on my setup.

Really? This would make it a non-starter for usage “in production”, there’s no way it’s worth waiting a few seconds for it to poll the zwave state and only afterwards send the command to a switch.

How is your performance with this solution, btw? Is it as quick as the proper zwave integration in your first instance or is there a noticeable difference?

I’ll check out Home Assistant Remote, but somehow I imagine that websockets might be even slower than MQTT…

Might give up on this for now, I have no idea why there are these issues, MQTT with the Monitor script on a separate Pi is very quick.

Well, in my use case I’m controlling from the secondary instance, not the zwave instance. So i haven’t noticed things getting out of sync/not working as you seem to be describing. Going the other way (from secondary instance to zwave) is nearly instantaneous, maybe a tiny delay but it’s not noticeable. The only slowness that is currently irking me revolves around my garage door but I’m still exploring the cause there.

There are times I’ll toggle a switch off on the secondary instance and it will turn off the light, but the state won’t be updated for maybe 2-3 seconds as i’m not running in optimistic mode (I’m relying on it using the state topic to get the state).

1 Like

ESPHome communicates with Home assistant via its WebSockets API. I haven’t read any complaints about poor performance in the ESPHome-related threads. However, as always, YMMV.

Right, so I started from scratch with this and so far so good!

I just removed the availability_topic and it seems to have made a difference and I’ve also taken into consideration that sometimes the status updates in 2 seconds, 100% success in flipping switches so far :slight_smile:

Awesome! And yeah, the status may not update instantly but the light should go on/off almost instantly. It’s just the status that slightly lags.

Thx! Mark to read