JNS IoT Gateway – lightweight resilient external Zigbee, Thread and Matter gateway architecture

Development
, , , ,
1

We are actively developing a lightweight, modular external radio gateway architecture for Home Assistant and would welcome feedback from people working with ZHA/zigpy, Zigbee2MQTT, Thread/OpenThread Border Router and Matter.

The project started from a redundancy problem: when a Zigbee coordinator is physically attached to a Home Assistant VM host, failure or maintenance of that host can also remove access to the Zigbee network.

Our aim is to separate the radio/device edge from the Home Assistant compute layer while continuing to use established Home Assistant and open-source technologies rather than replacing them.

Initial implementation

The first profile is deliberately simple:

Zigbee coordinator → dedicated Linux gateway → ser2net → wired LAN → Home Assistant ZHA

Home Assistant accesses the coordinator through a stable network endpoint.

Only one ZHA instance is allowed to own the coordinator at any time.

The gateway can initially run in a small Debian-compatible environment such as an LXC or Raspberry Pi/SBC.

We are specifically designing the core to have very low resource requirements. The JNS management component does not need to remain resident as a permanent daemon; profile application and health checks can run as oneshot/event-driven tasks while only the selected protocol service remains running.

One codebase, selectable modules

Rather than creating independent projects for every radio technology, the gateway has a small common core and selectable profiles.

Current/planned profiles are:

  • zigbee_serial — ZHA through ser2net

  • zigbee2mqtt — dedicated Zigbee2MQTT gateway

  • thread_otbr — OpenThread Border Router

  • matter_thread_edge — Thread/Matter integration helpers

  • maintenance

  • disabled

A major design rule is that unused functionality should not simply be disabled — it should not be installed or loaded unless the selected profile requires it.

For example, a zigbee_serial gateway should not be carrying a dormant Zigbee2MQTT, OTBR or Matter stack.

This is intended to keep CPU, RAM, storage use, maintenance burden and attack surface small.

Zigbee2MQTT

A future Zigbee2MQTT profile will use the same gateway platform but Zigbee2MQTT will own the locally connected coordinator directly.

We do not intend to leave ser2net unnecessarily between Zigbee2MQTT and a coordinator attached to the same gateway.

The purpose of the profile abstraction is to allow deployments to evolve without replacing the whole gateway management architecture.

Thread and Matter

We are deliberately treating Thread and Matter as different layers.

Thread is the IPv6 mesh/radio network and will use an established OpenThread Border Router implementation with a dedicated Thread radio.

Matter is treated as an application protocol operating over IP, including Matter-over-Thread.

We currently favour dedicated Zigbee and Thread radios rather than building the architecture around simultaneous multiprotocol use of a single radio.

This also allows multiple Thread Border Routers to be deployed later where useful for coverage and resilience.

Moving the radio away from virtualization hosts

A later stage will move the Zigbee coordinator onto a dedicated wired Raspberry Pi/SBC.

The resulting topology is approximately:

                 Zigbee devices
                       |
                Zigbee coordinator
                       |
              JNS IoT Gateway
              Raspberry Pi/SBC
                       |
                  wired LAN
                       |
             stable gateway address
                       |
             +---------+---------+
             |                   |
       Home Assistant A    Home Assistant B

The objective is that a virtualization host failure does not also remove the physical radio.

Home Assistant continues to use the same logical coordinator endpoint regardless of which HA instance is operating.

Future redundant coordinator research

When additional hardware is available, we also intend to investigate two matched Raspberry Pi/SBC gateway nodes with compatible Zigbee coordinators.

This would be active/passive, never two active Zigbee coordinators controlling the same network.

Conceptually:

Gateway A + coordinator A
          ACTIVE
             |
          Zigbee
             |
       logical endpoint
             |
        Home Assistant

Gateway B + coordinator B
         STANDBY
         FENCED

The standby would contain compatible gateway configuration and validated recovery information, but its coordinator would remain inactive.

The areas we intend to test include:

  • coordinator/network backup compatibility;

  • recovery onto a replacement coordinator;

  • matched coordinator hardware/firmware;

  • stable virtual service addressing;

  • coordinator identity handling;

  • device rejoin behaviour;

  • recovery following Raspberry Pi failure;

  • recovery following USB coordinator failure;

  • strict prevention of simultaneous coordinator ownership.

We are deliberately treating this as a future experimental stage until we have the hardware available and can repeatedly prove the recovery procedure.

Network syslog and observability

Each gateway also supports lightweight structured network syslog.

Syslog is intended for operational, security, audit and health events such as:

  • gateway/profile start and stop;

  • radio detected/missing;

  • module failure;

  • health degradation/recovery;

  • configuration validation;

  • coordinator ownership changes;

  • update/recovery events.

We intentionally do not intend to push high-frequency Zigbee/Thread/Matter device telemetry into syslog.

A failed logging server must not interfere with the radio service, so failed messages use a small bounded local retry queue.

Upstream-first approach

This project is intended to integrate existing projects rather than duplicate them.

Where appropriate we plan to use and work with:

  • Home Assistant ZHA/zigpy

  • Zigbee2MQTT

  • OpenThread Border Router

  • existing Home Assistant Matter support

  • MQTT

  • standard IPv6/mDNS behaviour

  • RFC 5424-compatible syslog

If development identifies generally useful fixes, interoperability improvements or documentation changes, we would prefer to contribute those improvements upstream where practical.

Public/open versus private infrastructure

We have deliberately separated the public gateway work from our private resilience and infrastructure control plane.

The following areas are public and open for discussion/development:

  • gateway architecture;

  • profiles and module interfaces;

  • ZHA/ser2net interoperability;

  • Zigbee2MQTT integration;

  • OTBR/Thread integration;

  • Matter interoperability;

  • radio identification;

  • resource-efficiency work;

  • generic recovery compatibility;

  • health interfaces;

  • syslog;

  • failure/recovery testing methodology.

Our private infrastructure layer is responsible for some wider deployment and resilience functions.

We will describe externally important guarantees — for example:

Only one Zigbee coordinator owner may be active before a standby is promoted.

However, the security-sensitive implementation of the private orchestration, authorization, cryptographic distribution and rollback systems is intentionally outside the scope of the public gateway project.

This separation is also intended to prevent the gateway becoming dependent on our private infrastructure: its public protocol modules should remain useful with standard Home Assistant technologies.

Current status

Already implemented:

  • common lightweight gateway core;

  • profile architecture;

  • zigbee_serial profile;

  • ser2net coordinator configuration;

  • persistent /dev/serial/by-id/... device binding;

  • single-client/no-preemption policy;

  • maintenance and disabled profiles;

  • network syslog over UDP/TCP/TLS;

  • bounded syslog retry;

  • low-frequency health checks;

  • future Zigbee2MQTT, Thread and Matter profile definitions.

Next hardware-testing stages:

  1. Commission the existing Zigbee coordinator through the gateway.

  2. Validate ZHA operation over a wired network.

  3. Test Home Assistant instance handover while keeping the same coordinator.

  4. Move the coordinator onto a dedicated Raspberry Pi/SBC.

  5. Develop and test the Zigbee2MQTT profile.

  6. Develop the Thread/OTBR profile.

  7. Validate Matter-over-Thread operation.

  8. When additional hardware is available, test controlled recovery onto a second matched Zigbee coordinator.

Feedback we would particularly value

We would be interested in feedback on:

  • assumptions about ZHA/zigpy coordinator migration or recovery that we should avoid;

  • whether there are existing upstream projects solving parts of this that we should reuse;

  • best practices for a wired ser2net/ZHA deployment;

  • coordinator families best suited to repeatable replacement/restore testing;

  • how best to expose gateway health without coupling it unnecessarily to Home Assistant;

  • OTBR deployment considerations for multiple independent border routers;

  • Matter/Thread architectural assumptions we should account for;

  • anything in this design that conflicts with planned Home Assistant architecture.

The public roadmap and the project’s publication boundary are available in the JNS Home Assistant Deployment repository.

The intention is to develop this openly where it can benefit the wider ecosystem while keeping unrelated security-sensitive infrastructure implementation separate.

2

Why though? Ethernet/PoE coordinators have been a thing for years now. They simply go through Zigbee coordinator → wired LAN → Home Assistant ZHA without requiring additional hardware in between.

Most major manufacturers (as well as minor ones) already sell these, and they already support ZHA, Z2M or Thread today, without having users having to wait for some future implementation.

I don’t mean to sound harsh, but the OP reads like some AI-hallucination with no grounding in the real world. You’d be spending loads of time & money developing something that is inferior to the current offerings for no reason.

3

Thank you for the reply.
Yes I am aware that ethernet poe coordinators are available but i am on a limited budget and looking at ways to reuse my existing hardware to improve redundancy. It also link to other projects I am currently working on but thought i would get some feedback from the community before just blindly progressing on my own path.
And yes it is AI assisted as I am engineer not a coder.

Once again feedback is greatly appreciated.

4

If you are looking at hardware redundancy, I am not sure there is a way to achieve that without purchasing 2 of each piece of critical hardware…

Logically if you do that, you just need to ensure your “standby” hardware:

  • Has been flashed so the co-ordinator has the same IEEE address as the primary.
  • That the HA configuration is synced regularly to the the backup hardware.

If you are okay with manual cut over you just need a script you can run on the backup to bring it up/live.

Trying to automate the cut-over is more tricky as you need the backup to be able to reliably detect when the primary is offline (I am not sure if there is anything in the Zigbee protocol to help you with that). It will get more tricky if you want to support multiple protocols, but the easiest solution is still going to be Primary + Standby hardware with a cutover which switches the roles.

There is is a YAGNI argument here, perhaps you should purchase 2 coordinators so the first bites the bullet you have a spare on hand, but I just don’t see the ROI in having 2 of everything, the cost of recovery in the case something dies is likely going to be roughly the same as trying to setup a high availability system in the first place (and that assumes that a hardware failure even occurs).

5

I already have a reliable system for the cut over. The basic idea is that this can be deployed into my mothers home and active redundancy. she is located the other side of the country and i need the home assistant to be self healing if hardware fails. This is part of an larger project to make home assistant have an active redundent backup. so that any hardware failure is mitigated hence me looking into ways of making the zigbee controller with an active redundent backup using cheep usb controllers as these are readily available where as poe adapaters would need power injectors as no poe switch on site.

6

I run HA dockerized and have moved HA between hardware - specifically:

  • Stop the containers.
  • Copy the config directory (or directories if also using Z2M).
  • Move the USB coordinators to the new hardware.
  • Start up the docker containers on the new hardware.

The untested piece (in my case - others have done it) is replacing a USB coordinator where the replacement has been flashed to the same address - that should be transparent, but I haven’t tested it.

The piece I don’t have is a way of detecting a failure inside the coordinator - for simplicity lets assume that the antenna snaps - I don’t have any way to detect that the antenna is radiating an RF signal (at full power).

So if I was to purchase a second mini PC and a second ZBT-2 (coordinator) I could run the coordinator test - I outlined above, then setup a standby HA installation.

Writing the scripts to stop and start docker containers on both instances is trivial.
That would give me a Primary/Standby with manual cut over.

With respect to automatic cut over I have a problem of how to identify if a failure occurred. I can come up with some heartbeat based solutions, but everything I come up with has a corner case / no reliable way to ensure exactly 1 instance is online at any given time, I can do:

  • 0 or 1
  • 1 or 2

Just not, exactly 1.

7

I am in the process of doing exactly that with a fenced secondary home assistance instance kept fully in sync with the primary. The failover heartbeat is also monitoring the zigbee network so it can switch to the second (networked) usb zigbee device and restore the current zigbee network. it still takes up to 10 mins for the zigbee network to all roll over and some devices need a manual push but so far testing is looking positive.

My previous way was a standby home assistant instance that rebuilt from the latest backup. but the last live test ( not planned ) took 1hour plus for the backup to get me back to a working state.

8

a quick screen shot of my cluster control dashboard.

image
image1677×1022 98.3 KB

9

You may want to consider Dockerizing:

  • 10 seconds - Stop HA
  • 0.7 seconds - Copy the config directory (I did it locally, but scp shouldn’t be much slower)
  • 25 seconds - Start HA

Timings above from old hardware - actually a 11 year old mac mini running Ubuntu linux.

As noted above with a “cloned” USB coordinator the transfer should be instant - again I don’t currently have the hardware to test that **

** - I am giving serious consideration to picking up another ZBT-2 when they are back in stock.