Modbus TCP integration

Third party integrations
1

Hi all, I have a question related to the modbus TCP integration home assistant has, specifically how to improve the speed at which inputs can be read.

TL;DR: how to optimize I/O for a master / slave setup that inherently requires polling, such as modbus TCP? Should I create an asyncio-like pymodbus library?

update Jan 2020; the unipi platform provides a suite of services called evok that exposes the events via a websocket. From that point, I could easily write my own websocket-to-mqtt translation, see https://github.com/mhemeryck/evok2mqtt

Background: Unipi – PLCs and modbus TCP

Some background: I am currently in the middle of the process of constructing a new house. I will do the wiring of the electricity myself; the idea is to have all wiring coming together at a central point, where I can add my own controller running home assistant.

I will specifically be using a (number of) unipi neuron modules, as controllers, which are essentially a raspberry pi with added hardware to interface with I/O at 24V (pretty much the standard for wired home automation).
See the figure for the involved hardware.

unipi layout
unipi layout1684Ă—1191 78.9 KB

The CPU on the figure is effectively a raspberry pi, which can run raspbian (or HASS OS); the ARM processors are slave devices that can be read through SPI. The I/O values can be read from the CPU device via modbus RTU or modbus TCP via their unipi Neuron modbus TCP overlay

Anyway, my question isn’t really unipi-specific – it involves any device that exposes I/O over modbus TCP, most notably PLC-like devices.

I already found out that the existing home assistant modbus integration is able to directly poll the modbus TCP server with minimum configuration, based on the pymodbus library for this. I actually made a forum post about this at the unipi community. As it’s a master / slave setup based on SPI, there’s no real other way than polling the slave device (reading through some docs, SPI does support an interrupt-mode, but I would probably not even know how start here :slight_smile:). The response time in home assistant does however seem quite slow for my setup (e.g. toggling light switches). I also fear that it will not scale that well when adding more inputs as this would mean just more polling as it seems the current implementation tries to poll for each I/O individually.

I did some experiments yesterday with the pymodus library on the unipi to figure out the bottlenecks in the setup; doing continuous polling on the modbus TCP server revealed it pretty much directly exposes the changed input state (so the obvious delay is at the side of home assistant). Given that the protocol itself did turn out to not be that complex, I am thinking of cooking up a more optimized way of polling the modbus TCP server.

Ideas for speedup

Asyncio: reading through the developer docs, it seems pretty much all of the core of home assistant shifted its behavior to use asyncio. Given that the pymodbus library does not have this behavior, I could try building an async version of it. Home assistant would still need to poll it though, but at least, it would not be blocking anymore.

Reading multiple inputs at once: home assistant seems to treat input individually, leading me to believe that each of them are polled separately – whereas the modbus protocol itself allows for multiple values to be read at the same time (!). I am not sure what would be the best way to integrate this into home assistant again though, as I think the binary_sensor / switch (component) abstractions are great.

Translate to MQTT first: I could also just write a separate abstraction layer that would do the polling and then push out (e.g. MQTT) events in case it detects a change. I would not be a big fan of this though, as it would just add another layer in between.

My own evaluation at this point: create an async modbus TCP client, check its performance and see whether polling multiple inputs is still a requirement then.

My question really is how big of speedup I should expect from asyncio part; whether it’s really worth investing time.

Additionally, any other suggestions / ideas / … are also welcome!

2

Answering my own question here: I did eventually go for the option of building a separate service between modbus and mqtt (“modbridge”). Basically, it polls the modbus TCP server and pushes out any changes it finds via MQTT.

I already have a working proof-of-concept here: https://github.com/mhemeryck/modbridge

I’m hoping to keep on improving it and ideally make it general enough such that it’s not constrained to this specific unipi / neuron setup.

1 Like
3

Hey Martijn,

I’m also implementing UniPi Neuron with Home Assistant for our rebuilt house.
We move in within 10 days, and still have to figure out how to do the neuron-HA stuff, while finishing drywalls and sanding ceilings :slight_smile:

Thanks for your information, I will follow it closely, and maybe give your modbridge a spin.

Biggest issue for me right now is adressing my neuron extension module, need it for all the lights in the kitchen, but HA only supports one modbus connection. Maybe I should try gong through evok, but I’m afraid that will slow things down even more…

Maybe I found a way to add a second modbus (copy and rename definition files to something like modbus-xS10 or so)

When I’m done figuring out that part I can worry about the speed… I surely hope it will be OK for controlling the lights.
If too slow, I will try to use the direct switch option…

Keep up investigating ! And if I can help you out (doubt it) I would gladly do so

Best regards,
Steven (from Belgium)

4

Hey Steven,

Thanks for reaching out. (Re)building houses is already quite intense, so anyone else willing to have a look at this is surely welcome :slight_smile:

About the neuron extension modules: I haven’t had a look at these, rather I bought an extra unipi module. Contrary to the built-in I/O groups, they do not communicate over modbus TCP, but rather modbus RTU. The two protocols are rather similar though (modbus TCP is just modbus RTU minus some headers) and I believe the current HASS integration already supports both. Check this explanation on modbus if you’d like some more information (it’s actually dead simple).

About modbridge: I have been slowly adding new features over the past couple of weeks; check the modbridge roadmap and modbridge release log for more info. I tried to keep it as generic as possible, but atm I guess it’s still quite tied to the unipi neuron-specific setup. If you’d like to test it on your own setup, you should:

  1. get the binary for your own platform (both x64 and arm are built – the arm one does not seem that stable atm though)
  2. download the example config.yml file and adapt for your own setup. Check the modbus register map files for this. Note that only coils are polled atm though and only over modbus TCP

About speed: at first, I also did try to do the polling directly from home assistant, but it didn’t really seem feasible to me. Also, I’d rather keep the polling part out of home assistant; for a light switch, there shouldn’t be any polling at all. This also made it possible for me to use golang, which makes it easier to do concurrent polling (also, I just wanted to learn the language – despite being a professional python dev).

The direct switch option: that should actually be the fastest option as it handles the input change directly on the I/O group. I didn’t really look into this as this has the inherent downside of requiring all of the input / outputs directly matched and I also just would like to read them into a central component first (like home assistant).

Anyway, good luck on your setup, I’be happy to hear from your experiences!

Best,

Martijn (also from Belgium aka the only country where people are that crazy to build / wire their own house).

5

Hi Martijn,

Seeing your profile pic I knew you must’ve been from Belgium. Beer !! :slight_smile:

I’ve tried using EVOK+REST, but that seemed too slow, and that way I miss pulses from the inputs. I needed something more interrupt-like.
Next try was evok-to-mqtt, but that seemed also quite slow, although I based this experience on console output from the daemon and not the response times from HA.

Last (current) try was creating multiple modbus-plugins in HA (HA only supports one, and because of the xS10 extension I need 2 modbusses). This works for me, and with only 1 binary sensor it looks usable in terms of performance.
Worst case I can manage the Direct Switch functions from Home Assistant to get started (lights have to work in 3 days :face_with_raised_eyebrow:)

I was wondering: what is the difference between the modbus polling from HA and the polling you created in your modbridge ? Is there a big difference in performance, and how comes ?
Does it support multiple modbus connections ? (one over TCP, one RTU)

Thanks for letting me know and the effort you are doing for the community.
Maybe the Neurons will get more popular for home automation this way :slight_smile:

Steven

6

Hi Steven, in response:

EVOK+REST: that would mean you would also need to poll the REST API exposed by EVOK, right? EVOK really is just an API layer on top of the modbus TCP server. Adding a layer typically means more complexity, more latency. If you’d look at the source code, you would just find that it’s also just polling with a modbus client and exposing it via a REST API layer (or websockets). I hadn’t really seen evok-to-mqtt, interesting to learn about. I should have a look at this myself, just to see how other people have done this.

The modbridge idea is pretty much a similar principle: it’s modbus-to-mqtt, so effectively bypassing the EVOK-layer.

As for polling from HA: that’s an interesting question, since I’ve also looked into that first. Before looking into my own custom solution, I also did take some time reading through the HA dev docs. You have to know that HA uses something like an event bus to schedule all its actions and have the automations act on events that appear in it (broadly speaking). For its integrations (e.g. modbus), it requires that they are made available as separate open packages. I suspect they do this for:

  1. Keeping the code base a bit more clean and lean
  2. Assuring that any integrations that are written stand on their own and can have their own community
    For modbus, they use pymodbus

As all these integrations all have their events on this same bus, this means that doing a continuous polling would mean a lot of events on this system – while in principle it is meant for true events, e.g. a single MQTT event to gets picked up and processed. In the case of the HA modbus component, you can see there’s a scan_interval attribute to set. From looking at it’s default value which is in the range of seconds, you can already derive it’s meant for scanning slow-changing values like a temperature sensors, rather than polling the status of a push button.

A note on HA and asyncio: one way to speed up the processing of events is making using of the async processing capabilities python and by extension HA now offer. The idea is that events are processed asynchronously i.e. rather than having all of the event actions occurring sequentially (and blocking on I/O actions which are typically slow). they now allow themselves to be interrupted so that other work can continue. Check the excellent video on asyncio 101 for more info on this. Specifically for modbus there’s a catch: in order for the event bus to perform the polling asynchronously, the integration needs to be adapted for asyncio itself. One of the ideas I had was thus to port that library to asyncio.

Rather than spending my time on porting the library to asyncio, I figured a custom solution based on golang would actually be more interesting:

  • separation of concerns: the principle of polling does not fit into the event-like system HA has, so put it in a separate service that sends out one-time MQTT events (which HA is well capable of handling)
  • deployment: related to that, I would be able to run the modbridge service and my HA service separately.
  • concurrency in golang vs python: while I think asyncio is a great addition to python, it still seems very new to me. Golang has the principle of concurrency in the core language itself and depending on the platform you’d be running it on, it can also run it in parallel.
  • modus polling for home automation: apart from HA, I have come across a number of home automation projects that rather than a unipi neuron use a true industrial PLC as their hardware (e.g. Siemens Logo, Wago PLC, …). I am guessing this was pretty much the reason why the unipi ppl figured to use modbus themselves. For all of these projects, something like modbridge would also provide an easy to use component for their solutions – so it goes beyond just HA.
  • learning golang: as a professional python developer, I’ve been wanting to do a project in golang for quite some time. This was a perfect occasion :slight_smile:

For modbus RTU vs modbus TCP: I haven’t actually tried modbus RTU. The golang modbus library I am using for it actually does support it, so it shouldn’t be that hard to implement it. Running multiple next to each other also needn’t be an issue. My goal is eventually to run it in docker, so that I can just run it as another image in hassos.

The current status is that I can actually run modbridge standalone for my unipi neuron L303 – so you could try that. Otherwise, you could best opt for the direct-switch option though. See my roadmap for a list of features I’d like to add. The main change I am currently working on is polling an entire group at once instead of individual coils – but for now, the individual coil polling seems faster for some reason :thinking:

Hope this gives some more background on what I’ve been working on :slight_smile:

7

Hi Martijn,

I tried the evok-to-mqtt project I found on GitHub, but suspect that it’s just a one-shot try and not completely what I need, so I decided to take my first baby-steps in Python. That way I’m sure I can finetune it in a way that is needed for my project.

Goal was to create some script that can easily run on the standard evok rpi image without adding too much overhead, and to be able to separate evok+script from HA when adding another RPi to the setup.
I also wrote a script to generate a custom-built EVOK package file for HA.

If you’re curious, you can take a look at my code at https://github.com/StevenBrs/evok2mqtt
All comments are welcome, so I can learn Python better. (I’m an ITer, but not a Python programmer :slight_smile: )
I didn’t take a look at your code, but it wouldn’t surprise me if it is similar to your project.

So far I get really good performance. Only problem at this moment is that sometimes the evok daemon looses connectivity with Modbus (and my xS10 extension), making the lights in the kitchen inoperable.
Although I think that’s a problem for the Unipi-people, I will try to find a solution for that too.

8

Hi Steven,

Just had a quick first look at your code and if I understand it correctly, you simply register a callback on the websocket exposed by evok, right? I was unaware that evok exposed the changes on the inputs as events on the websocket, so this makes programming against it much easier, good find!

I am curious how evok does this then, since I couldn’t find it myself directly in the source code. As evok is also based on the modbus TCP server, I suppose it should also do some kind of polling on it? I will surely have a look at that code again, thanks for the pointer!

edit: just looked through the evok code again, and I did indeed notice that it is also doing some async pollng based on https://www.tornadoweb.org/en/stable/ – asyncio’s predecessor. Perhaps lowering the scan interval might improve the load for my setup :thinking: (they only have a 10Hz scan interval by default)

I also looked at the updated docs and I noticed they also now have a custom-built debian package for that. It’s pretty neat they now offer this out of the box.

I also dug up an older thread on spidev which I asked about earlier. The thing is, in order for the modbus TCP server to have it’s values available, it needs to have the values available on the main device while reading the SPI devices. This mean that updated values from the SPI devices are made available every scan interval (which is pretty fast). This would be the ideal level to register something like a callback, since you would remove the step of polling the modbus TCP server. Turns out they now offer a kernel module that exposes the SPI device values in a sysfs-fashion, for you can also register callbacks. It all seem rather low-level C / linux tinkering though, so I am not quite sure whether it’s worth the effort :thinking:

9

He Guys, I wrote something on top of EVOK, perhaps it can help 1 of you. It’s fast enough for what i do with it (Home Assistant). Warning, I’m not a python dev guys, so if you have tips for the code, please do!

https://github.com/matthijsberg/unipi-mqtt

10

I have found that the toggling of switches is instantaneous but that the indication of status (via binary sensors) is the real problem.

I have found as a consequence that automations relying on feedback are very slow. For this reason I have kept my automations on my micro plc’s.

11

Hi everybody, fellow Belgian here I’ve rebuild my home with 24v switches and just bought a unipi 1.1. No I’m looking for the best way to integrate into hass. First I created myself some hardware and a python script: https://gist.github.com/Sennevds/98e2f2340eac7323f2723de353fcdd9f
The problem was that my hardware skills aren’t that good and it triggered randomly so I thought I go with the unipi.

So the question does anyone have a good working system? I only need a few 24v inputs and some 24v output(relays)

12

Been a while since I first posted this and frankly, I’ve changed my setup already a couple of times :slight_smile:

Currently, I just use the evok service that’s made available by the unipi guys themselves. As part of evok, a websocket interface is made available to interact with any incoming events; see https://evok.api-docs.io/1.0/mpqzDwPwirsoq7i5A/websocket . In that sense, I don’t do any modbus-like interfacing anymore, just websocket-to-mqtt. From what I understood, evok should also support unipi 1.1

My current implementation for “evok-to-mqtt” is in https://github.com/mhemeryck/evok2mqtt. Note that it does both evok-to-mqtt events as well as incoming mqtt events to evok triggers.

After you have the MQTT events readily available to push to / from home assistant, it’s quite straightforward to link inputs to output in home assistant.

One day … I should really do a write-up of all this stuff :slight_smile:

13

I already saw your git. I’m currently using this one: https://github.com/matthijsberg/unipi-mqtt from @matthijsberg which has some nice features like turn on a relay for a amount of time(handy for my dimmers).
The only problem I still have is that I get false positives on my inputs and haven’t figured it out how to filter these out. I’ve explained here https://forum.unipi.technology/topic/1131/digital-input-ghost-detection

1 Like
14

Is your issue related to the eltako dimming operation? How does the dimming work, analog 0/1-10V / PWM-like signal, …? I do have 5-wire cabling for all light sockets, with the idea to eventually add DALI-based dimming (https://www.unipi.technology/unipi-axon-s605-p179).

15

I thought it was a dimming problem but I’ve added an extra light(one where no dimmer is attached) and this sometimes toggle’s a other light.

It’s just a 24v signal and the longer you press it dims or brightenss the light attacht to the dimmer.
All my switches are connected with a svv cable and I think the end off the cables are to close to eachother so if I press somewhere it gives a signal to another as well.

16

OK, that does sound like a PWM-like dimmer. I’m wondering how you read the time the button is pushed with just the websocket approach, since I don’t see this time exposed there.

17

At this moment I don’t check the input signals of the dimmers.
I only need the signals of the relais at the moment. Only need to figure out the false positives

18

I fixed the false positives the developer way. I can’t figure out how to filter out the false positives with hardware so I’ve added a extra check in the python script that if a certain circuit is triggerd to check for another circuit and if that value is high stop checking.

It’s quiet a dirty hack but it works

19

thanks for mentioning this! Great to hear someone is actually using it. Please let me know if there is anything I can improve on. It’s definitely not complete; mostly written to my personal needs.

In terms of dimming, I just use the analog output of my UniPi and use this 0-10 volt output to steer led drivers with this as input (if your google for 0-10 volt led driver you’ll find a lot). Do make sure you use 0-10 volt, not 1-10 volt as that won’t turn off the led lights completely (you need to cut the 110 / 230 voltage to the driver in the 1-10v case).

Dali would be really cool, but since the 0-10 volts works really nice for me and I would need to buy a a new UniPi and LedDrivers that’s not an option right now.

Working on adding some additional 1-wire devices right now. Will adjust the code to make sure this works. Note: correct 1-wire values depend on EVOK version, some older versions gave me back incorrect values for some sensors.

20

Can you share an example of the false positives?