Graph your Z-Wave mesh (Python, auto-update)

OmenWild · January 24, 2018, 3:49am

I was intrigued by the Z-Wave connection graph graphs and by the hagraph and figured I could make a Python script that reads the Z-Wave mesh information directly from HA using the API. So I present to you, the Z-Wave mesh grapher.

I use split config files and the repository reflects that. Anyone with a similar layout should be able to drop the files into the right location and be good to go. By default an automation script will update the graph every 5 minutes.

There is a single secret needed for the full URL of the iframe:
z_wave_graph_url: http://YOUR_DOMAIN_HERE:8123/local/z-wave-graph.html

Other than that, I hope it will pretty much just work for folks.

I would be especially interested in seeing graphs from networks with more nodes. I’m still in the process of converting from SmartThings so most of my devices are still on that network.

If anyone knows Graphviz I would really love to make the nodes closer together and increase the font size. neato gave me the best layout with my nodes, but the spacing is really wide and I couldn’t get the font larger in the nodes. It seemed to mess up the fonts for the entire document when I set a default node font.

Omen

lmamakos · January 24, 2018, 5:14am

I fiddled around with your script; had to sort of hack on it a bit due to my non-standard installation on macOS and the directory the virtualenv was installed into. Still, here’s a somewhat more complex z-wave network that I have. Interestingly, the two minimotes show as disconnected since they’ve not been used in a bit…

Hard to figure out how you lay out this prettier in an automatic way. One thing that would be interesting is to colorize the edges in the graph back to the z-wave adapter.

I’m a bit surprised at how well connected my zwave devices are, and the fews that are not are interesting and make sense while I think about. For instance, the “Kitchen” dimmers (some of them, anyway) are in a congested 3 gang box, and there’s some large metal appliances directly in the path towards the zwave stick. So it makes sense they use some neighbors to get back…

Thanks for doing this!

magma1447 · January 24, 2018, 12:47pm

I can’t manage to get it to work. Probably something I don’t understand.

I have the iframe panel added in HASS, but it shows a 404: Not Found. The HTML file itself works, so I assume it’s because it can’t find a generated svg file.

I have triggered the automation for generating the file manually. I can’t see any errors but I can’t find any svg file either. I can’t manage to figure out where it’s supposed to be stored either.

Any hints?

OmenWild · January 24, 2018, 5:50pm

The SVG should output to the www directory at the same level as your configuration.yaml. Try running the Python script from the shell (~/bin/z-wave-graph.py) and see if you get any errors. If you’re using a venv make sure that’s activated before running the script.

Ah, one other thing you may need is to make the Python script executable (chmod 755 ~/bin/z-wave-graph.py). The shell_command assumes it can be executed directly.

OmenWild · January 24, 2018, 7:08pm

Good idea… And done, plus I made those lines thicker. I pushed the change to GitHub. Unfortunately colors that work for everyone might be a bit of a challenge. I’ve stuck with a lighter green for the node and darker for the edge(s), but am open to suggestions.

I went with the neato layout as it looked the best with my little network. Try the following patch (or apply the changes by hand). This switches the layout engine to dot and combines node paths where possible. The combined node paths look very strange on my small network, but may work well for yours. I would love to see the result.

--- a/bin/z-wave-graph.py
+++ b/bin/z-wave-graph.py
@@ -34,7 +34,7 @@ class ZWave(object):
 
         self.api = remote.API(base_url, api_password, use_ssl=use_ssl)
 
-        self.dot = Digraph(comment='Home Assistant Z-Wave Graph', format='svg', engine='neato')
+        self.dot = Digraph(comment='Home Assistant Z-Wave Graph', format='svg', engine='dot')
 
         self.dot.attr(overlap='false')
 
@@ -42,6 +42,7 @@ class ZWave(object):
         self.dot.graph_attr.update({
             'label': r'Z-Wave Node Connections\nLast updated: ' + datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
             'fontsize': '24',
+            'concentrate': 'true',
             # 'rankdir': 'BT',
         })

Tinkerer · January 24, 2018, 7:16pm

On many installs it’ll fail because:

graphiz isn’t installed in the venv
to_check = ['~/.config/configuration.yaml', '~/config/configuration.yaml'] won’t work, because the files are in ~/.homeassistant/configuration.yaml
python3 isn’t in /home/homeassistant/bin

It also doesn’t like not having an api_password defined (I do my auth in NGINX), same for ssl_key (also in NGINX) and base_url (not needed because I use NGINX).

All that grumping aside, this is really, really neat! Thank you!

Tinkerer · January 24, 2018, 7:28pm

And, because us recipients of such things are always going to complain about something else…

Is there any way to spread the nodes out slightly more? The bottom row is a bit crunched together, and the controller and the node to the left are almost touching (it looks like it’s possibly just the arrow ends that’re visible, maybe they’re overlapping).

magma1447 · January 24, 2018, 7:29pm

The problem is that I am running in a virtual environment. I realize that now. I made some adjustments but didn’t get far enough. My knowledge about python and the virtual environments are a bit too low.

I updated the python script’s first line to point to where I actually have python. I also updated the ~/config/ to point to ~/.homeassistant/ instead.

After that I ran it. No graphviz. Of course, it’s not in my virtualenv. Did a ‘pip install graphviz’, not sure that is enough. I ended up with:

(homeassistant) homeassistant@ha-01:~/bin$ python z-wave-graph.py
Error fetching states
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/utils.py", line 868, in check_header_validity
    if not pat.match(value):
TypeError: expected string or bytes-like object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/remote.py", line 203, in get_states
    URL_API_STATES)
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/remote.py", line 88, in __call__
    url, params=data, timeout=timeout, headers=self._headers)
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/sessions.py", line 494, in request
    prep = self.prepare_request(req)
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/sessions.py", line 437, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/models.py", line 306, in prepare
    self.prepare_headers(headers)
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/models.py", line 440, in prepare_headers
    check_header_validity(header)
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/utils.py", line 872, in check_header_validity
    "bytes, not %s" % (name, value, type(value)))
requests.exceptions.InvalidHeader: Value for header {X-HA-access: 2431} must be of type str or bytes, not <class 'int'>
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/graphviz/backend.py", line 124, in render
    subprocess.check_call(args, startupinfo=STARTUPINFO, stderr=stderr)
  File "/usr/lib/python3.5/subprocess.py", line 266, in check_call
    retcode = call(*popenargs, **kwargs)   
  File "/usr/lib/python3.5/subprocess.py", line 247, in call
    with Popen(*popenargs, **kwargs) as p: 
  File "/usr/lib/python3.5/subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1282, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'neato'
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "z-wave-graph.py", line 128, in <module>
    zwave.render()
  File "z-wave-graph.py", line 109, in render
    self.dot.render(filename=self.filename, directory=self.directory)
  File "/srv/homeassistant/lib/python3.5/site-packages/graphviz/files.py", line 176, in render
    rendered = backend.render(self._engine, self._format, filepath)
  File "/srv/homeassistant/lib/python3.5/site-packages/graphviz/backend.py", line 127, in render
    raise ExecutableNotFound(args)
graphviz.backend.ExecutableNotFound: failed to execute ['neato', '-Tsvg', '-O', '/home/homeassistant/.homeassistant/www/z-wave-graph'], make sure the Graphviz executables are on your systems' PATH

It’s not a big deal for me, I can wait a few weeks until someone else figures out what to do.

Anyway, I am the one who wrote one of the scripts you refer to. And my hope really was that someone would take the idea and make a more proper integration of it. I considered my own a proof of concept. The other project you linked I had missed. So I am grateful for your implementation, even more so the day I manage to run it.

Tinkerer · January 24, 2018, 7:34pm

Try this patch.

That finds the right python3 executable, connects to localhost, no API password or SSL, and finds the config in the default locations. Having applied it, I got the graph I pasted above.

OmenWild · January 24, 2018, 8:03pm

You’re missing the graphviz package. In Debian (or Debian like) sudo apt-get install graphviz. For RHEL/CentOS yum install graphviz graphviz-devel .

Tinkerer · January 24, 2018, 8:18pm

Also, for me, needed pip install graphviz inside the venv.

OmenWild · January 24, 2018, 8:20pm

Thanks for the comments. My installations are usually non-standard, so I’m not surprised there are issues for others. I’ve pushed an update to GitHub with changes based on your feedback. Specifically:

Also look for ~/.homeassistant/configuration.yaml
Use #! /usr/bin/env python3 to try to find python3
Default to localhost for base_url. Only set password and ssl if base_url exists and isn’t localhost.
Updated the README to mention that it also needs the Python package graphviz.

I also switched to the dot engine which might work better with large meshes and set Graphviz to combine node paths where possible. I’d love to see an updated graph with the dot engine.

And yes, getting the graphs to look nice for everyone might be a difficult job.

Tinkerer · January 24, 2018, 8:56pm

Hmm

Traceback (most recent call last):
  File "./z-wave-graph.py.2", line 141, in <module>
    zwave = ZWave(config)
  File "./z-wave-graph.py.2", line 27, in __init__
    if 'base_url' in self.haconf['http']:
TypeError: argument of type 'NoneType' is not iterable

That aside it works really well, and produces an uglier, but far more usable, graph:

If you take feature requests, can there be tags/markings for directly connected nodes (those connected by the green lines), and different ones for each further layer? Z-Wave has a limit of 4 hops, so for those on bigger meshes it’d be good to be able to see that. I appreciate that might be tricky to work out though.

Similarly it’d be handy to be able to visually separate out the battery powered devices from the rest if possible.

OmenWild · January 24, 2018, 9:05pm

You caught me in the middle of a bad push. The change was too trivial to require testing, so of course it didn’t work. If you re-pull, or just change line #27 to if 'base_url' in self.haconf['http']: it should work again.

I’ll look into doing the layers properly. Graphviz supports the notions of ranks and clusters, so it should be do-able. That’s something I’ll need as I re-task my SmartThings nodes.

As far as battery devices, sure, that’s easy. What do you think would look good and not too messy?

lmamakos · January 25, 2018, 4:30am

Here’s my mine again with the new script…

OmenWild · January 25, 2018, 7:23am

I wanted to see what my full network would look like, so I pulled everything from SmartThings and added it to HA.

I managed to get all the nodes arranged into levels and output a subgraph format, but it causes the dot engine to core dump. Other engines work, but they don’t understand clusters, so the graphs are really hard to read.

I did create a circular graph of all my nodes and it looks like many of them are connected to almost every other node, so I’m not sure how useful this type of graph is. Fun, but ultimately not too useful as is. If I get some time I’ll try some web2.0 interactive graphs and see if any of those are cleaner.

magma1447 · January 25, 2018, 1:29pm

Thanks. It was a lame thought by me. When I installed “graphviz” in my virtual environment I removed it from the system. I was thinking I installed graphviz itself in my venv, but I guess it was only the bindings. As mentioned, my knowledge around Python and its venv is quite low.

Anyway, I updated to the latest version of your py-script, and now it gets as far as it did with my own patches, after installing graphviz correcty. It ends with an error though. I have not yet tried to solve it myself, I might try that later today. It seems to be some typecasting issue if I understand it correctly.

(homeassistant) homeassistant@ha-01:~/bin$ python z-wave-graph.py
Error fetching states
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/utils.py", line 868, in check_header_validity
    if not pat.match(value):
TypeError: expected string or bytes-like object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/remote.py", line 203, in get_states
    URL_API_STATES)
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/remote.py", line 88, in __call__
    url, params=data, timeout=timeout, headers=self._headers)
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/sessions.py", line 494, in request
    prep = self.prepare_request(req)
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/sessions.py", line 437, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/models.py", line 306, in prepare
    self.prepare_headers(headers)
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/models.py", line 440, in prepare_headers
    check_header_validity(header)
  File "/srv/homeassistant/lib/python3.5/site-packages/requests/utils.py", line 872, in check_header_validity
    "bytes, not %s" % (name, value, type(value)))
requests.exceptions.InvalidHeader: Value for header {X-HA-access: 2431} must be of type str or bytes, not <class 'int'>

By the way, I now get z-wave-graph.svg, but the chart is missing. It only has the text “Z-Wave Node Connections\nLast updated: 2018-01-25 13:40:00”.

I just upgraded Home Assistant from 0.58.1 to 0.61.1 to make sure it wasn’t an issue with old versions.

EDIT: Solved that issue and submitted a patch.

Next issue, my base_url is wrong, but it’s because my configuration is wrong, because I don’t have open access from the internet. I only access it from LAN/VPN. That was an easy patch.
Now I am trying to figure out how to get it to ignore the fact that my certificate is self-signed. Hopefully that is the final issue.

Tinkerer · January 25, 2018, 5:19pm

That’s what happens when I’m impatient

Unfortunately:

Traceback (most recent call last):
  File "./z-wave-graph.py.3", line 143, in <module>
    zwave = ZWave(config)
  File "./z-wave-graph.py.3", line 27, in __init__
    if 'base_url' in self.haconf['http']:
TypeError: argument of type 'NoneType' is not iterable

My configuration.yaml is here if it helps you with testing.

Is it possible to use a different shape?

Oh, and I found you a bug:

On the right, you’ll see a green line that merges with a black one, and ceases to be green.

magma1447 · January 25, 2018, 5:49pm

FYI, I have managed to get things working now. The last thing I needed to do was to patch Home Assistant. As I understand it, it doesn’t support self-signed certificates properly. I tried to add my certificate to the Debian system, but it didn’t seem to respect that.

In my opinion, the code has access to the configuration, where I have defined the certificate, therefore the API should use that configuration. I don’t know how to make a proper patch, so I made an ugly hack instead.

/srv/homeassistant/lib/python3.5/site-packages/homeassistant/remote.py:88
url, params=data, timeout=timeout, headers=self._headers)
->
url, params=data, timeout=timeout, headers=self._headers, verify='/home/homeassistant/.homeassistant/certificate.pem')

magma1447 · January 25, 2018, 5:56pm

I would also like to explain what my own implementation does to make it easier to read (at least if you know what it is).

First of, z-wave only supports 4 jumps. Therefore it’s quite easy to find 4 colors depending on how far from the controller it is. As you have it now, the green is used for those with 1 jump to the controller. What I did was that I added a few more colors.

However, your dot implementation seems to share lines whenever useful, with the downside that a green line suddenly becomes black. This was mentioned in an above note by Tinkerer as well.

Since it’s a mesh network, every device/node has multiple connections, and multiple paths to contact the controller, or get contacted. What I tried to do was to have solid lines for all paths that was the closest, dashed lines for second closest path, and finally dotted lines for paths worse than that.

I then colored the nodes as well, according to how many jumps they had towards the controller. What I wanted to see was if any node had more than two jumps, and I wanted to see it at a glance.

If you implement something similar to these ideas, I can just throw away my own implementation, then it has zero advantage.