New Vantage InFusion integration using AsyncIO

loopj · July 12, 2023, 11:56pm

Hey folks, I’d like to share something I’ve been working on over the past few months, a new integration for Vantage InFusion lighting/automation controllers.

https://github.com/loopj/home-assistant-vantage

If your house has a Vantage InFusion lighting and automation controller, you can use this integration to control your lights and other devices from Home Assistant.

Previously, I’d been using the integration created by @gjbadros (thanks Greg!), but over the past few months I’ve been working on a new asyncio python library called aiovantage for interacting with Vantage controllers. I built a new Home Assistant integration that leverages aiovantage, and I’ve been using it in my own home for a few months now.

Here are the key differences between the integrations:

Supports a few additional Vantage objects types, such as thermostats, wind sensors, etc.
Implements full config flow based setup, handling setup failure and authentication errors, and automatically discovering the Vantage controller on the network.
The integration follows almost all of the best practices from the Integration Quality Scale.
As well as using asyncio for requests, aiovantage uses a completely different approach to fetching objects from the Vantage controller, leveraging the same API endpoints used by Design Center rather than parsing Design Center backup files.
Responses from this API are parsed using xsdata into dataclass models, rather than cherry-picking nodes from the XML responses. This approach should in theory be more robust, and makes it much easier to add support for new object types.
Uses a straightforward approach to entity/device naming, directly using the names from Design Center rather than building a naming hierarchy.

I’d love to get more feedback from other Vantage users. Adding support for new object types should be relatively easy, so please let me know if something you expect to see isn’t showing up.

Feel free to share any feedback here, on GitHub, or on Discord.

Thanks to @ptr727 for early feedback, to @gjbadros for his integration, and to the folks in the Home Assistant Discord for answering my questions!

gjbadros · July 13, 2023, 1:02am

Cool – please note that this PR Add support for secure sockets by loopj · Pull Request #14 · gjbadros/pyvantage · GitHub I merged from you (which was great for its secure sockets support) has a bad socket-reading bug in it that you should be sure your new code also corrects. You cannot assume that you can read an entire response from the vantage controller with a single socket read up until a \n – for lots of reasons, one socket read might give you only part of the response and you have to keep reading until you get a \n being careful to not read past the newline and save everything left over for the next read. See my subsequent commits that fixed this. Telnet, of course, provides that functionality built in, so when we dropped to socket-level interactions, we needed to manage it directly.

loopj · July 13, 2023, 3:21am

Thanks Greg. aiovantage uses an asyncio.StreamReader under the hood which is a higher level wrapper over sockets which handles buffering, so I think we should be fine here.

jpcastel · July 26, 2023, 8:10am

Hi James, thanks so much for all the hard work! I’ve deployed this integration via hacs and I can’t seem to connect to the controller. I have an EMC1000 E Master Controller running firmware 2.3.65. It’s quite an old setup in my parents home and I’m trying to gain access. Are there any minimum specs that you know of? I can ping the controller via telnet as well as hit the http landing page, but sadly don’t have access to design center or the website documentation library as it seems to all be locked down via a vendor login.

loopj · July 26, 2023, 2:35pm

Hey @jpcastel, it looks like the EMC1000 is the controller for Vantage’s previous generation platform called QLink. This integration was built for Vantage Infusion controllers, and unfortunately the two platforms are not compatible.

jpcastel · July 30, 2023, 8:29pm

Hey @loopj thanks so much for the info. This has sent me down a rabbit hole. Turns out the controller is actually a Vantage Controls IC-36 Infusion Controller. Strangely the controller is presenting itself as an EMC1000 on its http local ip. Looks like I need to track down a copy of design center to take a look at what is going on and prob update the firmware. I did get some interesting errors in the HA logs:

TypeError: Master.__init__() missing 4 required positional arguments: 'mtime', 'display_name', 'module_count', and 'serial_number'
2023-07-30 01:41:18.359 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 85, in security_filter_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 80, in ban_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 148, in handle
    result = await handler(request, **request.match_info)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 181, in post
    return await super().post(request, flow_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 72, in wrapper
    result = await method(view, request, data, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 110, in post
    result = await self._flow_mgr.async_configure(flow_id, data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 297, in async_configure
    result = await self._async_handle_step(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 393, in _async_handle_step
    result: FlowResult = await getattr(flow, method)(user_input)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/vantage/config_flow.py", line 70, in async_step_user
    return await self.async_finish()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/vantage/config_flow.py", line 155, in async_finish
    serial_number = await get_serial_from_controller(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiovantage/discovery.py", line 112, in get_serial_from_controller
    async with Vantage(host, username, password, use_ssl=ssl) as vantage:
  File "/usr/local/lib/python3.11/site-packages/aiovantage/__init__.py", line 113, in __aexit__
    raise exc_val
  File "/usr/local/lib/python3.11/site-packages/aiovantage/discovery.py", line 113, in get_serial_from_controller
    master = await vantage.masters.afirst()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiovantage/query.py", line 151, in afirst
    await self.__populate()
  File "/usr/local/lib/python3.11/site-packages/aiovantage/controllers/base.py", line 367, in _lazy_initialize
    await self.initialize()
  File "/usr/local/lib/python3.11/site-packages/aiovantage/controllers/base.py", line 133, in initialize
    async for obj in get_objects(self.config_client, types=self.vantage_types):
  File "/usr/local/lib/python3.11/site-packages/aiovantage/config_client/requests.py", line 50, in get_objects
    response = await client.request(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiovantage/config_client/__init__.py", line 164, in request
    method = self._parser.parse(method_el, method_cls)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/xsdata/formats/dataclass/parsers/bases.py", line 54, in parse
    result = handler.parse(source)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/xsdata/formats/dataclass/parsers/handlers/native.py", line 59, in parse
    return self.process_context(ctx)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/xsdata/formats/dataclass/parsers/handlers/native.py", line 76, in process_context
    self.parser.end(
  File "/usr/local/lib/python3.11/site-packages/xsdata/formats/dataclass/parsers/bases.py", line 152, in end
    return item.bind(qname, text, tail, objects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/xsdata/formats/dataclass/parsers/nodes/element.py", line 88, in bind
    obj = self.config.class_factory(self.meta.clazz, params)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/xsdata/formats/dataclass/parsers/config.py", line 10, in default_class_factory
    return cls(**params)  # type: ignore
           ^^^^^^^^^^^^^

Any Ideas?

loopj · July 30, 2023, 9:06pm

Hey @jpcastel, you’ll see that error when there are some fields missing from the expected XML response, I’m guessing because you have an older firmware version than the ones I have tested. If you have Discord, I’d like to ask a few debugging questions to see if we can get it working here’s the link.

cazwax · September 11, 2023, 3:58pm

hey there @loopj, just fired this up and I’m pleasantly surprised at how well it found my system and decyphered the configuration.

I have my own code, written years ago, which integrates my Vantage with my Elk security system and a fleet of Axis cameras. It’s a set of node.js ( 0.6?) runtimes which uses its own messaging bus to coordinate behavior. A set of rules provides ‘automation’ of a sort. It’s been useful, but picky and difficult to maintain and debug. I mean, the lights come on properly and all, but the race conditions between, say security lights coming on and camera’s edge detection getting confused by that ‘motion’ and doing countdown timers on involved motion detectors … shessh.

One thing I don’t understand is the power consumption sensors. On my installation they appear to be disabled. Is this a known condition - if not I’ll be glad to file a bug report.

Thanks for this nice work.

_{footer - running in a proxmox8 HAOS VM on a 2018 64Gig Mac Mini, current HAOS, using zfs for storage ( probably a mistake ). moving this to timescale/proxmox as a backend storage.}

loopj · September 11, 2023, 4:18pm

Thanks for the kind words Casey!

The power consumption sensors are actually disabled by default intentionally (as mentioned here). You can enable them in the Home Assistant UI and they should start working immediately.

I took this approach since it is recommended in the Integration Quality Scale to disable or hide less popular entities to help users have a clean automatic dashboard.

I’m open to changing this approach in future, potentially hiding the entities by default instead of disabling them.

Let me know if you have any questions or feedback!

loopj · September 27, 2023, 4:34am

Some updates for people following along

Added support for Vantage thermostats, temperature sensors, light sensors, and wind sensors
Added support for (very) old Vantage InFusion controllers and firmware versions (2.x)
Devices and entities update automatically when the Vantage system is programmed
We’re now available as a default repository in HACS

cazwax · October 5, 2023, 8:57pm

Hey Loopj, I’ve installed your latest build and so far everything seems find.
however,
even with the Current & Power Sensors enabled I get no measurements.
I do get a Temperature Sensor measurement (?) of 32F, which is odd because my mechanical room / server room is about 82F

I’ll turn on logging and see if anything much comes up. I imagine your code is polling now and again?

(dramatic pause)

2023-10-05 13:53:31.401 DEBUG (MainThread) [aiovantage.command_client.commands] Sending command: INVOKE 865 Temperature.GetValue

2023-10-05 13:53:31.406 DEBUG (MainThread) [aiovantage.command_client.commands] Received response: R:INVOKE 865 -0.999 Temperature.GetValue

2023-10-05 13:53:31.406 DEBUG (MainThread) [aiovantage.command_client.commands] Sending command: INVOKE 866 PowerSensor.GetPower

2023-10-05 13:53:31.412 DEBUG (MainThread) [aiovantage.command_client.commands] Received response: R:INVOKE 866 -0.999 PowerSensor.GetPower

invoking the jobs/VIDs produces responses alternating between 0 and -0.999

now it’s quite possible I am confused - were I getting a proper response would I be seeing the lighting lode, or the serial network load?

edit:
OS kernel-20100930.dwn
Root File System: rootfs-20130610.dwn
App Firmware: app-3.2.2.1.dwn
1 controller,
station bus voltage 36
station bus current 5.6

kevinjoshai · October 25, 2023, 6:52pm

Hi @loopj Nice integration! Do you have any plans to get it merged into Core?

loopj · October 31, 2023, 8:13pm

Hey Casey thanks for the feedback.

32F is 0 degrees C, so that would line up with your other sensors coming in with “zero” values.

You are correct in thinking there is polling happening for these sensors. Almost all entities from vantage support push, except a couple of sensor types including dimmer module sensors.

I’m not quite sure what would cause you to see null values back from these sensors, the values you are seeing will be the values I’m getting from the infusion controller. Can I ask what type of inFusion controller you have, and what software version it is running?

loopj · October 31, 2023, 8:25pm

Hey Kevin, thanks for the kind words!

I would certainly consider doing a PR into core but I feel like a custom integration is a great way to “beta test” things. Once I get an idea that there is enough interest and/or people successfully using the integration that could be a good time to make that happen.

loopj · October 31, 2023, 8:29pm

Oops, just saw you mentioned Vantage firmware 3.2.2.1 at the bottom of your comment. I don’t have access to a controller with a 3.x firmware so it is tricky for me test this. If you happen to be available to help debug in discord we can try a few things.

cazwax · November 7, 2023, 6:03pm

hi again Loopj,
I have been visiting an old band-mate in LA for a week or so, before he begins shooting again. we added some controllers to his garage and gate, neat.

we can figure out some time to meet up in Discord ( etc ) later this week or next. I will be out in the forest doing some community fuel reduction ( I am in california ) for most of that time tho, so even next week is best.

I pulled the voltage and version stats out of my DC connection.

I haven’t upgraded anything due to our very friendly salesperson having been dropped in the purge a few years ago. So I don’t want to lose my little Win7VM / DC <-> Controller setup.

psycomp · April 3, 2024, 11:52pm

@loopj Just wanted to give you a heads up that home-assistant-vantage broke as soon as I upgraded to 2024.4.0. I saved this error log before I reverted:

Log details (ERROR)

Logger: homeassistant.setup Source: setup.py:316 First occurred: 5:49:39 PM (1 occurrences) Last logged: 5:49:39 PM
Setup failed for custom integration 'vantage': Unable to import component: cannot import name 'Entity' from 'homeassistant.components.group' (/usr/src/homeassistant/homeassistant/components/group/__init__.py)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/loader.py", line 979, in async_get_component
    comp = await self.hass.async_add_import_executor_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1037, in _get_component
    ComponentProtocol, importlib.import_module(self.pkg_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/config/custom_components/vantage/__init__.py", line 30, in <module>
    from .entity import async_cleanup_entities
  File "/config/custom_components/vantage/entity.py", line 16, in <module>
    from homeassistant.components.group import Entity
ImportError: cannot import name 'Entity' from 'homeassistant.components.group' (/usr/src/homeassistant/homeassistant/components/group/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 316, in _async_setup_component
    component = await integration.async_get_component()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 997, in async_get_component
    self._component_future.result()
  File "/usr/src/homeassistant/homeassistant/loader.py", line 989, in async_get_component
    comp = self._get_component()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1037, in _get_component
    ComponentProtocol, importlib.import_module(self.pkg_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/config/custom_components/vantage/__init__.py", line 30, in <module>
    from .entity import async_cleanup_entities
  File "/config/custom_components/vantage/entity.py", line 16, in <module>
    from homeassistant.components.group import Entity
ImportError: cannot import name 'Entity' from 'homeassistant.components.group' (/usr/src/homeassistant/homeassistant/components/group/__init__.py)

loopj · April 10, 2024, 5:14am

Thanks @psycomp! This has been fixed in the latest release, which is now on HACS.