0.89 Breaking Change: Prevent partial custom component overlays

Configuration
1

I want to get ahead of an upcoming Breaking Change in 0.89, namely this PR:

I’ve read the PR and hope someone can tell me if I’ve understood it correctly.

I currently use modified versions of MQTT HVAC (climate.py) and MQTT Alarm Control Panel (alarm_control_panel.py). In version 0.88.X, they are both located in this directory and work correctly:

<config>/custom_components/mqtt/

Am I correct in saying that for version 0.89, I will also need to place a copy of the MQTT component’s __init__.py file in that directory as well?

Is that all there is to it or do I have to copy all MQTT platform files (everything listed here) into <config>/custom_components/mqtt/? In other words, my custom_components/mqtt directory will be a full duplicate (of the release code) and the only difference in it will be my two modified versions of climate.py and alarm-control_panel.py.

3 Likes
2

Still waiting to get my hands on 0.89 to determine how this Breaking Change affects me.

I think all I need to do is copy:
/components/mqtt/__init__.py
to:
/custom_components/mqtt/__init__.py.

I believe that’s sufficient to make 0.89 use my customized climate.py and alarm_control_panel.py.

However, I also use the stock versions of light.py and switch.py. If I understand the following statement correctly, I will also have to copy them to /custom_components/mqtt otherwise 0.89 will fail to find them.

Example: if I look up the hue component, and it is provided by a custom component, then all platform lookups will also be looked up in the custom component dir.

1 Like
3

I’d mosey on over to discord and ask there… Good proactive actions though so kudos…

4

Thanks but I already spend waaay too much time here and the discord channel sounds like an even deeper time hole.

I’ll muddle through this somehow …

5

I just asked a very similar question to baloob in the new release thread. Maybe hang out there and see what the answer is? At least there you may have his attention. :wink:

6

Saw the answer. Didn’t help me. Would that it were so simple! :slight_smile:

  • Just pulled docker container 0.89.0.
  • custom_components/mqtt contains MQTT’s __init__.py plus my customized climate.py and alarm_control_panel.py (which worked with 0.88.1).
  • 0.89 fails to load my customized versions. The warning messages suggest it’s still using the stock versions of climate.py and alarm_control_panel.py

I stopped the docker container, purged all files in custom_component/mqtt/__pycache__, restarted the container and … no joy; it still chooses to use the stock versions.

FWIW, I know it’s using the stock versions because the warning messages complain about unexpected keys (which only the customized versions knows how to handle) and, of course, the behavior of the two platforms lack the enhancements I put into the customized ones.

Meh, I’ll try again tomorrow (… like copy the entire MQTT directory structure into custom_components)

7

He hasn’t answered my follow on question yet. Maybe there will be an answer in there.

8

Some progress to report but it’s pretty clear people will experience headaches. I know because I’m living it right now.

I copied the entire stock MQTT component directory (and its sub-directory for light) to custom_components/mqtt. For good measure, I also purged the contents of the __pycache__ sub-directory.

Now 0.89 finds my customized alarm_control_panel.py (and loads it successfully) as well as the non-customized components I use such as light, switch, sensor, binary_sensor, and cover. Progress!

However, there’s still no joy in Hassville because 0.89 stumbles with my customized climate.py. It rejects it because it is unable to resolve STATE_AUTO.

Well now, that’s a head-scratcher because STATE_AUTO is not something I added to my customized climate.py. It also exists in the 0.89 version of climate.py.

Here’s the error message:

HASS[1457]: 2019-03-07 09:05:52 ERROR (MainThread) [homeassistant.loader] Error loading custom_components.mqtt.climate. Make sure all dependencies are installed
HASS[1457]: Traceback (most recent call last):
HASS[1457]:   File "/usr/src/app/homeassistant/loader.py", line 166, in _load_file
HASS[1457]:     module = importlib.import_module(path)
HASS[1457]:   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
HASS[1457]:     return _bootstrap._gcd_import(name[level:], package, level)
HASS[1457]:   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
HASS[1457]:   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
HASS[1457]:   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
HASS[1457]:   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
HASS[1457]:   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
HASS[1457]:   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
HASS[1457]:   File "/config/custom_components/mqtt/climate.py", line 16, in <module>
HASS[1457]:     from homeassistant.components.climate import (
HASS[1457]: ImportError: cannot import name 'STATE_AUTO' from 'homeassistant.components.climate' (/usr/src/app/homeassistant/components/climate/__init__.py)
HASS[1457]: 2019-03-07 09:05:52 ERROR (MainThread) [homeassistant.loader] Unable to find platform mqtt. Search path was limited to path of component: custom_components
HASS[1457]: 2019-03-07 09:05:52 INFO (MainThread) [homeassistant.setup] Setting up climate
HASS[1457]: 2019-03-07 09:05:52 INFO (MainThread) [homeassistant.setup] Setup of domain climate took 0.0 seconds.

STATE_AUTO is defined outside the realm of the MQTT component. It’s defined in the Climate component, in homeassistant/components/climate/const.py.

Do I also need to copy that file into custom_components/climate/?

Based on what I’m experiencing, I’d say the breezy, easy-peasy documentation for this Breaking Change assumes a ‘best-case’ scenario. :wink:

9

I found the source of the problem and the solution does not involve copying homeassistant/components/climate/const.py.

The 0.89 version of MQTT’s climate.py contains a subtle difference from the 0.88.1 version of the same file.

Here’s how 0.88.1 gets STATE_AUTO:

from homeassistant.components.climate import (
    ATTR_OPERATION_MODE, DEFAULT_MAX_TEMP, DEFAULT_MIN_TEMP,
    PLATFORM_SCHEMA as CLIMATE_PLATFORM_SCHEMA, STATE_AUTO, STATE_COOL,

Here’s how 0.89 gets STATE_AUTO:

from homeassistant.components.climate.const import (
    ATTR_OPERATION_MODE, DEFAULT_MAX_TEMP, DEFAULT_MIN_TEMP,
    STATE_AUTO, STATE_COOL,

0.88: homeassistant.components.climate
0.89: homeassistant.components.climate.const

Just enough to break my customized climate.py which is based 0.88 source code.

I updated my custom version and now 0.89 loads it successfully (and, most importantly, it works).

So the solution is … troubling. Maybe I misunderstood the reasoning behind this architectural change (no more partial custom component overlays) but I thought one factor was to ensure the customized component’s integrity in the face of underlying changes.

Today, if you want, you can override the Hue light platform, but not the other parts of the Hue integration. If a future update evolves the Hue component, removing or changing internal methods that the custom platform relied upon, the custom platform will start failing (like this report).

To avoid this, we’re going to no longer allow custom components to be partial overlays

Yet this was an example of a custom component requiring additional modifications in order to adapt to underlying changes. It failed it the same way when I had attempted to use a customized version of climate.py based on 0.80 in version 0.86. There were so many changes between 0.86 and 0.80 that it was understandable that my customized component would fail.

To be clear, I’m not finger-pointing but simply wish to understand how this architectural change didn’t insulate my customized climate.py from an underlying modification.

10

Your custom component is outdated. As you can see in the 0.89.0 release, the STATE_AUTO constant is imported from homeassistant.components.climate.const: https://github.com/home-assistant/home-assistant/blob/0.89.0/homeassistant/components/mqtt/climate.py#L14-L16

11

Yes, thank you, I discovered that was the case. The same was true, several weeks ago, when I attempted to use a customized MQTT climate.py based on 0.80 code in version 0.86. Naturally, it failed. There were so many changes in the MQTT component from 0.80 to 0.86 that it was understandable that my climate.py would fail.

I believed this latest architectural change (no more partial overlays) was a strategy to insulate my ‘partial overlay’ from underlying changes. However, I find I still have to modify it to adapt to the new way climate.py resolves constants (between 0.88 and 0.89). Effectively, I’m still having to adapt the code for new versions (as in the past) plus there’s the extra task of copying the entire family of MQTT components.

Same juice but involving a lot more squeeze! :wink:

I realize this change is ostensibly to provides benefits but, honestly, my first contact with it isn’t revealing them to me. Perhaps I’ve completely misunderstood its application or simply found a use-case where it offers no self-evident advantages.

Anyway, all’s well that ends well and I’ll adapt to the new way of doing things. :slight_smile:

12

i also have a question about new custom component, i am still using the harmony as a custom component

so i have a new file structure : custom_components\harmony

in here;

  • remote.py
  • init .py

but i receve 2 warnings?

2019-03-07 15:17:17 WARNING (MainThread) [homeassistant.loader] You are using a custom component for harmony which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant.
2019-03-07 15:17:17 WARNING (MainThread) [homeassistant.loader] You are using a custom component for harmony.remote which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant.

why do i have 2 warnings instead of 1 ?

13

If the component is sourced from custom-components, 0.89 will issue a warning. One warning for each component.

Here’s what I mean:
I have all MQTT components in custom_components/mqtt.
Two of them are customized: climate.py and alarm_control_panel.py.
All others are not customized.
I also use light, switch, cover, sensor, binary_sensor, lock.

When Home Assistant starts it issues warnings that I’m using custom components for climate, alarm_control_panel, and for light, switch, cover, sensor, binary_sensor and lock. So I get at least 8 warnings.

Here’s an abbreviated version of the log:

2019-03-07 10:01:39 WARNING (MainThread) [homeassistant.loader] You are using a custom component for mqtt.alarm_control_panel which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant.
2019-03-07 10:01:40 WARNING (MainThread) [homeassistant.loader] You are using a custom component for mqtt.lock which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant.
2019-03-07 10:01:40 WARNING (MainThread) [homeassistant.loader] You are using a custom component for mqtt.binary_sensor which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant.
2019-03-07 10:01:40 WARNING (MainThread) [homeassistant.loader] You are using a custom component for mqtt.climate which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant.
2019-03-07 10:01:40 WARNING (MainThread) [homeassistant.loader] You are using a custom component for mqtt.cover which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant.
2019-03-07 10:01:40 WARNING (MainThread) [homeassistant.loader] You are using a custom component for mqtt.sensor which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant.
2019-03-07 10:01:40 WARNING (MainThread) [homeassistant.loader] You are using a custom component for mqtt.switch which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant.
2019-03-07 10:01:40 WARNING (MainThread) [homeassistant.loader] You are using a custom component for mqtt.light which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you do experience issues with Home Assistant
14

Sure, you are not 100% insulated from all breaking changes, but it’s a lot better. Changes as we did to media player here, are one time clean ups which are now possible now that platforms are no longer in that folder. We’re breaking up a 1000 line file into smaller, more manageable ones.

15

It appears the excitement isn’t over. I just discovered another surprise.

I created a simple automation whose trigger uses platform: mqtt.

- id: 'light_toggle'
  alias: Light Toggle
  trigger:
    - platform: mqtt
      topic: "test/toggle"
  action:
    - service: light.toggle
      entity_id: light.kitchen

It doesn’t work because 0.89 reports it cannot find the mqtt.automation platform.

HASS[1457]: 2019-03-07 23:35:23 ERROR (MainThread) [homeassistant.loader] Unable to find platform mqtt. Search path was limited to path of component: custom_components
HASS[1457]: 2019-03-07 23:35:23 ERROR (MainThread) [homeassistant.setup] Unable to prepare setup for platform mqtt.automation: Platform not found.

It lives here /components/automation/mqtt.py but 0.89 can’t find it because now it assumes everything related to MQTT is in custom_components.

This is … unexpected. Will I have to copy that into custom_components as well?

16

oh crap!!! This is exactly what @pnbruckner found with his sun.py as well and he fixed it… there was quite a discussion here about this. I didn’t have any issue and didn’t even do the rename as I wasn’t getting a warning… but @klogg and I think Phil had the same issue with an automation…

17

Yes. I simply created custom_components/sun/automation.py and put this inside:

from homeassistant.components.automation.sun import *
1 Like
18

Thanks, I’ll try that.

IMHO, this whole process of customizing one or two MQTT components has gone from straightforward to Byzantine.

19

BTW, I did the same thing with other files I didn’t need to customize with my custom amcrest component. Just create a file in the right place with the right name and put an appropriate from xxx import * statement in it for each of the files you don’t need to customize. It’s a trade-off. Either you copy the file, and you’re “insulated” from changes (unless you’re not, like you found), or you do an import of the standard module and you get updates as they come and you only deal with incompatibilities if and when they come.

1 Like
20

Thanks again! Tested and confirmed it works.

Created:
/custom_components/automation/mqtt.py

containing:
from homeassistant.components.automation.mqtt import *

In lieu of creating duplicates of standard modules in custom_components, this technique redirects Home Assistant to use the standard modules. In effect, this re-establishes the concept of ‘partial overlays’!

There’s the irony. It failed to deliver on one of its chief benefits: insulating a custom component from underlying changes. I understand it can’t be 100% insulation but, damn, this failed to deliver on its promise immediately. I believe the same thing happened to you with the armcrest component.

I was going to make a joke that this added complexity is just a roundabout way of nudging developers to submit their components into the main codebase (and not as custom components). However, in light of your past experiences with submitting your sun component, maybe it’s not all that funny.
:wink:

1 Like