0.89 Breaking Change: Prevent partial custom component overlays

Configuration
41

But, if I understand how it works correctly, both of these solutions would reintroduce the types of breakages that were meant to be avoided, right? It also has the caveats that it’s treated as a different platform and can’t as easily be switched back to the stock platform as partial overlays could. It may be the cleanest option now, but it still seems like a downgrade in every regard. You have the same issues, plus more, with more work involved.

If all of that is correct, I think it would be good to bring it to the attention of the devs. If people are just going to bypass the restrictions, they might as well not be there and it should just be a warning if you use a partial overlay. Or perhaps there could be a simpler way to “opt-in”.

42

Not really. The recommended solution, to copy all files over to custom_components, still broke for me (I mentioned this in previous posts). The “Great Migration” started with 0.88. One of my two modified components was based on 0.88 code. When I used it in 0.89, it failed despite having all its ‘relations’ in the same directory. The solution was to revise the failed component by basing it on 0.89 code (because the component accesses the underlying system in a slightly different way). This is the same maintenance I had to do in versions prior to 0.88.

The alternatives discussed above don’t make my component more vulnerable, nor less vulnerable, to the kind of change that caused the failure I experienced. However, handling it as a derivative platform reduced complexity a great deal.

What will improve stability in the future is when the interfaces between components and the core become carved in stone. Then the ideas behind the “Great Migration” will bear fruit and custom components will survive unscathed through multiple upgrades of the core. It’s a bumpy ride until then.

It also has the caveats that it’s treated as a different platform and can’t as easily be switched back to the stock platform as partial overlays could.

Here’s how you would switch back to the stock platform:

When using with partial overlays:

  1. Delete the partial overlay(s) located in custom_components.
  2. Upon restart, Home Assistant defaults to using the stock version.

When using a derivative platform (like my_mqtt):

  1. Edit the configuration file and replace all instances of platform: my_mqtt with platform: mqtt.
  2. Upon restart, Home Assistant defaults to using the stock version.

Both are equally easy.

FWIW, I consider seeing platform: my_mqtt in the configuration file useful because it reminds me that the entity is using a modified version of an mqtt component.

43

This is actually closer to what I meant. You aren’t really shielded from the types of breaking changes that cause issues for partial overlays with either of these solutions since you’re still only modifying part of a component and using the built-in parts that could change at any time for the rest. Even with a stable API, the solutions mentioned here still bypass that. This effectively negates the entire purpose of the restrictions that were put in place on partial overlays. Thus, what’s the point in having the restrictions?

You also forgot to mention the step of needing to edit entries in the entity registry to make HA see devices from the new component as the same ones as from the old one. That would need to be repeated when switching back to the built in component. Imagine having to do this for every single light if you customized a light platform. I wouldn’t call that equally easy. One is definitively easier than the other. Even if it’s only marginally in some cases.

As for seeing my_mqtt in the configs, that was already possible before as well. This change did nothing to improve on that.

So, maybe we just disagree, but I really don’t see any advantages to these solutions over just allowing partial overlays. Even doing it the “right” way and copying the entire component comes with a whole other set of issues. In the end, I just don’t think the restrictions had the effect that they were intended to and came with some unintended consequences.

Of course, I’ll still be using one or both of these solutions myself for the time being, since it’s the closest we can get to partial overlays now.

44

That’s minimized when you copy them to custom_components, which is the recommended solution (see Paulus’ blog post).

You also forgot to mention the step of needing to edit entries in the entity registry to make HA see devices from the new component as the same ones as from the old one.

If they have unique_id set. I mentioned that in a previous post. I described how only one of the two entities had to be tweaked in the entity_registry (namely the one that had unique_id).

That would need to be repeated when switching back to the built in component.

Again, only for entities with unique_id. Anyway, I’m only entertaining this ‘switching back and forth’ concept because you brought it up but, honestly, it’s hardly a real-world requirement. The motivation for using a customized component is because the stock version lacks something. Nine versions later and MQTT climate and alarm_control_panel don’t have what I need so I use customized versions. That’s not to say no one has tried to submit a PR to revise MQTT climate (the PRs have died, for several reasons).

As for seeing my_mqtt in the configs, that was already possible before as well.

Yes, in previous version I had used mqtt_hvac.py and mqtt_alarm.py instead of the generic mqtt.py. So my config’s entries looked like platform: mqtt_hvac and platform: mqtt_alarm. However, the files resided in separate directories under custom_components because that’s how it was done pre-Great Migration.

This change did nothing to improve on that.

It has because now a platform’s components reside in one directory, the platform’s directory, which makes things simpler. Previously, a platform’s components were scattered over many directories.

So, maybe we just disagree, but I really don’t see any advantages to these solutions over just allowing partial overlays.

Maybe so but they’re history now. It’s early-days for the Great Migration and the concept of packages lies in the future (refer to the PR where it’s discussed).

45

I think you’re conflating two separate changes. I have no issues with the great migration, just the restrictions on partial overlays, which is not necessary for the great migration to be successful.

There are also cases for custom components other than adding features, e.g. bug fixes. I know because I did exactly that for the MQTT light platform. An OS update could fix the issue, but in the meantime I simply tweak a value and everything works. I would definitely like to switch back to the built-in component when it’s fixed in the OS. Updating IDs for each light would not be as trivial as one or two devices.

Anyway, I’m not really trying to argue with you over this. This is mostly just my way of logging my ideas and getting feedback from others to potentially bring up with the devs.

However, I also want to make sure that other people reading this are aware that, unless I’m completely mistaken about how the solutions mentioned here work (which I always could be, I’m not a core developer here), using them will bypass the entire reason behind blocking partial overlays. That isn’t something that should be recommended lightly without a serious warning first. Otherwise, it suggests that the restrictions really weren’t that necessary to begin with.

As I mentioned, the “right” solution of copying the entire component does prevent most of the incompatibility issues, but not all and it also introduces other issues as well.

In the end, I would like to hear what the devs have to say about the solutions mentioned here. Perhaps they’re better shielded than they appear, in which case they should be the recommended approach. Otherwise, we need to be careful about suggesting others use them as they should be reserved for advanced users who are aware of the risks involved.

46

Or if anyone could explain how these solutions are shielded from breakages where partial overlays were not (the primary reason for preventing them), that would suffice as well.

47

No one has claimed the proposed alternatives are ‘shielded from breakages’. Frankly, anyone who has carefully read this, now long-winded, thread has gathered that the alternatives definitely have disadvantages, not the least being that they behave like ‘partial overlays’.

I refrained from marking any of the alternatives as being the “solution” because the choice will be based on one’s own needs. The one I chose may not be best for others.

NOTE: At the request of @tboyce1, I’ve marked this post as a “Solution” (mostly to help locate it in this long thread). Please understand that this post is less of an ultimate solution and more of an overview of the currently available options to get your custom component working with 0.89.

Three ways to get your custom component to work in 0.89

I think the readers of this lengthy thread have gained enough insight to judge the merits of each approach. For those who have arrived late, here’s a summary. I’ll be using the MQTT platform in the examples (Paulus used Hue in his examples) only because I use it extensively and it happens to be one of the more comprehensive ones, supporting many different components (and even its own flavor of automation).

Replacement
This is the approach recommended by Paulus in his blog post (and PR).

If you have modified MQTT climate.py, copy it to custom_components/mqtt and all other unmodified MQTT components and related resources. Why? Because it only takes the presence of one MQTT component (modified or not) in custom_components/mqtt for Home Assistant to exclusively use this directory for finding all other MQTT components. In other words, it stops using the stock components/mqtt directory.

The reported advantage of this approach is it is self-contained and more insulated from underlying changes. It also opens the door to future improvements such as packages. The MQTT platform can become modular, support versioning, and be upgraded independently (within reason) of the core system.

The one caveat of a full replacement (in its current form in 0.89) is that it may encompass more than one might expect. MQTT also offers its own automation service. Therefore if you wish to use a service like mqtt.publish you must also copy its directory as well. Otherwise, Home Assistant will fail to find it because as mentioned earlier, its search-path for MQTT resources will be custom-components/mqtt.

Derivation
This approach uses a new platform derived from the desired platform. Place the modified MQTT components into a new directory (your choice of name) like custom_components/my_mqtt. In configuration.yaml, set the affected entities to use platform: my_mqtt.

The primary advantage of this approach is simplicity. If your custom component is used by many novices, this approach offers the easiest installation instructions. Because it is derived from the MQTT platform (and simply assumes a different name), it continues to share all resources with the stock version of MQTT. Home Assistant continues to use the stock components/mqtt directory for all MQTT-related resources.

This approach has several disadvantages. It behaves like the ‘partial overlays’ allowed in pre-0.88 versions so it lacks the ‘insulation’ offered by full replacement. It also creates a potential, future ‘conversion task’ in the event you ever wish to switch from using the custom component to the stock version (i.e. switch from platform: my_mqtt to platform: mqtt). If your entities have a unique_id then they are uniquely registered in the entity registry (if they don’t then this task doesn’t apply to you). If you change a registered entity’s platform, Home Assistant will re-register it as a different entity. For example, climate.thermostat will be re-registered as climate.thermostat_2 (effectively it’s a duplicate with the only difference being the platform it uses). Now you have a conversion task on hand and must edit the entity registry and clean up the duplicates (otherwise your Lovelace UI will be missing entities). There’s a way to simplify this task … but that’s for another post.

Redirection
This approach is sort of a hybrid of the two previous approaches. The advantage it has over derivation is the platform name remains unchanged so you’ll never have the potential conversion task. The advantage it has over full replacement is that, in effect, it doesn’t require you to make a true, full copy of the stock version (more about that in a moment).

Its disadvantage is the same as with derivation, it behaves like ‘partial overlays’ so it remains vulnerable to changes made in the underlying system.

Place the modified MQTT components into custom_components/mqtt. For the unmodified components you can create a placeholder or ‘redirection’ file containing a single line of code telling Home Assistant to refer back to the stock version of the component. For example, create
/custom_components/automation/mqtt.py containing a single line:

from homeassistant.components.automation.mqtt import *

This obviates the need to copy everything related to mqtt.automation. Other MQTT components can be handled in a similar fashion or just copy them over completely. You decide how much, or how little, is to be handled by placeholder/redirection files.

One other disadvantage of this approach is, quite simply, you have to create each redirection file. It may make sense in the case of mqtt.automation (avoids having to copy all of that) but maybe less so for more self-contained components like light.py, sensor.py, etc.

The installation instructions for this approach have the potential to be more complex than either ‘full replacement’ or ‘derivation’ (especially more complex than derivation). That might be factor if you intend to share your custom component with novices.

4 Likes
48

Great informative post!

1 Like
49

As mentioned in my previous post, the ‘derivative’ approach may involve more work should you ever wish to convert back to using the stock version. Specifically, if your entities use unique_id, they are uniquely identified in the entity registry. Simple changing an entity’s platform will cause it to be re-registered (duplicated) with a new entity_id. So an entity like climate.thermostat will be duplicated as climate.thermostat_2. Fortunately, it’s fairly easy to prevent this duplication from occurring (thereby sparing you a lot of messy editing).

The following instructions assume you are switching from a derived platform called my_mqtt to the stock version mqtt. It’s important to complete steps 2 and 3 before restarting Home Assistant (especially step 3).

  1. Stop Home Assistant.
  2. Edit configuration.yaml and replace instances of platform: my_mqtt with platform: mqtt
  3. Edit .storage/core.entity_registry and replace instances of "platform": "my_mqtt" with "platform": "mqtt"
  4. Restart Home Assistant.
50

I wasn’t suggesting that anyone claimed the solutions prevented the breaking changes. I was merely asking if anyone knew for sure because I don’t. At first sight, they appear to have the same issues as partial overlays, but I’m not 100% sure.

Very good summary though. I think we can leave it with that here and I’ll bring it up with the devs to see if they can shed some light on the situation.

Thanks!

51

I am a little confused to be honest.

I am trying to change my file structure so a custum component of Alarm Control Panel - concord232 will continue to work.

home-assistant/homeassistant/components/alarm_control_panel/concord232.py

If I understand it correctly I will have to rename it to alarm_control_panel.py and place it under a concord232.py folder:

custom_components/concord232/alarm_control_panel.py

plus I have to place all the other files from the HA alarm_control_panel folder in this custom_components/concord232 folder.

But because I am also using a concord232 Binary Sensor, I am getting a missing component error, so obviously I have to ad the Binary Sensor to this custom folder as well. So the following:

home-assistant/homeassistant/components/binary_sensor/concord232.py

becomes a file in the custom folder like this:

custom_components/concord232/binary_sensor.py

But now I am conflicted. If I also have to copy all the binary_sensor files over, what do I do if there is a conflict between the files from the alarm_control_panel folder? Specifically the init.py file?

I am sure I have misunderstood something. The whole process seems backwards to me.

52

You have a good grasp of what’s involved just that you have to take it one step further. The recommended ‘Full Replacement’ approach means you need to copy everything from components/concord232 to custom_components/concord232 including _init_.py.

Keep in mind that once Home Assistant finds some portion of concord232 in custom_components/concord232 it stops using components/concord232 so there’s little chance of a ‘conflict between the files’. This is the whole idea behind eliminating ‘partial overlays’.

53

123,
But are you supposed to combine the two init files?
In my case two init files that were previously in two seperate directories now ends up in one due to the inverted folder structure.
So concord232 has a binary_sensor component AND an alarm_control_panel component.

I understand the reasoning for copying all component dependencies, but why the inverted structure???
Why didn’t they just keep the Folder structure identical?

54

You’re actually doing more than the bare minimum by moving them to a new component. The current recommended approach (as per the official documentation) would mean you copy everything as follows:

homeassistant/components/alarm_control_panel/* -> custom_components/alarm_control_panel/*
homeassistant/components/binary_sensor/* -> custom_components/binary_sensor/*

There’s no need to merge the directories. That’s because that specific component hasn’t been moved into it’s own directory yet. That is part of the “The Great Migration” but it isn’t complete, as far as I can tell, so not all components are set up like that. So, by merging them into custom_components/concord232/*, you’re actually going one step further and doing something that possibly needs to happen to the official component at some point as well.

That being said, there’s a decent chance you don’t need anything at all in your __init__.py file. I would initially try just leaving it empty. I think as long as your files are named concord232/alarm_control_panel.py and concord232/binary_sensor.py, HA knows that they belong to those domains.

See, for example, the firetv component. It has a media_player.py, but there is nothing in __init__.py. The service.yaml also only adds the ADB service, so it somehow knows that all the other media_player services apply as well. Maybe I’m missing something, but it appears to me that it just goes by the filename.

55

Edit: I just tested this by copying the random sensor and binary sensor into a new random component and adding an empty __init__.py and it seems to work fine. HA knows that

binary_sensor:
  - platform: random
sensor:
  - platform: random

Should use my new custom versions instead of the built-in ones from homeassistant/binary_sensor/random.py and homeassistant/sensor/random.py.

56

Ok, reading https://developers.home-assistant.io/blog/2019/02/19/the-great-migration.html had me believe that from 0.87 it was a requirement to use the new structure.
I can now see that it is probably component dependent since some component already exists in Home Assistant in the new format:

home-assistant/homeassistant/components/hue/

while concord232 has not yet been migrated, so the following folder doesn’t exist:

home-assistant/homeassistant/components/concord232/

If I just create:

custom_components/concord232/__init__.py (Blank)
custom_components/concord232/binary_sensor.py
custom_components/concord232/alarm_control_panel.py

I get an error:

Error loading custom_components.concord232.binary_sensor. Make sure all dependencies are installed

Probably because the components haven’t been rewritten to contain the info that’s in their respective init.py files-

So in my case I need to stick to:

 custom_components/binary_sensor/concord232.py
 custom_components/alarm_control_panel/concord232.py

until the component is migrated.

Is that a correct assumption?

57

You’re probably right because I missed the part where concord232 has not yet been re-arranged into the new directory structure. It’s worth a try … and Home Assistant’s system log will certainly tell you if it’s working or not.

58

So I tried to do it this way:

 custom_components/binary_sensor/concord232.py
 custom_components/alarm_control_panel/concord232.py

But that way no longer works. It doesn’t throw an error, but it’s not using my component. So next step I guess is to somehow try and include the init content in the binary_sensor.py and the alarm_control_panel.py files and go with the new folder setup.

Any better suggestions?

59

If you do it that way, you do need to also copy

custom_components/binary_sensor/__init__.py
custom_components/alarm_control_panel/__init__.py

and they should be exact copies of the ones from HA.

60

Is there anything in the logs other than that? Or can you post your custom version somewhere? I don’t really see anything in the built-in component that would need a special __init__.py, but maybe the custom version does?