The Great Migration

Development
1

This blog post merits more attention. It affects anyone who has created custom components or uses them.

If I’ve understood it correctly, the following change is very significant:

… if you want to override a built-in component/platform with a custom version, you will need to copy over all component and platform files, not just the one you want to load.

All component and platform files? So if i want to use a modified version of MQTT climate.py, I have to copy over (to my custom component folder) everything related to MQTT?

4 Likes
2

That does sound pretty ominous doesn’t it?

Especially given the fact that the change was posted almost as an aside in the release blog post like it was not a big deal.

2 Likes
3

FYI

Paulus’ reply is here. In a nutshell, not yet implemented in 0.88 (but it’s on the roadmap).

4

It seems like in this thread, and the 88 release post, you are insinuating that this change is not properly being announced.

Not only are we printing warnings, which can be seen from the system log in the UI, we also made a blog post. This has been posted on:

And not to forget, these changes have also been discussed since December in the architecture repo.

If you think that more awareness should be created, please let us know how you would want to help with this.

1 Like
5

The requirement to no longer allow partial overlays of integrations is to avoid a Home Assistant upgrade breaking your custom component. With the new approach, as long as the custom component does not rely on an internal API, it will not break with a Home Assistant upgrade. This is a better user and developer experience.

1 Like
6

in the system log file…that nobody ever looks at unless something is broken right now… How is that considered the best way to inform users that something will break in the future?

I’m sure every one of those ways that you have discussed the change is perfectly reasonable for your core developers. What about the non-core developers who make one or two components to solve an issue that they have and then decide to post it so that others who have the same need can benefit from their work and never revisit the component after that? Those people can’t be expected to follow all of those listed discussions continuously on the off chance that their component might break later on. Not to mention the users of those components who installed it and it worked perfectly fine until one day it didn’t (and didn’t happen to look at a link that was specifically aimed at “developers”). And now both of those groups need to wade through the breaking change lists, and in this case possibly old breaking change lists, to try to intuit from the (tbh, at times kind of poor) documentation to try to figure out what went wrong.

And I’ve tried to jump in the Discord chat to get help once or twice and IMHO that is the worst way possible to try to get help on a problem. Maybe it’s just me, but I don’t think so, but I find trying to wade through the stream of consciousness format of that channel is way too difficult to follow when you are trying to wrap your head around a complex/confusing topic.

Since I’m not a “developer” and I’m not involved in all of those discussions taking place in those locations the only thing I can reasonably do is try to point out to the people “in the know” where there is confusion. Most of the time that effort is met with disdain from others (kind of like this and the other thread) who then “circle the wagons” and act like this is some sort of “sacred cow” that can’t be criticized or, even worse, the problem is completely ignored. And not to mention the extraordinary amount of time I spend on this forum almost literally every day with a lot of that time spent trying my best to help other people figure out how this stuff works and fix problems they are having while trying to figure it out myself at the same time. I think that counts for something.

And I already gave a recommendation on what I thought could help with this in the other thread. The other thing that I suggest (and have suggested before) is to provide more detail in the breaking change listing in the release post of exactly what the change was or somehow annotate the docs to indicate what the change was. Right now the way it’s done by just pointing people to the new updated (??) documentation is almost useless to figure out exactly what changed since no one has the docs memorized to know what they looked like before to allow them to figure out what is different now.

But I’ll also make a better effort to try to fix the documentation where I think that I can reasonably make it better. Again, not being a developer my knowledge of the nuances of the code is pretty limited and is based completely on what is already in (or frequently not in…) the docs on the HA main website.

Please don’t get me wrong I love this software and I really appreciate the work by all of the “official” developers and especially the “non-official” one-off developers who improve HA on a regular basis. But this software is literally nothing without the support of people who use it and the documentation and announcements should be aimed at those people. Not the developers who already know all of this stuff in the first place.

OK, so right now I’m using 12 custom components, 8 of which add to a pre-existing official component, 4 of which are “sensors”.

When a developer creates a new custom sensor component then all of the files for the official sensor component have to be copied over and included in the directory for every new sensor component created? So you will have multiple replica’s of the same file over and over again in different sub-directories in the custom_components folder?

3 Likes
7

A little tongue-in-cheek metaphor:

After a new version is released the docs indicate “Oceania was at war with Eastasia. Oceania had always been at war with Eastasia.”

Except we’re pretty sure it was Eurasia, but there’s no evidence because all the history books have been altered.

:wink:

What would be handy is a means of inspecting a doc page’s change history like you can do on Wikipedia (see 1984 change history). Maybe this is already possible via the doc’s Github repo but:
a) I don’t know how to that.
b) It’s not as easy as clicking “History” directly on the doc page being viewed.

Food for thought; maybe worth adding as a Feature Request.

4 Likes
8

I’ve already suggested something to that effect in the past (and it basically got the same fury of responses then as now) but it should be done directly in the doc page. You shouldn’t have to go and dig thru a change log in a separate github page somewhere.

My suggestion then was, since there is already someone in the process of actively, right now, editing the docs who is likely the most intimately familiar with the new code changes (or they shouldn’t be making “breaking change” doc changes in the first place) it would (should…) be trivially easy to indicate in those docs what change was made with which release.

That is awesome!! :smiley:

Does anyone of any import actually read those and act on them? How many votes does it take to make it onto the radar of the devs if at all?

9

Well, It’s a variation of that, but there really is just a “History” button to click, but you have to know how to get there. The results might not be the most helpful, but it’s there. Since the entire project, including the documentation is stored in github, all change history is stored.

For example, the Utility Meter component:

The component documentation page (linked here) has a link in the top left named “Edit this page on GitHub”.
Clicking that link will take you to the github page for the documentation (linked here).
Once there, there’s a “History” button which will take you to the github changelog of the documentation page (linked here).

I’m not saying these are good answers, but I’m just listing the methods avilable.

1 Like
10

Thanks! :+1:

I agree it’s not obvious (“Edit this page on GitHub”) but then, if one looks beyond mysterious terms (to the layperson) like “Raw” and “Blame”, one sees the magic word “History”.

I used it to navigate to the History page for Dark Sky Sensor and there’s the record of “Eurasia” changed to “Eastasia” … err, I mean update_interval changed to scan_interval (link). :slight_smile:

That works for me but it would be more obvious if it were shown as a “History” link next to the “Edit this page on GitHub”.

11

I see “Feature Request” as “Wish List”. Submit your wish and maybe someone will make it come true.

The voting aspect is just a gauge showing how many others have the same wish. However, there’s no threshold beyond which a wish is automagically granted.

Basically, it all boils down to your wish piquing the interest of someone possessing the necessary coding skills to grant your wish. A whole bunch of people might have the same great wish but if all coding genies finds it boring, it ain’t gonna happen. :slight_smile:

1 Like
12

yeah, those questions were largely rhetorical.

I don’t bother to ever make a feature request anymore and I only vote on requests just as a show of support for the person making the request to give them some validation that someone else thinks their idea is a good one even if it never gets implemented.

2 Likes
13

Hum… As always, I read the docs carefully before upgrading, but this time, I’m lost ! :thinking:

I’m no dev, can’t handle github and so, but I’m trying to do stuff. Those seem likely to get broken if I upgrade.

So, what should I do, I have the following cases :
1 - made a custom component on my own, new platform. Eg : FortiOS.py as a device tracker. With 0.87, it stays in <HA>/custom_component/device_tracker/FortiOS.py. If I got this correctly, I’ll have to put it in : <HA>/custom_component/fortios/device_tracker.py ? Can I do that right now with 0.87 or just as I upgrade ? What about the __init__.py ?
2 - changed an existing component, denon.py as a media_player. Same thing, I have to create a folder denon and put the code in a media_player.py file ? That’s it ? Or should I add more files ? which ones ?
3 - I use several custom component called “broadlink.py”, climate, media_player. If I get it well, they’ll all go in the custom_component/broadlink folder, each named on their category, so climate.py, media_player.py. Again, what about this __init__.py ? What about the other broadlink component that remain in the core installation ? Should I copy ?
4 - and what about the customizer that is also in custom_component ?!

14

Did you get an answer to your questions? Especially what to do about the init.py

15
  1. Exactly, you create a folder <config>/custom_components/FortiOS/and an empty __init__.py. And you move your FortiOS.py there and rename it as device_tracker.py. You should be able to do all that before upgrading without damage. But I’ve heard that <config>/custom_components/device_tracker/FortiOS is still valid.
  2. Exactly, just put that file there, just like in 1).
  3. Since there is no broadlink component yet just do as you said and create an empty __init__.py as stated in the blog article.
  4. I don’t know about that, sorry.

Rather than overriding built-in integrations, make a slowhand_darksky/sensor.py , and put sensor: platform: slowhand_darksky in your config.

16

It might be helpful to clarify in the original blog article, for us average users:
In what case should we copy the __init__.py? In what case would a empty __init__.py be OK? In what case should we copy the entire folder to the <config>/custom_components/xxxxx/ folder, along with my my_customized_and_replacement.py?

A couple of examples/scenarios would be great.

17

1

2

3

I’m a bit late with this, but apart from comment 1 above which I accept is slightly subjective how can anyone not agree with comments 2 and 3? Frankly I’m amazed that such a generally ‘professionally’ run project like this has no way for the ordinary user to track what has changed (broken their system) and when.

18

Ok thanks… I begin to understand how to make it work. Not how it’s supposed to work (and why this changed), but never mind, I’ll see that later.
Basically, I should just change directory structure, add an empty __init__.py in each folder and pray it works !

I rely on custom component for device tracking and heating, so I’d rather have this working !
And if it doesn’t, time to restore a snapshot !!

19

@k8gg As far as I know Paulus will update the information on that topic, but as far as I know as a general rule of thumb, if you don’t modify a component like for example hue or mqtt which have several platforms, just create a new folder for your custom version and add an empty __init__.py. Also don’t override the built in components but rather create <config>/custom_components/<my_darksky>/sensor.py for example. I hope this helps.

@klogg If you read the changelog for each release you haver a pretty good overview of what has changed, I don’t see what that is not sufficient? Add an individual changelog
to each platform is a lot of overhead in my opinion. The docs should present the state that things are not how they once were in my opinion.

@Mister_Slowhand The reason this was changed is to not break a users modified version by changing any of his dependencies. That is why you’d have to copy tho whole directory if you want to modify something like netatmo for example. Good luck with your migration. And don’t hesitate to ask for help.

20

I think the reason people find this ‘insufficient’ is that it means reading through all the changelogs for all components for all upgrades since your installed version (ignoring for now the idea that not keeping up to date with latest releases might not be in some peoples opinion the best course of action).

I can’t argue with that, but from the outside, one compromise solution would be to simply keep a copy of the previous version of the docs. Again from the outside it seems that all that would require would be a simple file copy before updating? (I admit to having little experience of maintaining documentation on a project this size so I apologise if this isn’t sensible).

Again, I can’t disagree with you here but still think the docs for such a ‘fluid’ project should provide some (simple) way to see what has changed and when.