Request to bring back YAML - arguments against The Future of YAML

5

Excellent post!!

Very well written, reasoned and explained.

It will be interesting to see what the response will be (if any).

4 Likes
6

Excelent post. Thanks.

7

@finity @rak @eifinger

Thank you. I hope they do reply. The post itself is on GitHub and you can contribute to the messaging and perspective.

I am also working on capturing all the broken user flows and expand on the potential fixes; although, of course, I do not work in the architechture, so the input from the Home-Assistant team would especially be needed there.

8

You are incorrectly stating that we can write to YAML files. We can’t without losing structure of “include” tags and preserving comments. We’ve did this once with Lovelace and got a lot of anger from the community.

About your same data in JSON and YAML. You’re greatly oversimplifying things. It’s not just OAuth2 flows.

Thank you for agreeing that it increases the maintenance cost. It’s not just that however. It’s contributors that don’t want to maintain it for their integration got harassed. By making it a project rule, people will voice their anger at Home Asssistant as a whole instead of individual contributors. If creating a space that allows contributors to not be harassed means some people will not contribute, that is a choice I will stand by.

I can’t believe that you’re looking at 100 comments to this blog post and try to draw conclusions from it and use it to predict how the whole community is going to feel. You’re ignoring all the other places that people comment on the blog post and the self-selecting bias of people that will actually comment.

Your solution is over simplified and completely ignores how config entries, devices and areas work.

As always, the devil is in the details.

9 Likes
9

Hi @balloob, first of all, thanks for taking the time in reading and replying.

I am not sure where I wrote that, but that’s not what I meant. If you can point it out, happy to Edit it. The request is simply about being able to read from (not write to) editable configuration files that users write. In all cases I said YAML, but other structured, editable and documented formats are equally valid.

I will expand on the solution in an upcoming post.

I would love to understand more to help think through a solution. I will be posting an extended version of the proposal (although essentially the same) and I would love to know: (1) how should I bring it on to ADR and (2) what flaws are in there so we can work with a proposal that does work.

Happy to partner in the right way, if you truly are willing to listen.

I said that it doesn’t have to, though, if you just read from configuration, change the schema and pass it on to the same flow that the JSON files use. There might be other solutions which do not involve a significant cost of maintenance (compared to the cost of dropping it), or that it doesn’t put this cost on the contributors.

I actually touched on all the reasons that were mentioned on your post. I spent the time to understand and bring counterarguments to all of them.

This is really unfortunate. I have never seen such declarations on the board or GitHub, but it is sad that people feel that way. Are there any examples and/or stats that you could share without affecting the privacy of the people involved?

While I do not want to see harassment, I do not think that taking this decision is the right one.

For one, you can see that this decision is also generating a lot of negative comments and will continue. Some people were asking to distinguish projects with and without YAML for example and you can expect similar behaviour.

Second, people will find other reasons to harass contributors for any other decisions. As such, if you really want to avoid all bad behaviour, the slippery slope is simply close the project so there are no communication channels. But that doesn’t make sense, right?

Closing YAML does not feel right for the same reasons either.

This is great. We are all aligned. In what ways can we achieve this that would work for the whole project? I really cannot believe that all harassment comes from whether you support YAML or not on your component.

Let’s work on solving the root problem, not in using this as an opportunistic excuse to take a product decision. (Same as for what the “health” part of the Linux post was heavily criticised, essentially).

Definitely agree.

I actually read all the 600 posts and I am about to compile a list of all the user flows that are broken. Nevertheless, not even 600 posts are enough to represent the whole community.

If you read again on my post, I was asking for the data that you used to take that decision. Lacking any data on the original post, some data is better than none. If you can provide better data, we would appreciate it and we can use that instead.

Definitely. I do not work on core or architecture of Home-Assistant. I would love to know, though, where exactly the solution is breaking to change it and adapt it. I will be posting more on this soon. However, just saying “your solution does not work” is not good enough without at least an explanation on why not and where it fails.

Let’s work that devil down, so we can have a solution that satisfies everyone.

Thank you

8 Likes
10

In the previous post, I left open what user flows are broken. I’ve gone through all the 600+ responses from the post and tried to compile them into a few user flows that are broken.

This is based on the work and posts of many, including: speedfire, eifinger, mpex, ReneTode, balthisar, Tinkerer, Razgriz, someone, scstraus, danielo515, Julien, debackerl, CentralCommand, finity, bhaonvashon, angelnu, mf_social, mig2008pt, 123, Mariusthvdb, nickrout, tom_l, wellsy, lindsayward, kanga_who, lmamakos. (Not allowed to mention people).

I hope I was able to capture your opinion correctly. Otherwise, similar to the other post, this is public on GitHub and accepts contributions.

After this, I will be expanding on the potential fixes; and seeing how’s the best way forward to get them proposed and reviewed.

What workflows are broken when YAML is phased out for Integrations

In the post The future of YAML, Home-Assistant team explains the decision of ADR0010 to drop YAML support for integrations.

One of the main points is to Make things easy for users. This is partially true as having a powerful UI makes it easy for many users. However, there are different users and use cases; and UI is better for some, but worse for others compared to YAML.

In this article, we talked about some user flows that were broken; and here we can expand on them.

Summary of Broken Workflows

The full list of reasons, whose use case and explanation is expanded below:

  1. All current configurations will eventually break.
  2. No easy bulk addition or support.
  3. No partial versioning of components.
  4. No flexible backups and restoring.
  5. Reduced shareability for integrations and related entities.
  6. Increased difficulty in troubleshooting.
  7. Reduced documentation for users and developers.

Breaking existing integrations

With UI configuration only, the developers are effectively encouraged to remove YAML support, even from existing integrations. This will effectively break every single current Home-Assistant installation which uses this. In may cases, these are hundreds of integrations over long periods of time.

With YAML configuration, everything keeps working as today. You can still use YAML, or happily move over to UI; progressively at your own pace, or at once.

Users affected: (everyone, technically), 595, 634

Bulk Addition or Editing

There are changes that may affect several or all entities. To name some of these:

  1. A change of local IP subnet or IP ranges that affects all devices.
  2. A revamp of your identity or name that may affect usernames.
  3. A decision to name all your devices on a certain structure.
  4. Hiding or showing all/most entities on Google Assistant.

With UI configuration only, making bulk changes to many entities in bulk will be painful and time consuming. Having to go one by one, without having a good way to keep track the entities that were changed. Not only that, but if I want to change all the usernames where I used to have value “x”, there is no effective way for me to search and find all the configurations affected.

With YAML configuration, making bulk changes can be done easily in a configuration file. Using editors or git also will allow to search, and compare changes. As such, it is easy to find the places where changes are required, edit them and keep track of progress as you do it.

Users affected: 27, 44, 58, 114, 220, 450

(Partial) Versioning

With UI configuration only, it becomes harder to create “dated” versions of configurations. If I change the settings of one component to experiment, and I want to revert them one week later to the status one week before, there is no way for me to do it. The options are restoring a full back up (which may be reverting more changes that I want to keep), annotating screenshots, having my own means of documenting them (i.e. my own useless configuration file), or reading the .storage files and manually trying to understand the undocumented configuration files. In other words, there is no easy way to import partial past configurations (i.e. one or two components only).

With YAML configuration, all the versions are kept in track if you use a software like Google Drive or GitHub, you can see the history of changes and you can easily import the state to any of those versions.

Users affected: 2, 27, 124, 185, 457, 478, 481

Flexible Backups and Partial Restores

Backing up is supposed to be well covered by the current system. However, this is only under certain conditions. This is the solution proposed in the post:

Using the Home Assistant snapshot feature, this is not an issue. However, if you do manual backups on a system that runs just Core, you need to make sure to back up the .storage folder as well (which hopefully you’re already doing). Otherwise, there is no difference.

And related to git:

This is actually not true, the .storage folder contains all Home Assistant managed configuration files in JSON format, which in those cases, can be stored and versioned in a git repository.

With UI configuration only:

  • The snapshots that Home-Assistant takes are full copies (you can select which options, but you copy all those). When you restore, you are restoring a full version.

  • The backups are heavy and it might not be ideal to keep the backups for months or weeks in case you want to restore to older changes.

  • Snapshots are not available in all systems (e.g. Docker installations).

  • If the backups are on a system like a Raspberry Pi, you are at a risk of losing them if the SD card goes corrupt and starting from scratch.

  • To solve all these, you require additional systems (e.g. move them to NAS) which are not part of the native system; and requires technical implementations much harder than using Git for your config files.

  • Backing up .storage allows you to store this data, but it has sensitive data which only would work on private respositories.

With YAML configuration:
With YAML you still have all the previous options; however, now you are empowered to do more.

Adding configurations.yaml plus a simple (and available to everyone) like git or GitHub, you can:

  • Create and import different versions of components; in full or partial (see versioning above).

  • Keep years of history and be able to check, revert and have detailed data to make back ups and when you did the changes.

  • Import your configuration to new and fresh systems without importing everything that comes with the backups.

  • All the sensitive data is input by you and can be stored in secrets or means that make sense without; as opposed to having to download them from .storage folder and having to either update all or nothing depending on whether you can upload sensitive data. Note that the .storage folder so you do need to make continuous backups in case schema or other areas change.

Users affected: 72, 185, 193, 272, 335, 363, 367, 481

Shareability

A lot of us learnt and started using Home-Assistant by learning from the configurations of others. This is not only from automations, but also about the key integrations and how they are used.

With UI configuration only, the integrations are no longer shareable as part of your configurations. The JSON files contain sensitive data (like tokens or passwords), which cannot be removed automatically using systems like secrets. As such, only manually edited and cherry picked files can be shared. As a result, the sharing ecosystem will progressively weaken. Even for the elements that are still shareable (like automations or template sensors), they lose a lot of context when you are not aware of the integrations implemented (e.g. that binary sensor, what is it tracking?).

With YAML configuration, one can share their configurations easily for others to learn and get inspired. Using secrets allow to make this secure without manual intervention.

Users affected: 2, 9, 135, 181, 185, 187, 193, 449

Effective testing and troubleshooting

The post addresses this point by stating:

YAML configuration testing is often done to see if a specific YAML configuration is still valid against (newer versions of) Home Assistant. With integrations set up via the UI, this is not a concern, since Home Assistant ensures the data structure is compatible between versions and migrates it for you.

Moving across servers or Home-Assistant versions

It is possible for users to have several versions of Home-Assistant at points of time; be it because there are migrations between servers, or tests between development and production installs. This is not just a problem of testing upgrades to the next version.

With UI configuration only, there is no effective way to fully migrate between servers. Snapshots provide part of that functionality, but it has the problems described in the backup section (e.g. not being able to only move part of the installation, or installing in different versions of Home-Assistant). The alternative is to edit across multiple files of undocumented JSON (risking breaking the system as you are not meant to edit them), or having to start from scratch.

With YAML configuration, you can easily copy/paste all or parts of the configuration and install; allowing you to start fresh installations.

Partial Breakages

Configuration entries is not the only way a component can fail. There can be bugs or conflicting modules that may not make your system to work.

For example, some components temporarily failed (example nmap) and I had to disable them. Note that this is not a problem with the configuration, but with the component itself. The UI would not load at all, so the only way I was able to solve it was by connecting via Samba, commenting out the configuration and restarting via SSH. All worked after.

With UI configuration only, there is no way to isolate components temporarily. Moreover, if the UI is broken, there is no way to disable or remove components. You are stuck with a broken system. In other words, if the system is crashing due to a bug, there is no way to access the UI to disable or remove the component and JSON files do not allow an easy way to do it. There’s not a good way to recover from this mode.

With YAML configuration, you have control over which components to copy and isolate. You can also comment out components even without access to the UI, which allows you to test hypothesis, iterate and recover the system from failure.

Troubleshooting

In the past, when I reported bugs, I had to recreate and isolate the bugs (example). To do this easily, I usually copied the affected configurations from my main installation into a docker or dev environment.

In some cases, it was conflict between two or more components (some might be custom, but not always or all of them). For example, I recently had a conflict between a custom component (Hue Sync Box remote) and an official component (Harmony). To troubleshoot it, I moved the two components to a dev environment and I was able to identify and fix the issue.

With UI configuration only, there is no way to isolate components temporarily or move them into a dev environment quickly. Testing and troubleshooting becomes costly as you need to reproduce the partial setup on the UI, or restore an exact copy which might not be helpful. There is also no easy way to share the relevant configuration for others to recreate the environment and be able to fix it.

With YAML configuration, you have control over which components to copy and isolate. You can share the configuration for others to reproduce. You can also comment out components even without access to the UI, which allows you to test hypothesis, iterate and troubleshoot.

Users affected: 128, 183, 255, 481, 561, 568

Documentation for developers, users and the curious

One of the greatest things about Home-Assistant is the really good documentation of components; with all their parameters that allows you to learn and experiment.

With UI configuration only, all the data is opaque to the user and developers. One could argue that this is by design. However, it is limiting contributors who can learn, tweak and in the future contribute to the main components.

With YAML configuration: those who are willing to learn and contribute have plenty of documentation on the integrations page. Since YAML is an option, the configuration details need to be present.

Users affected: 68, 89, 332, 419, 454, 594, 625

25 Likes
11

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)

12

This is a fantastic summary! Well written and very acurate

3 Likes
13

After you read and understand all you have summarised @nitobuendia I find it impossible to understand the reasoning of developers in their current development path.

New and future users will not even understand exactly what it is they are losing (what has been taken from them?) unfortunately as they are being blinded by how easy it will be with a shiny new UI!

4 Likes
14

Something is bugging me about this. This isn’t actually the whole story is it? If someone was going to support both YAML and UI configuration this would only be step 1 (getting a config entry created from the YAML config). In addition to this they would need to:

  1. Define the schema for the YAML so that check config correctly tells users whether they have entered YAML correctly or not. Otherwise they would restart and encounter runtime failures.
  2. Handle changes to that schema over time. Since YAML cannot be automatically changed the YAML schema can get complex as multiple versions are introduced. Either the complexity of that validation increases over time or the developer must introduce breaking changes to clean up the config schema
  3. Add tests for their config schema to ensure that code works correctly
  4. Write documentation telling users where and how to get these pieces of metadata. Since there’s no UI driving the config collection process they must write documentation telling users how to do it separately.

Unless I’m missing something your solution above doesn’t really cover any of this right? This is all still work developers would have to do in addition to the work creating their UI config flow in order to support both YAML and UI configuration. I think this is all the part that’s potentially expensive to maintain and requires extra work on the part of the developers. I agree that the actual process of converting YAML into a config entry doesn’t seem particularly difficult but there’s more to a YAML based method of config collection then that.

Now one might argue that the schema I’m referring to is required by both methods of config collection so this isn’t extra work. But that’s not really true. I’m looking at some of these config_flow.py files and the schema referenced in them is per screen. Each screen has a schema describing what inputs should be shown and how those should be validated but those aren’t necessarily the integration schema. Some of those are and some of those are intermediary fields used to collect other data (like oauth tokens for example). At no point does the final configuration schema seem to be consolidated and laid out. Probably since the author is expecting the UI to collect it so its not really necessary to do so. Which means a separate schema would need to be created to support YAML configuration.

I’m also noticing while going through this that these config flow files are very complicated. The shortest ones appear to be a minimum of ~60 lines (Emulated Roku, Flu Near You, GDACS). PS4’s is significantly more complex at 200 lines and the longest I found was Vizio’s at a whooping 500 lines.

This makes me wonder if perhaps we’re targeting the wrong thing here. Perhaps instead of looking at how we can make YAML work easier with config entries (since as you showed, that’s actually pretty easy) we should thinking about how we can make it easier to make UI configuration? If these UIs required less work to configure then developers might be more willing to support YAML config as well. And if we figured out a way to make these config flows more data-driven then they may not even have to since the same schema could be re-used for YAML entry without requiring users to physically go through this UI.

6 Likes
15

I approve every lines you write. I stopped updating my HA because of that and was thinking about a solution similar to what you describe if I were to fork this project. I admire that huge amount of work you put to speak out loud and summarize our concerns with that project direction. Impressed.

2 Likes
16

You are right. Aligned with your points. On the sample component I did for PS4, I had to do the following:

  1. Define the PLATFORM_SCHEMA (i.e. define a YAML schema which would validate) [6 code lines].
  2. Document the structure documented on integrations (not done yet).
  3. Adapt YAML structure, to the structure needed by the device (this is independent to the JSON structure) and it is know by the creator as they use those fields from config_entries [~20 code lines].
  4. The tests would be simply making sure that your device has the right values gathered. Everything else is already tested by the other tests.

This obviously requires updates when schema changes, but so does the UI one and the changes should be a handful of lines. I am advocating for breaking changes if you use YAML instead of UI-managed.

I am working on a full view on the solution (from different angles and problems) and I will bring this feedback in. If you want to discuss ideas, let me know.

Thank you.

I have a partially different view.

  1. The integration schema is definitely known to the contributor, as it is the required one for creating the Entity. For example, most of PS4 are found here (some exceptions within init function).

  2. Yes, the config flow stages this in several steps. The important part is that, at the end, you will have all that information. One option to reduce the cost of maintenance of the documentation:

    a. User goes through the UI flow that guides them.
    b. At the end, show the config information. The user can decide to Save (UI-managed) or copy this information to a manual YAML configuration (e.g. UI manage supports Edit on UI, YAML does not).
    c. As such, we only need to say the schema; not document how to obtain them.

This feels like it is integrated here. The attributes are collecting the data of the flow and you can show this data, store it, or do whatever you want with it. My solution would be to show the data in the UI without storing it until the user confirms. This is just a change in the approach UI flow works (ADR agreement), but no extra cost.

This could make sense too. What is important is the overall maintenance cost, and not the individual parts. If we get a word from Contributors that this is the issue, I think it is a fair point.

However, this has not been raised as the issue. YAML, and maintaining both workflows, has. You might be right, I am just wondering if we could get a Core Contributor perspective on it to know where to focus our efforts.

Knowing really what is the real cost here would make a huge difference to come up with a good solution. I did feel the original reasons were not the full picture here.

1 Like
17

One of the custom components uses the async_setup to import the YAML config seems to work quite well.

This one specificly.

3 Likes
18

This is a great example! So looking at this you’ll notice the lengths the author had to go to in order to support both options given the complicated config. The schema for their component is kept here which is then shared by both YAML and UI config. And they even note some of the challenges they faced doing this in the comment:

"""
Type and validation seems duplicare, but I cannot use custom validators in ShowForm
It calls convert from voluptuous-serialize that does not accept them
so I pass it twice - once the type, then the validator :( )
"""

But the net result is good! Exactly what I was talking about. This is a significantly more data driven UI. You’ll notice they’ve created their own schema for describing the config options that includes how to validate, type, method, default, conditional dependencies and which screen to show it on when in the UI. This allows them to use this common compile_schema method in both places to generate the schema used for validation whether in the UI or YAML.

Adding YAML support when the config is defined this way is clearly quite simple. There’s a few conditionals in __init__.py based on where the config is coming from but for the most part everything about the YAML config is shared with the UI config.

Unfortunately despite all this work the author put in the config_flow.py file is still 500+ lines long. I had hoped that with all the work the author put in to define common config and that config_singularity.py file the UI would require less work but looks like its still pretty complicated. An initial scan shows that a lot seems to be in error handling which kind of makes sense given the authors earlier comment. Since validation extends beyond basic type and into more complicated validators they then have to write additional code to handle that while in the UI.

Anyway I wonder if there’s something that can be pulled from here into the proposed solution to simplify the process of providing both config options to users with reduced developer effort.

1 Like
19

Well the component in itself is kind of complex, here is a simpler one. Much less code same YAML/UI Config ability.

3 Likes
20

Maybe a good path forward is to showcase what would a “best class” implementation of UI + Configuration looks like.

The PS4 example was a first example and fell short in some areas. I am happy to work on the Cast component and see how it goes.

The goals would be:

  1. Minimise SCHEMA to the minimum to avoid breaking changes and maintenance cost.
  2. Use the UI flow to:
    • Configure the entity, if the user wants that UI-configured.
    • Obtain the YAML configuration, if the user prefers to use manual set up.
  3. Allow the user to input YAML directly, even without going to the UI.
  4. Share as much code as possible between both set ups, so there’s no additional cost.
  5. Have a way to work with Auth token that requires user interaction.
  6. Have a way to work with refresh tokens (and alike) that are dynamic over time.

If we are able to find a good, easy to maintain path forward; it can be presented to ADR.

If someone is trying similar things, feel free to contribute to the repository too. It would be nice to create a repository of YAML solutions (opinions exposing limitations of UI-only, potential solutions to bring back YAML, or custom components building on core components to enable YAML).

4 Likes
21

While I work on other experiments, I have added some documentation to the PS4 one. This expands on all the changes that were required to make it work, and also a bit of additional explanation if someone else wants to build similar components and experiments.

Once completed, I will also add it to Custom Components as a full solution. At the moment, it’s ready and works, but there’s no documentation on how to obtain the token without using the UI. I want to change that.

In the meantime, the full explanation:

Objectives

What was the objective

To showcase that adding YAML is not a high maintenance task if the set up logic is share between the main config_flow, async_setup_entry and the deprecated async_setup_platform.

By delegating the implementation of async_setup_platform to async_setup_entry, the extra code and maintenance is very low and would allow to add YAML with minimal changes.

What was not the objective

This solution does not represent an end to end solution since PS4 does not require a lot of discovery of devices, OAuth flows or refresh tokens. Further experiments should explore these other areas and find a solution.

What was required to make it work

The changes are very minimal and explained below.

Do note that as this was distributed as a custom_components, we also imported config_flow, translations, services.yaml, manifest.json, etc. This is done in order to keep support for the UI and the custom component working perfectly. However, in the case of a core component this would not have been part of the implementation.

1. Define schema

In order to be able to validate the schema, it is required to define it first. While this is optional, it provides a lot of functionality for very little lines.

import voluptuous
from homeassistant.helpers import config_validation

PLATFORM_SCHEMA = config_validation.PLATFORM_SCHEMA.extend({
    voluptuous.Required(const.CONF_HOST): config_validation.string,
    voluptuous.Optional(const.CONF_NAME): config_validation.string,
    voluptuous.Required(const.CONF_REGION): config_validation.string,
    voluptuous.Required(const.CONF_TOKEN): config_validation.string,
})

This defines the 4 fields that are required to run a PS4 media player. With this code, a new entry can be created using the usual configuration:

media_player:
  - platform: ps4
    name: "PS4"
    token: !secret ps4_token
    host: !secret ps4_wifi_ip
    region: "Singapore"`

2. Create an async_setup_platform (setup_platform)

The only task that async_setup_platform performs is adapting the data that comes from the configuration.yaml to the required input of async_setup_entry.

The schema that the device requires is formatted like this:

config_entry: {
  entry_id: string  # Unique entry id. You can use uuid or simply entity id as it's unique.
  data: {  # Contains devices and platform information.
    token: string  # PS4 token obtained from the app.
    devices [  # List of devices.
      {
        host: string  # IP of the PS4.
        name: string  # Name of the PS4.
        region: string  # Region of the PS4.
      }
    ]
  }
}

The only condition is that entry_id and data are accessed like attributes of a class (i.e. config_entry.entry_id or config_entry.data), whereas all the data inside config_entry.data is shaped like a dictionary (i.e. config_entry.data['token'] or config_entry.data.get('devices')).

The code that was needed to transform from our configuration to config_entry is like this:

import collections

Config = collections.namedtuple(
    'Config', f'{const.CONF_ENTRY_ID} {const.CONF_DATA}')


async def async_setup_platform(
        hass, config, async_add_entities, discovery_info=None):
  """Loads configuration and delegates to official integration."""
  # Load configuration.yaml
  host = config.get(const.CONF_HOST)
  name = config.get(const.CONF_NAME, const.DEFAULT_NAME)
  region = config.get(const.CONF_REGION)
  token = config.get(const.CONF_TOKEN)

  # Format it in the new config_entry.yaml format
  config_entry = Config(
      util.slugify(name),
      {
          const.CONF_TOKEN: token,
          const.CONF_DEVICES: [
              {
                  const.CONF_HOST: host,
                  const.CONF_NAME: name,
                  const.CONF_REGION: region,
              },
          ],
      }
  )

  await ps4_media_player.async_setup_entry(
      hass, config_entry, async_add_entities)

  return True

The method async_setup_entry uses a different format of config_entry than what the configuration.yaml provides. As such, we used a namedtuple as a proxy for a very simple class that allows us to access fields like config_entry.data as it’s done in the PS4 entity.

This currently supports only one device, but you could just keep appending the data to devices if using the same token or create a full new Config if it comes from a different platform.

3. Documentation

When manual configuration is available, the last step would be adding some documentation. In this case, we advocate to leave very minimal configuration and rely on the UI flow to generate the details if ever needed. This way, documentation does not become a burden.

Further experiments can explore how the UI can become a source of data for configuration.yaml without having to create advanced documentation.

4. Delegating all the other code

For those of you reading this with the intention of creating similar custom_components, I want to add a bit of explanation on how to achieve it. This is not part of the experiment, but of the custom_components logic and it would not be needed if the rest of the code was integrated in the core component.

You need to copy services.yaml and manifest.json to describe your services. This will allow the platform to know all the services that are required by your system.

Do not copy the whole component, you can easily delegate the implementation by doing this:

from homeassistant.components.ps4 import media_player as ps4_media_player

async def async_setup_entry(hass, config_entry, async_add_entities):
  await ps4_media_player.async_setup_entry(
      hass, config_entry, async_add_entities)

This way, any changes on the core component will not affect your component and you will be able to take advantage of them by simply maintaining the Adapter logic.

If you want to support the UI flow as well, you can simply use the same mechanism to delegate it:

from homeassistant import config_entries
from homeassistant.components.ps4 import config_flow

@config_entries.HANDLERS.register(const.DOMAIN)
class PlayStation4FlowHandler(config_flow.PlayStation4FlowHandler):

  def __init__(self):
    super().__init__()

This will implement the same UI flow that the core component has without having to do any coding. In other experiments, I will showcase a better way to improve the UI flow so it also spits out the required code for configuration.yaml, but for now this is a good sample start.

If you do this, you also want to import the translations folder so the messages show. In this experiments, I copied both manually, but it might best to have a way to copy the core folder to your custom_components automatically. This could be showcased in another experiments, but the main focus is to show how to minimize the core components support for YAML.

As for the constants, in my case I decided to create my own const.py to have flexibility, but virtually all the constants are coming from core constants or ps4 constants really:

# Platform constants.
DOMAIN = 'ps4'
PLATFORMS = ['media_player']

# Attributes.
CONF_HOST = const.CONF_HOST
CONF_NAME = const.CONF_NAME
CONF_REGION = const.CONF_REGION
CONF_TOKEN = const.CONF_TOKEN

# Secondary data attributes.
CONF_DATA = const.CONF_SERVICE_DATA
CONF_DEVICES = const.CONF_DEVICES
CONF_ENTRY_ID = 'entry_id'

# Services.
SERVICE_SEND_COMMAND = 'send_command'

# Service attributes.
ATTR_ENTITY_ID = const.ATTR_ENTITY_ID
ATTR_COMMAND = const.ATTR_COMMAND

# Default values.
DEFAULT_NAME = 'PlayStation'
4 Likes
22

During the weekend, I tried a different experiment. I used Hue instead of Cast because Cast uses Discovery instead of the UI flows.

The objective of this experiment was to showcase how easy would it be to show the device information in the UI before writing the data. The ultimate objective is to show that the UI can be a good way to onboard the component, that would reduce the cost of maintenance of documentation; while still providing an option for YAML Components. As always, the conclusions are at the end and you can navigate through the titles.

I am compiling all experiments on this component, and it’s open to anyone who is working on similar objectives.

I want to send this experiment (or a variant) to ADR, so all feedback is welcomed (positive or negative).

Hue Experiment

This is an experiment to showcase how UI flow can act as a self-documenting feature for YAML configuration.

Objectives

What was the objective

To showcase that the UI configuration flow can be used to generate the same data needed for the configuration.yaml and allow the user to take the last decision on the configuration.

The underlying objective is to prove that the same config_flow can be used to set up both UI and YAML configuration without adding significant cost to the component owner.

What was not the objective

To implement the code required to read YAML configuration. This is why we focused on Hue component which still supports YAML configuration.

This experiment is also not intended to be a custom_component as the original core component has all the required documentation and features, but it could be used in the future. Nonetheless, this would work if it gets installed as a custom_component.

What was required to make it work

The only change required is to add an additional setup before async_create_entry that confirms the output with the user for validation.

0. Understanding the original device creation

The method async_step_link in the hue core component is responsible for getting the bridge information and creating the entry device. This is the logic responsible for this:

async def async_step_link(self, user_input=None):
  # ...
    return self.async_create_entry(
        title=bridge.config.name,
        data={
            "host": bridge.host,
            "username": bridge.username,
            CONF_ALLOW_HUE_GROUPS: False,
        },
    )
  # ...

However, we wanted to add an additional step to ensure that we could validate and read the configuration. As such, we have replaced this logic with a few changes described below.

1. Store device data into a class attribute

Instead of directly creating the entry, we store the data required into a class attribute, which will allow us to use it later to finally create the entry.

  self._set_up_data = {
      'title': title,
      'data': {
          'host': host,
          'username': username,
          'allow_hue_groups': allow_hue_groups,
      }
  }

Note: while it was not required, for this experiment, we have stored the original values into variables as we will be using them in two different places on this implementation.

  title = bridge.config.name
  host = bridge.host
  username = bridge.username
  allow_hue_groups = False

2. Format data into YAML Configuration format

Change the format from config_entry (_set_up_data) to the one required by YAML configuration.

import yaml

yaml_data = {
    'hue': {
        'bridges': [{
            'host': host,
            'allow_hue_groups': allow_hue_groups,
        }]
    }
}
yaml_configuration = yaml.dump(yaml_data)

Note: yaml_data is only required because the data required by async_create_entry and YAML configuration are different. We advocate to have an equivalent setup so there is no need to maintain two different schemas. If this was true, this would only have required this line: yaml_configuration = yaml.dump(self._set_up_data)

3. Show data before creating the device

We need to edit the original async_step_link to make a call to a new step before calling async_create_entry. This is done by passing the previous data into a form like this:

return self.async_show_form(
    step_id="confirmation",
    description_placeholders={
        'yaml': yaml_configuration,
    }
)

Note: this logic replaces the previously described return self.async_create_entry(...).

4. Add additional step to show YAML configuration

When the form is executed, it triggers a form which will show the YAML data. Once the user submits the form, it will create the entry. The code responsible for this:

  async def async_step_confirmation(self, user_input=None):
    """Creates device entries after confirmation."""
    if not self._set_up_data:
      raise ValueError('Configuration flow failed.')

    return self.async_create_entry(
        title=self._set_up_data.get('title'),
        data=self._set_up_data.get('data'),
    )

As a new step is created, you need to also add some translations to show the message:

"confirmation": {
  "title": "Confirm Configuration",
  "description": "Your device configuration is ready. \n\nIf you prefer to set up the device manually, use this code on your `configuration.yaml`:\n\n```yaml\n{yaml}```\n\nIf you prefer to set it up via UI, simply click submit below."
}

Summary

While the boilerplate may seem long, by sharing the same configuration structure between YAML and UI, this would only require to make one self.async_create_entry() with a yaml processed details of the device to really make it work. This should not require more than 5-10 lines of code (including English strings) and provide great functionality.

Result

This will show the following message in the UI:

Form with YAML configuration for hue component

The user can copy the code into configuration.yaml or simply press submit to complete the UI setup.

Alternative implementations

This implementation may not look friendly for UI-only users. There are a few ways to improve this.

Optional YAML configuration

One option is to only showcase the YAML configuration to those who are YAML configuration. This could be a user flag that allows us to know if it’s an advanced user or not. This is similar to enabling/disabling lovelace dashboards or showing the Developer Tools.

The code will be something like this:

if is_advanced_user:
  return self.async_show_form(
      step_id="confirmation",
      description_placeholders={
          'yaml': yaml_configuration,
      }
  )
else:
  return self.async_step_confirmation()

Make configuration editable

Another option is to additionally show the configuration data to the user, so they can Edit it. This is not a replacement for YAML configuration, but a way to make this confirmation box also users by UI-only users.

This would require defining the form data via a Schema:

  data_schema = voluptuous.Schema({
      voluptuous.Required("title", default=title): str,
      voluptuous.Required("host", default=host): str,
      voluptuous.Required("username", default=username): str,
      voluptuous.Required(
          "allow_hue_groups", default=allow_hue_groups): str,
  })

Additionally, the form call would change slightly:

return self.async_show_form(
          step_id="confirmation",
          data_schema=data_schema,
)

As the form has changed, the translation text would also be different:

  "confirmation": {
    "title": "Confirm Configuration",
    "description": "This is the data configuration that will be set up.",
    "data": {
      "title": "Title",
      "host": "host",
      "username": "username",
      "allow_hue_groups": "allow_hue_groups"
    }
  }

The form data could be gathered by the confirmation step from the user_input instead.

  async def async_step_confirmation(self, user_input=None):
    """Creates device entries after confirmation."""
    config_title = user_input.get('title')
    config_data = {
        'host': user_input.get('host'),
        'username': user_input.get('username'),
        'allow_hue_groups': user_input.get('allow_hue_groups')
    }

    return self.async_create_entry(
        title=config_title,
        data=config_data,
        title=self._set_up_data.get('title'),
        data=self._set_up_data.get('data'),
    )

At the end, this would show like this:

Form with all details from UI configuration for hue component

The form data could also be combined with the one from YAML configuration to offer the best of both worlds.

Overall, this alternative implementation empowers users to confirm the configuration before creating devices, and can be a good step to also provide the data needed for YAML configuration in a more user-friendly way. The extra schema seems to add to the cost (despite being exactly the one required to be stored), and as such, we have opted for a more lightweight option for the main proposal.

Centralized confirmation on core code

This code was implemented by a custom_component and, as such, it has focused in modifying a core component to add this additional step. However, a more efficient way to implement this would be by implementing this natively on self.async_create_entry under core/data_entry_flow.py. Any data received to create a new entry will be shown to the user in a form who would need to submit before creating the entry. This works best when the configuration is optional and only shown for advanced users.

This effectively would have no cost for the components developers.

Conclusions

For Component Owners

The same UI flow can be used to generate data to both manual YAML configurations as well as UI configurations. This will allow to ease the need to document the source of the configuration.yaml parameters. This reduces the cost for component owners to provide YAML configuration.

For UI Users

For users who prefer to use the UI, there is no changes in their current workflows. At worst, the may have one additional step where they see the configuration. However, this can also be optionally shown only to user who enable “advanced” mode; similar to how some Configuration are shown only to some users. The config_flow allows to do this type of conditional send to one step or another.

For YAML Users

Users who want to maintain components via YAML are encouraged to use the UI to set up if they are not sure how the data should be filled in. However, once the data is filled in, the users can edit it or create new ones using the same structure.

7 Likes
23

There would also need to be some function to reload the integration upon reconfiguration as well to get the same functionality from YAML configuration as the config flow.

24

Hi @firstof9, thanks a lot for the feedback :slight_smile:

One clarification: this code does not actually write to the configuration.yaml file. As Paulus described, this would have problems like potentially breaking secrets, includes, etc.

As such, it just shows you the configuration information in case you want to opt-in for manual YAML configuration. You would need to copy/paste the YAML snippet there.

In YAML mode, edits are done Editing the configuration.yaml file, not via the UI. I might be wrong, but as far as I know these YAML configurations do not show up under Integrations and therefore support edit. If they do show up, then yes, when Editing you would also want to show a confirmation step before, but I am not aware of how this works at the moment.

1 Like