One of the custom components uses the async_setup to import the YAML config seems to work quite well.
3 Likes
This is a great example! So looking at this you’ll notice the lengths the author had to go to in order to support both options given the complicated config. The schema for their component is kept here which is then shared by both YAML and UI config. And they even note some of the challenges they faced doing this in the comment:
"""
Type and validation seems duplicare, but I cannot use custom validators in ShowForm
It calls convert from voluptuous-serialize that does not accept them
so I pass it twice - once the type, then the validator :( )
"""
But the net result is good! Exactly what I was talking about. This is a significantly more data driven UI. You’ll notice they’ve created their own schema for describing the config options that includes how to validate, type, method, default, conditional dependencies and which screen to show it on when in the UI. This allows them to use this common compile_schema method in both places to generate the schema used for validation whether in the UI or YAML.
Adding YAML support when the config is defined this way is clearly quite simple. There’s a few conditionals in __init__.py based on where the config is coming from but for the most part everything about the YAML config is shared with the UI config.
Unfortunately despite all this work the author put in the config_flow.py file is still 500+ lines long. I had hoped that with all the work the author put in to define common config and that config_singularity.py file the UI would require less work but looks like its still pretty complicated. An initial scan shows that a lot seems to be in error handling which kind of makes sense given the authors earlier comment. Since validation extends beyond basic type and into more complicated validators they then have to write additional code to handle that while in the UI.
Anyway I wonder if there’s something that can be pulled from here into the proposed solution to simplify the process of providing both config options to users with reduced developer effort.
1 Like
Well the component in itself is kind of complex, here is a simpler one. Much less code same YAML/UI Config ability.
3 Likes
Maybe a good path forward is to showcase what would a “best class” implementation of UI + Configuration looks like.
The PS4 example was a first example and fell short in some areas. I am happy to work on the Cast component and see how it goes.
The goals would be:
- Minimise SCHEMA to the minimum to avoid breaking changes and maintenance cost.
- Use the UI flow to:
- Configure the entity, if the user wants that UI-configured.
- Obtain the YAML configuration, if the user prefers to use manual set up.
- Allow the user to input YAML directly, even without going to the UI.
- Share as much code as possible between both set ups, so there’s no additional cost.
- Have a way to work with Auth token that requires user interaction.
- Have a way to work with refresh tokens (and alike) that are dynamic over time.
If we are able to find a good, easy to maintain path forward; it can be presented to ADR.
If someone is trying similar things, feel free to contribute to the repository too. It would be nice to create a repository of YAML solutions (opinions exposing limitations of UI-only, potential solutions to bring back YAML, or custom components building on core components to enable YAML).
4 Likes
While I work on other experiments, I have added some documentation to the PS4 one. This expands on all the changes that were required to make it work, and also a bit of additional explanation if someone else wants to build similar components and experiments.
Once completed, I will also add it to Custom Components as a full solution. At the moment, it’s ready and works, but there’s no documentation on how to obtain the token without using the UI. I want to change that.
In the meantime, the full explanation:
Objectives
What was the objective
To showcase that adding YAML is not a high maintenance task if the set up logic is share between the main config_flow, async_setup_entry and the deprecated async_setup_platform.
By delegating the implementation of async_setup_platform to async_setup_entry, the extra code and maintenance is very low and would allow to add YAML with minimal changes.
What was not the objective
This solution does not represent an end to end solution since PS4 does not require a lot of discovery of devices, OAuth flows or refresh tokens. Further experiments should explore these other areas and find a solution.
What was required to make it work
The changes are very minimal and explained below.
Do note that as this was distributed as a custom_components, we also imported config_flow, translations, services.yaml, manifest.json, etc. This is done in order to keep support for the UI and the custom component working perfectly. However, in the case of a core component this would not have been part of the implementation.
1. Define schema
In order to be able to validate the schema, it is required to define it first. While this is optional, it provides a lot of functionality for very little lines.
import voluptuous
from homeassistant.helpers import config_validation
PLATFORM_SCHEMA = config_validation.PLATFORM_SCHEMA.extend({
voluptuous.Required(const.CONF_HOST): config_validation.string,
voluptuous.Optional(const.CONF_NAME): config_validation.string,
voluptuous.Required(const.CONF_REGION): config_validation.string,
voluptuous.Required(const.CONF_TOKEN): config_validation.string,
})
This defines the 4 fields that are required to run a PS4 media player. With this code, a new entry can be created using the usual configuration:
media_player:
- platform: ps4
name: "PS4"
token: !secret ps4_token
host: !secret ps4_wifi_ip
region: "Singapore"`
2. Create an async_setup_platform (setup_platform)
The only task that async_setup_platform performs is adapting the data that comes from the configuration.yaml to the required input of async_setup_entry.
The schema that the device requires is formatted like this:
config_entry: {
entry_id: string # Unique entry id. You can use uuid or simply entity id as it's unique.
data: { # Contains devices and platform information.
token: string # PS4 token obtained from the app.
devices [ # List of devices.
{
host: string # IP of the PS4.
name: string # Name of the PS4.
region: string # Region of the PS4.
}
]
}
}
The only condition is that entry_id and data are accessed like attributes of a class (i.e. config_entry.entry_id or config_entry.data), whereas all the data inside config_entry.data is shaped like a dictionary (i.e. config_entry.data['token'] or config_entry.data.get('devices')).
The code that was needed to transform from our configuration to config_entry is like this:
import collections
Config = collections.namedtuple(
'Config', f'{const.CONF_ENTRY_ID} {const.CONF_DATA}')
async def async_setup_platform(
hass, config, async_add_entities, discovery_info=None):
"""Loads configuration and delegates to official integration."""
# Load configuration.yaml
host = config.get(const.CONF_HOST)
name = config.get(const.CONF_NAME, const.DEFAULT_NAME)
region = config.get(const.CONF_REGION)
token = config.get(const.CONF_TOKEN)
# Format it in the new config_entry.yaml format
config_entry = Config(
util.slugify(name),
{
const.CONF_TOKEN: token,
const.CONF_DEVICES: [
{
const.CONF_HOST: host,
const.CONF_NAME: name,
const.CONF_REGION: region,
},
],
}
)
await ps4_media_player.async_setup_entry(
hass, config_entry, async_add_entities)
return True
The method async_setup_entry uses a different format of config_entry than what the configuration.yaml provides. As such, we used a namedtuple as a proxy for a very simple class that allows us to access fields like config_entry.data as it’s done in the PS4 entity.
This currently supports only one device, but you could just keep appending the data to devices if using the same token or create a full new Config if it comes from a different platform.
3. Documentation
When manual configuration is available, the last step would be adding some documentation. In this case, we advocate to leave very minimal configuration and rely on the UI flow to generate the details if ever needed. This way, documentation does not become a burden.
Further experiments can explore how the UI can become a source of data for configuration.yaml without having to create advanced documentation.
4. Delegating all the other code
For those of you reading this with the intention of creating similar custom_components, I want to add a bit of explanation on how to achieve it. This is not part of the experiment, but of the custom_components logic and it would not be needed if the rest of the code was integrated in the core component.
You need to copy services.yaml and manifest.json to describe your services. This will allow the platform to know all the services that are required by your system.
Do not copy the whole component, you can easily delegate the implementation by doing this:
from homeassistant.components.ps4 import media_player as ps4_media_player
async def async_setup_entry(hass, config_entry, async_add_entities):
await ps4_media_player.async_setup_entry(
hass, config_entry, async_add_entities)
This way, any changes on the core component will not affect your component and you will be able to take advantage of them by simply maintaining the Adapter logic.
If you want to support the UI flow as well, you can simply use the same mechanism to delegate it:
from homeassistant import config_entries
from homeassistant.components.ps4 import config_flow
@config_entries.HANDLERS.register(const.DOMAIN)
class PlayStation4FlowHandler(config_flow.PlayStation4FlowHandler):
def __init__(self):
super().__init__()
This will implement the same UI flow that the core component has without having to do any coding. In other experiments, I will showcase a better way to improve the UI flow so it also spits out the required code for configuration.yaml, but for now this is a good sample start.
If you do this, you also want to import the translations folder so the messages show. In this experiments, I copied both manually, but it might best to have a way to copy the core folder to your custom_components automatically. This could be showcased in another experiments, but the main focus is to show how to minimize the core components support for YAML.
As for the constants, in my case I decided to create my own const.py to have flexibility, but virtually all the constants are coming from core constants or ps4 constants really:
# Platform constants.
DOMAIN = 'ps4'
PLATFORMS = ['media_player']
# Attributes.
CONF_HOST = const.CONF_HOST
CONF_NAME = const.CONF_NAME
CONF_REGION = const.CONF_REGION
CONF_TOKEN = const.CONF_TOKEN
# Secondary data attributes.
CONF_DATA = const.CONF_SERVICE_DATA
CONF_DEVICES = const.CONF_DEVICES
CONF_ENTRY_ID = 'entry_id'
# Services.
SERVICE_SEND_COMMAND = 'send_command'
# Service attributes.
ATTR_ENTITY_ID = const.ATTR_ENTITY_ID
ATTR_COMMAND = const.ATTR_COMMAND
# Default values.
DEFAULT_NAME = 'PlayStation'
4 Likes
During the weekend, I tried a different experiment. I used Hue instead of Cast because Cast uses Discovery instead of the UI flows.
The objective of this experiment was to showcase how easy would it be to show the device information in the UI before writing the data. The ultimate objective is to show that the UI can be a good way to onboard the component, that would reduce the cost of maintenance of documentation; while still providing an option for YAML Components. As always, the conclusions are at the end and you can navigate through the titles.
I am compiling all experiments on this component, and it’s open to anyone who is working on similar objectives.
I want to send this experiment (or a variant) to ADR, so all feedback is welcomed (positive or negative).
Hue Experiment
This is an experiment to showcase how UI flow can act as a self-documenting feature for YAML configuration.
Objectives
What was the objective
To showcase that the UI configuration flow can be used to generate the same data needed for the configuration.yaml and allow the user to take the last decision on the configuration.
The underlying objective is to prove that the same config_flow can be used to set up both UI and YAML configuration without adding significant cost to the component owner.
What was not the objective
To implement the code required to read YAML configuration. This is why we focused on Hue component which still supports YAML configuration.
This experiment is also not intended to be a custom_component as the original core component has all the required documentation and features, but it could be used in the future. Nonetheless, this would work if it gets installed as a custom_component.
What was required to make it work
The only change required is to add an additional setup before async_create_entry that confirms the output with the user for validation.
0. Understanding the original device creation
The method async_step_link in the hue core component is responsible for getting the bridge information and creating the entry device. This is the logic responsible for this:
async def async_step_link(self, user_input=None):
# ...
return self.async_create_entry(
title=bridge.config.name,
data={
"host": bridge.host,
"username": bridge.username,
CONF_ALLOW_HUE_GROUPS: False,
},
)
# ...
However, we wanted to add an additional step to ensure that we could validate and read the configuration. As such, we have replaced this logic with a few changes described below.
1. Store device data into a class attribute
Instead of directly creating the entry, we store the data required into a class attribute, which will allow us to use it later to finally create the entry.
self._set_up_data = {
'title': title,
'data': {
'host': host,
'username': username,
'allow_hue_groups': allow_hue_groups,
}
}
Note: while it was not required, for this experiment, we have stored the original values into variables as we will be using them in two different places on this implementation.
title = bridge.config.name
host = bridge.host
username = bridge.username
allow_hue_groups = False
2. Format data into YAML Configuration format
Change the format from config_entry (_set_up_data) to the one required by YAML configuration.
import yaml
yaml_data = {
'hue': {
'bridges': [{
'host': host,
'allow_hue_groups': allow_hue_groups,
}]
}
}
yaml_configuration = yaml.dump(yaml_data)
Note:
yaml_datais only required because the data required byasync_create_entryand YAML configuration are different. We advocate to have an equivalent setup so there is no need to maintain two different schemas. If this was true, this would only have required this line:yaml_configuration = yaml.dump(self._set_up_data)
3. Show data before creating the device
We need to edit the original async_step_link to make a call to a new step before calling async_create_entry. This is done by passing the previous data into a form like this:
return self.async_show_form(
step_id="confirmation",
description_placeholders={
'yaml': yaml_configuration,
}
)
Note: this logic replaces the previously described
return self.async_create_entry(...).
4. Add additional step to show YAML configuration
When the form is executed, it triggers a form which will show the YAML data. Once the user submits the form, it will create the entry. The code responsible for this:
async def async_step_confirmation(self, user_input=None):
"""Creates device entries after confirmation."""
if not self._set_up_data:
raise ValueError('Configuration flow failed.')
return self.async_create_entry(
title=self._set_up_data.get('title'),
data=self._set_up_data.get('data'),
)
As a new step is created, you need to also add some translations to show the message:
"confirmation": {
"title": "Confirm Configuration",
"description": "Your device configuration is ready. \n\nIf you prefer to set up the device manually, use this code on your `configuration.yaml`:\n\n```yaml\n{yaml}```\n\nIf you prefer to set it up via UI, simply click submit below."
}
Summary
While the boilerplate may seem long, by sharing the same configuration structure between YAML and UI, this would only require to make one self.async_create_entry() with a yaml processed details of the device to really make it work. This should not require more than 5-10 lines of code (including English strings) and provide great functionality.
Result
This will show the following message in the UI:
The user can copy the code into configuration.yaml or simply press submit to complete the UI setup.
Alternative implementations
This implementation may not look friendly for UI-only users. There are a few ways to improve this.
Optional YAML configuration
One option is to only showcase the YAML configuration to those who are YAML configuration. This could be a user flag that allows us to know if it’s an advanced user or not. This is similar to enabling/disabling lovelace dashboards or showing the Developer Tools.
The code will be something like this:
if is_advanced_user:
return self.async_show_form(
step_id="confirmation",
description_placeholders={
'yaml': yaml_configuration,
}
)
else:
return self.async_step_confirmation()
Make configuration editable
Another option is to additionally show the configuration data to the user, so they can Edit it. This is not a replacement for YAML configuration, but a way to make this confirmation box also users by UI-only users.
This would require defining the form data via a Schema:
data_schema = voluptuous.Schema({
voluptuous.Required("title", default=title): str,
voluptuous.Required("host", default=host): str,
voluptuous.Required("username", default=username): str,
voluptuous.Required(
"allow_hue_groups", default=allow_hue_groups): str,
})
Additionally, the form call would change slightly:
return self.async_show_form(
step_id="confirmation",
data_schema=data_schema,
)
As the form has changed, the translation text would also be different:
"confirmation": {
"title": "Confirm Configuration",
"description": "This is the data configuration that will be set up.",
"data": {
"title": "Title",
"host": "host",
"username": "username",
"allow_hue_groups": "allow_hue_groups"
}
}
The form data could be gathered by the confirmation step from the user_input instead.
async def async_step_confirmation(self, user_input=None):
"""Creates device entries after confirmation."""
config_title = user_input.get('title')
config_data = {
'host': user_input.get('host'),
'username': user_input.get('username'),
'allow_hue_groups': user_input.get('allow_hue_groups')
}
return self.async_create_entry(
title=config_title,
data=config_data,
title=self._set_up_data.get('title'),
data=self._set_up_data.get('data'),
)
At the end, this would show like this:
The form data could also be combined with the one from YAML configuration to offer the best of both worlds.
Overall, this alternative implementation empowers users to confirm the configuration before creating devices, and can be a good step to also provide the data needed for YAML configuration in a more user-friendly way. The extra schema seems to add to the cost (despite being exactly the one required to be stored), and as such, we have opted for a more lightweight option for the main proposal.
Centralized confirmation on core code
This code was implemented by a custom_component and, as such, it has focused in modifying a core component to add this additional step. However, a more efficient way to implement this would be by implementing this natively on self.async_create_entry under core/data_entry_flow.py. Any data received to create a new entry will be shown to the user in a form who would need to submit before creating the entry. This works best when the configuration is optional and only shown for advanced users.
This effectively would have no cost for the components developers.
Conclusions
For Component Owners
The same UI flow can be used to generate data to both manual YAML configurations as well as UI configurations. This will allow to ease the need to document the source of the configuration.yaml parameters. This reduces the cost for component owners to provide YAML configuration.
For UI Users
For users who prefer to use the UI, there is no changes in their current workflows. At worst, the may have one additional step where they see the configuration. However, this can also be optionally shown only to user who enable “advanced” mode; similar to how some Configuration are shown only to some users. The config_flow allows to do this type of conditional send to one step or another.
For YAML Users
Users who want to maintain components via YAML are encouraged to use the UI to set up if they are not sure how the data should be filled in. However, once the data is filled in, the users can edit it or create new ones using the same structure.
7 Likes
There would also need to be some function to reload the integration upon reconfiguration as well to get the same functionality from YAML configuration as the config flow.
Hi @firstof9, thanks a lot for the feedback 
One clarification: this code does not actually write to the configuration.yaml file. As Paulus described, this would have problems like potentially breaking secrets, includes, etc.
As such, it just shows you the configuration information in case you want to opt-in for manual YAML configuration. You would need to copy/paste the YAML snippet there.
In YAML mode, edits are done Editing the configuration.yaml file, not via the UI. I might be wrong, but as far as I know these YAML configurations do not show up under Integrations and therefore support edit. If they do show up, then yes, when Editing you would also want to show a confirmation step before, but I am not aware of how this works at the moment.
1 Like
I know, I wasn’t implying this just merely asking how a reload would be handled. Currently via the config flow you can have to unload and then load the entities again. In YAML mode you’d have to restart Home Assistant completely, while not a show stopper, I like this functionality myself as it leaves less gaps in my sensor data.
Makes sense. Thank you for clarifying. If I understand correctly, it would be something like “Reload Automations” but for “Reload Components or Integrations”, is that correct? i.e. Similar to this feature request.
Right but currently only works for config flow integrations. So additional core would would need to be done, as is my understanding, to get the YAML configuration to be able to be “reloaded”.
I could be wrong here but that’s my assumption 
The difference is that those are in their own included files. A new engine would be required to just load partial configs from full files with multiple configs.
Actually I think they’re reloaded by domains in that case, since you can have automation & scripts spread across packages as well.
I could be wrong, but that’s how I understood it. Since they’re not really working like a component ie: no requirements, no external polling etc.
Packages are loaded differently if you follow the code. Everything loads from a single file at first and separated after.
1 Like
I’d love templates to have a reload function as well.
2 Likes
Ugh, don’t we all!
2 Likes
That’s a really great question, @hmoffatt. There are a few angles from which we can explore it. I should probably also do an experiment that touches on this as well. Thanks for the idea.
For now, this is my view of the situation:
Code Changes
First and foremost, Developers already have to update schemas, regardless of whether this works on YAML and/or the UI entities. The schema will change on the set up as well as how it’s passed to the DeviceEntity most likely.
With this proposed changed, the only additional change is to display that new data. (i.e. changing step 2 on the experiment). However, as stated before, what I would advocate is to make sure that the UI entities and YAML configuration match hence there are no additional changes needed, the output for future changes will already be fully updated.
The most important part here is that it would break for users. Hence, that’s where we should focus and expand a bit more.
Option 1: YAML is expected to break
The first way to look at it, and my personal preference, is that if there’s a schema change, YAML is expected to break. As such, the user is responsible for updating the configuration.yaml file. You can do that by going through the UI, for example, or reading an update on the changes as we have done until today (and, as a user, this has not been very frequent in my experience).
If this is not your preferred way, then you should probably stick to UI entities. I think this is an expectation if you are a tinkerer and want your YAML power.
Option 2: Piggyback on the versioning that UI entities use
Your question not only applies to YAML entities, but also to UI; hence, the first question is how UI handles it.
Let’s see at this code on the PS4 component from my previous experiment. The code responsible for the change of schema is async_migrate_entity. What this code does is read a parameter called “version” and change the schema from version 1 to version 2, from version 2 to 3, etc. At the moment we are on version 3. This data gets stored with the entity configuration so you can migrate from v1 to v3.
This very same logic can be applied to YAML. We could potentially add the version to the code (using Hue here instead of PS4, but it could be the same):
hue:
version: 1
bridges:
- allow_hue_groups: false
- host: 192.168.1.105
As this YAML should match the same as the UI config, it can use the same flow to input version 1 and output 2. The only complication here would be merging data when tokens and alike are present, but this is a different problem and I intend to also look at this a at a future experiment.
Caveats
There are still issues when new data is required (unless default values are possible), but I think that would be the same for UI configured entities and the migration flow should handle that too.
1 Like
You’re welcome to continue discussing this, but at this point it’s probably best if you make your own thread. Feel free to link to it from here.
I’m not sure why this should be split. All these messages are a request to bring back YAML, arguments against the original ADR-0010 and proofs of concepts that showcase the arguments.
This is not about custom components. I’ve just used custom components as a way to prove those points.
I feel like we are trying to silence the original blog posts making it seem like there really are no alternatives, and no more discussion to be made. And it’s a bit disappointing to see that on an open source community.
1 Like