The future of YAML

Blog
1

Recently, comments and questions in the community have been posted, about the announcement of integrations becoming available via the frontend and the removal of YAML support from some of them.

This raises questions and opinions that need addressing. It mainly comes down to this single question:

“Is YAML going away?”

The answer to this question is: “No!, but…”

Realizing you may not like that loose answer and, in that case, you are possibly looking for a hard and simple “NO” answer to this question. However, things are a bit more complicated.

This blog post aims to bring more clarity to this question, so everybody knows why things are the way they are right now and what to expect for the future of configuring Home Assistant.

Enabling configuration via the UI

Before talking about YAML, let’s explain the reasoning behind all the UI features introduced.

The Home Assistant project has grown at a ludicrous rate in the last few years. Being one of the few solutions to provide a local home automation platform, that puts privacy first, has gathered interest from all kinds of people on this planet.

In the beginning, Home Assistant was a small project; aimed at developers and the tech-savvy. With the growth of privacy concerns, but also the growth in the availability of IoT devices in our homes in general, the project gained traction; attracting lots of new community members that are not always the tech-savvy types.

This is great, but that also means we need to adjust, to make it usable for everybody. We want everybody to enjoy what we all have been enjoying already. No matter your level of experience.

Considering that, enabling and empowering people with managing their Home Assistant instance via the user interface is needed. Not just for the lesser tech-savvy, since it also brings convenience, that even many technically capable users enjoy and prefer.

Goal: Making things easier.

The goal of making it easier is currently the main focus.

Breaking changes

We all dislike them, those breaking changes when a new release is published. We all scan the release notes hoping you are not affected by any of them in that release.

There is a big upside on using configuration via the UI: Home Assistant manages that configuration part for you and handles the upgrades and migrations. Virtually removing the chances of hitting a breaking change, we all hate just as much.

Shareability & Security

This is a screenshot of one of the slides, showed during the State of the Union 2019:

![](upload://iHJAcaCW3hfYzi4cE6DSGgx8nBb.png) Presentation slide showed during the State of the Union 2019.

We all like to share our experiences, and thus parts of our configurations. GitHub is full of repositories that shares people’s homes.

This is great! Sharing ideas, providing inspiration for all of us!

There is a great downside to this, which is privacy and security. There are a lot of things we don’t want to share. For example, our passwords, sensitive (personal) information or (historical) data.

A bit more technical, but integrations using OAuth2 as the authentication method to integrate with, for example, a service like Somfy don’t need to store the username/password at all.

Both YAML & UI

Home Assistant is moving towards a better separation of YAML configuration versus configuration done in the UI. This is already partly showing in things like zones, Lovelace and the recently introduced “Helpers” (also known as the input_* integrations in YAML).

Zones can be configured via the UI and via YAML (even at the same time!). Both configuration sources will be loaded by Home Assistant when considering the configured zones and both can be changed on the fly without restarting Home Assistant.

For the future, Home Assistant will be moving towards that concept more and more. This allows anybody to use the method they prefer.

In these cases, YAML support has been extended and improved by adding features like reloading and by removing parts that mess with your YAML files.

We are still an open-source project

Home Assistant is an open-source project and relies on contributors adding those amazing integrations, maintaining them and extending their functionality to make them even more capable.

Those contributors do this in their free spare time, for which we all are eternally grateful. It is their work that enables Home Assistant to do what it can right now. It is what automates your home.

So what about integrations that remove YAML support?

Some contributors have decided to remove the YAML support to reduce their maintenance and support burden. The amount of energy that needs to be put in (to maintain both capabilities) can be too much and is complex. We have to understand and accept that. If we do not do that, a contributor could simply stop contributing.

Unfortunately, such a move creates breaking changes and often leads to a few pretty de-motivating comments, towards the contributor and the project in general. This is harmful to everybody, as the contributors get demotivated or, even worse, don’t want to implement new features or create a breaking change.

This halts our project goals, slows down innovation and induces the risk of losing contributors and maintainers. In the end, leads to a greater loss for everybody.

The future of YAML

So, with all of the above set context: What is the future of YAML?

As of today, ADR-0010 (Architectural Decision Record) has been approved that states:

  • Any new integration that communicates with devices and/or services, must use configuration via the UI. Configuration via YAML is only allowed in very rare cases, which will be determined on a case by case basis.
  • Existing integrations that communicate with devices and/or services, are allowed and encouraged to implement configuration via the UI and remove YAML support.
  • We will no longer accept any changes to the YAML configuration for existing integrations that communicate with devices and/or services.

So what stays available in YAML in the end?

All other integrations that do not communicate with a device and/or service, are configured via YAML or via Storage Collections (these enable both YAML and UI capabilities used by, e.g., Lovelace and zones). Examples of these other integrations:

  • Integrations that integrate transports. These integrations allow users to define their own protocol, e.g., MQTT, serial, GPIO.
  • Integrations that process Home Assistant data and make this available to other integrations, e.g., template, stats, derivative, utility meter.
  • Integrations that provide automations, e.g., automation, device_sun_light_trigger, alert.
  • Integrations that help controlling devices and services, e.g., script, scene.
  • Integrations that expose Home Assistant data to other services, e.g., Google Assistant, HomeKit.

This ADR is set to remove confusion and questions for everybody, builds upon the goals that have been set out (as presented during the State of the Union), and sets the guidelines for our contributors to work with. This ADR brings clarity for us all.

Myth-busting

In the raised concerns and comments across the community, some comments have been found multiple times. Please note, these are not exact quotes, as we don’t want to address anybody personally.

  • “Making backups became harder!”

    Using the Home Assistant snapshot feature, this is not an issue. However, if you do manual backups on a system that runs just Core, you need to make sure to back up the .storage folder as well (which hopefully you’re already doing). Otherwise, there is no difference.

  • “It is harder to test my configuration if integration does not support YAML”.

    YAML configuration testing is often done to see if a specific YAML configuration is still valid against (newer versions of) Home Assistant. With integrations set up via the UI, this is not a concern, since Home Assistant ensures the data structure is compatible between versions and migrates it for you.

  • “I like to use a private git repository where I store all my data in, including my sensitive data, since it is not public. Without YAML this is not possible.”

    This is actually not true, the .storage folder contains all Home Assistant managed configuration files in JSON format, which in those cases, can be stored and versioned in a git repository.

Is YAML going away?

No!

Not for the things we like to share publicly. It will also remain available for the advanced features we currently cannot provide a UI for. The goal is not phasing out YAML, the goal is to make the best home automation platform in the world that is easy to use for everybody. Enabling users of all experience levels, to enjoy this wonderful hobby we all share and allowing everyone to focus on what matters most: automating our homes.

Closing with the YouTube recording of the State of the Union 2019, starting at the part discussing this.

This is a companion discussion topic for the original entry at https://www.home-assistant.io/blog/2020/04/14/the-future-of-yaml/
19 Likes
2

What about tools for migrating away from YAML? I’d love to use the new UI based zones editor, but I don’t feel like spending my time manually entering all my 35 YAML zones into the editor because at that point, the zone editor is creating more work than it reduces. Why not allow a one time import to switch over to the UI?

Also, I don’t feel that just adding .storage into a Git repository really solves the primary use cases of those of us using Git repositories

These use cases for me are the following:

  1. Sharing. I like to share parts of my config with others, and with YAML, I can easily show exactly what I’ve created to others. I don’t believe .storage provides the same benefits. What is the strategy for sharing configurations with others in a UI based world?

  2. Revision tracking. With YAML, I can easily go back and look as far as I want and see how I was doing something previously in a readable and understandable format. So if I want to go roll back just that part because it worked better or because I broke something, it’s an easy copy paste to get it back. The same is true of if I break something and hass won’t start. I can fix the config without hass needing to be running first. Putting things in the UI increases the chances that something will get broken that can’t be fixed because we have the chicken and egg scenario that hass needs to be running to fix it but it won’t start up.

Even as a backup mechanism in a git repo, .storage is pretty much useless, because I can’t really just copy part of it back into my active .storage folder and expect it to work. Restoring is an all or nothing proposition, so it provides no benefit over supervisor snapshots. If one fails, the other almost definitely will too. That was not the case with YAML, where I could copy only part of my config back over.

9 Likes
3

I’m not sure you’ve read the whole thing. Automations, templates, scenes, scripts…those are all shareable and yaml.

Sharing how you added an allergy sensor to your HA instance is not needed.

5 Likes
4

Guess we all just better get used to editing unreadable JSON files under .storage versus YAML. And /or watching a lot of click here, click there Youtube videoes to figure out how to do something that could have been described in a few lines on the forum…

13 Likes
5

I read the whole thing, but I came away with the impression that things like automations and scripts are still being allowed to exist in YAML, but it was not stated that that would be the primary method encouraged to be used in the future. Also, what about sensors? Some of my most complex and interesting customizations are sensors.

3 Likes
6

template sensors will stay in YAML I think, they want to add it to the UI too but more like how they did timers and input_* so hopefully in the future it means you won’t need to restart to add a template sensor :slight_smile:

7

If you use the UI editor for an automation it generates the YAML in the background. That YAML can still be edited by hand and can still be viewed / shared as needed.

As for sensors, is there even a GUI way to edit them today? I believe they would fall into one of these categories from the original post:

Integrations that integrate transports. These integrations allow users to define their own protocol, e.g., MQTT, serial, GPIO.
Integrations that process Home Assistant data and make this available to other integrations, e.g., template, stats, derivative, utility meter.
Integrations that provide automations, e.g., automation, device_sun_light_trigger, alert.
Integrations that help controlling devices and services, e.g., script, scene.
Integrations that expose Home Assistant data to other services, e.g., Google Assistant, HomeKit.

To me, the biggest take away is that configuration of integrations will be via the GUI (which is, frankly, better) and eveything else will still be YAML. Please help me to understand why this is an issue (and I’m asking honestly because I keep seeing posts about this and no one really had a solid reason).

6 Likes
8

What are you trying to edit in .storage?

I’ve moved many of my integrations from the YAML setup to the GUI setup and in all cases it was easier in the GUI. Are you seeing a place where it’s harder / less intuitive to setup via the GUI? If so, where?

2 Likes
9

No, but sharing the fact that you DO have an allergy sensor was useful.

I learned everything about homeassistant from others’ githubs, and would look and say to myself “hmm, where does sensor.xyz come from?” and then look through the rest of the configuration and find out what integration it was. Even inspired me on what hardware to buy.

I then shared my configuration with the same level of detail, and people could look at my lighting package and see what integrations I used and what automations controlled them.

My github then basically became a load of automations only as the configs moved to UI. I started getting messages saying “I’ve copied this and it doesn’t work…” and I had to reply with instructions for them to add the integrations that the automations depended on, whereas before they could have seen it in the same file they got the automation from.

I got fed up with it in the end and binned my github repo. I still get people begging me to put it back up, but I’m not happy to do so until I can think of a way of somehow maintaining a list of what things rely on a UI configured integration without making a frick load of extra work for myself, as unfortunately I have a lot of other things irl that take up my time too.

And as for

That’s not strictly true is it? Everyone driving these changes is on the payroll, and thus they aren’t providing these contributions for free, they’re being paid. All the ‘old school’ big names who were staunch yaml lovers haven’t been taken on as ‘staff’ and haven’t committed to the project in a long while. Presumably politics, but it’s weird to hear the “we do this for free” argument from people who are getting paid for their time.

26 Likes
10

Having been ‘one of those people’ (see for example the 0.108 release thread, I got many likes!), I must say I feel much better having thought about it a few days. I meant to post in the 0.108 thead but this presents a more opportune place.

Leaving scenes/automations/scripts available to use yaml, as well as still being able to do templating and commandline and utility integrations via yaml makee me feel much better about the direction this is taking.

Cheers all.

10 Likes
11

If I’ve understood it correctly, this is the summary:

YAML will continue to be an available means of configuring most everything except for integrations that automatically discover their devices.

That suits me fine.

The only thing I would like to see is the Automation Editor storing things in human-friendly YAML as opposed to the unfriendly YAML it currently produces (sorted and compressed).

17 Likes
12

I think this is a well balanced solution. Thanks devs.

2 Likes
13

I think this is the most well though out reply that I’ve seen and I think this actually makes pretty good sense. I suppose you would lose some of the context of the automation YAML if you weren’t aware of the integration and it is certainly less informative from a “what to buy” standpoint.

I wonder if there might be a way around these issues (I’m certainly not the person to come up with one but it’s an interesting thought experiment).

1 Like
15

Out of curiosity, why was JSON chosen for Storage Collections instead of YAML?

They’re functionally interchangeable but YAML is easier for people to read. Or is that why JSON was chosen … (an implied “Don’t Touch”).

14 Likes
16

That is the most succint statement you will ever see describing this situation right there…sums it up clearly in as few words as possible in my belief!

4 Likes
17

Well at least we aren’t having xml foisted on us.!

4 Likes
18

I avoid integrations unless it is the only option. They have not been reliable for me across upgrades or even sometimes reboots. One of my upgrade techniques depending on how long it has been since I have upgraded has been to just resort to wiping out .storage completely and reconfigure those few things I have integrations for. If HASS doesn’t come back up due to an integration issue, I don’t know what else to do. And then there are times I reboot and something disappears because it wasn’t communicating when discovery was run. Or it prompts me to re-auth stuff again even though it is still working.

As far as the JSON thing, I have only started exploring that, but if that is the only means I have to deterministically configure things then that is probably what i will do.

I mean I get it, trying to make it easier for people because YAML is a sucky space delimited syntax. And more people are going to find a GUI easier. And you can configure the GUI on any device, etc. But JSON and YAML are easily convertible, so I don’t get why you can’t have a GUI and YAML.

8 Likes
19

I’m stuck at the part where the representation of the config data is YAML and entered with a text editor versus the config data in JSON and entered with a fixed UI specific for the use cases. Talking UI vs YAML is chalk and cheese. Why not just have the UI spit out YAML for non-sensitive config, and to a different file/format for the Personal Identifiable Information?

9 Likes
20

Breaking changes

We all dislike them, those breaking changes when a new release is published. We all scan the release notes hoping you are not affected by any of them in that release.

There is a big upside on using configuration via the UI: Home Assistant manages that configuration part for you and handles the upgrades and migrations. Virtually removing the chances of hitting a breaking change, we all hate just as much.

its sad that for the UI config there has been taken action to automaticly check and migrate, and that that same action wasnt taken long ago for yaml.
i wouldnt have stopped updating if that was the case.
its still possible by the way, but probably will never happen.

3 Likes
21

I think it’s just a matter of including a list of devices and integrations that you use on your github page if that’s what you’re in to. Users do it already with lots of nice images and everything. The dev tools > info page even has a list of integrations in used to be copied. It could be made better and more easily copied.

3 Likes