WTH are keys in yaml files (re-)sorted alphabetically by UI editors?

Archive Month of "What the heck?!" (2020)
3

Haven’t looked at the underlying storage but even if it’s processed as JSON, I guess the keys could be sorted using a custom order prior to saving as YAML?

4

as I said order in json is not preserved and not guaranteed. Parsers likely sorts them somehow (alphabetically?) for performance reasons.

5

I personally don’t care so much about the original order, I’m fine with the keys being resorted, as long as it is in a reasonable order.

This could be something like main key (e.g., trigger for trigger section, condition for condition section, and service for action section) or main keys (it could be several - my list may not be exhaustive) first, and then the rest in alphabetical order

6

Where?

Automation Editor stores automations in automations.yaml in YAML.

If it’s being converted to JSON for processing, it’s peculiarly selective about how it converts the JSON back to YAML: it doesn’t sort the trigger, condition, and action keys, only the keys they contain.

Interestingly, every online YAML to JSON converter (and the opposite direction) can perform the task without sorting the keys.

2 Likes
7

Maybe I’m wrong.
My answer is based on how it does work in case of lovelace (in GUI mode). Because OP describes very similar behaviour I thought it might be the same root cause.

8

Anything core brings in from the configs is parsed with pyYAML, which produces an OrderedDict upon parse (or an empty OrderedDict if the config file is empty). This applies to automations.yaml, configuration.yaml, secrets.yaml, scenes.yaml, and scripts.yaml (should they exist).

When writing the files back to the file system, the OrderedDict that was created during parsing is transformed back to YAML and then written out. That’s why automations get written in an ordered fashion.

AFAIK, none of these base configuration files gets parsed to JSON at all. I do remember that there was talk about moving over to JSON eventually, but I haven’t seen much movement towards that in the source files.

9

Similar to what? The OP is discussing the UI-based Automation Editor which stores everything it reads in sorted YAML format. So if you use it to open an unsorted YAML automation, it will save it as sorted YAML (and discard comments).

This is destructive behavior and means the Automation Editor should never be used to modify automations that were created in a text editor. However, by default,others is no safeguard in place to prevent it. Many people, including me, discover it empirically (try the Automation Editor then discover it has mangled all the automations created with a text editor).

The simplest way to prevent it is to separate the automations created via text editor from those created via the Automation Editor. The documentation describes how to do it and it involves creating a directory to hold one’s ‘manually created’ automations. Then you add an option to the config file to indicate where the ‘manual automations’ are located.

However, this only prevents damage to ‘manual automations’ and doesn’t address the issue of Automation Editor producing sorted YAML.

1 Like
10

I’d like to learn more about this OrderedDict behaves because, as I mentioned previously, it’s rather specific about how what it does and does not ‘order’. Here’s a simple automation I just created using the Automation Editor (0.114.1):

- id: '1598274610865'
  alias: Turn on the lights when the sun is set
  description: ''
  trigger:
  - event: sunset
    platform: sun
  condition: []
  action:
  - device_id: cb08dbe923044ce59f4f90e013ff98b7
    domain: light
    entity_id: light.family_ceiling
    type: turn_on
  mode: single

The first-level keys are not ordered, only the second-level keys. If everything was sorted, it would look like this “dog’s breakfast”:

- action:
    device_id: cb08dbe923044ce59f4f90e013ff98b7
    domain: light
    entity_id: light.family_ceiling
    type: turn_on
  alias: Turn on the lights when the sun is set
  condition: []
  description: ''
  id: '1598274610865'
  mode: single
  trigger:
  - event: sunset
    platform: sun

This implies there is some selective mechanism available for sorting.

EDIT

OrderedDict

Return an instance of a dict subclass, supporting the usual dict methods. An OrderedDict is a dict that remembers the order that keys were first inserted. If a new entry overwrites an existing entry, the original insertion position is left unchanged. Deleting an entry and reinserting it will move it to the end.

Maybe I’m guilty of confirmation bias but that description suggests to me that OrderedDict isn’t responsible for sorting the keys.

1 Like
11

You’re absolutely right. I haven’t directly used OrderedDict in a long time (I much prefer OmegaConf for all my dict management :wink: )

In researching a bit more, I’m wondering if they are sorting deeper in the code base as that would support your examples.

I’m pretty sure that there’s something more going on when the automation editor writes the file back based upon what you demonstrated. My thought is that they are parsing the config file upon load and then looping over it using something like for key, value in [dict] and using that same loop with an added .sort() for the child objects. So, the top level keys don’t get reordered, but the child items do.

I’ll dig deeper into the code base and see if I can find anything.

1 Like
12

Trying to fix this:

9 Likes
13

LOL That’s exactly what I was looking at in dumper.py. It looks like pyYAML added the sort_keys=False (issue: https://github.com/yaml/pyyaml/pull/143) last year.

14

Any chance of support for JSONC?

JSON with comments

Or would that first require a discussion in the Architecture repo?

1 Like
15

For where, we only use YAML?

16

For reading/writing automations (with the Automation Editor).

If an automations.yaml file contains comments, writing to it with Automation Editor purges all the comments. Understandable, because JSON doesn’t support comments, but destructive behavior nonetheless.

17

Thanks for looking into this so quickly, Bram!

18

The same happens when you edit Lovelace.
Maybe the same fix can be applied to Lovelace too?

1 Like
19

I also applied it to storage for lovelace

2 Likes
20

JSON with comments is not supported by a browser, so that won’t work. There probably are other ways to make it work, but that is not a simple fix.

21

The process now converts YAML into JSON which strips away comments. When it converts back to YAML, it has no comments to write.

The suggestion is to convert from YAML to JSONC in order to preserve the comments. When it converts back, it has comments available to write.

Anyway, stopping it from sorting is already a big step in the right direction.

EDIT

Not seeing the comments displayed in the browser is an acceptable price for not having them silently deleted.

22

Where does it do that? I’ve been poking around the code, but I don’t see anywhere that it converts YAML to JSON. Is it in frontend?