YAML/JINJA attribute assignment

Configuration
1

I would like to understand the JINJA interpreter logic and YAML rules that result in the following behavior.

Apparently, this is fine:

attr1: 0.0

0.0 yields a float value which is assigned to attribute "attr1". Which in YAML is really just another object.

This:

        attr1: >
          {{ iif (trigger.id=='end',
            states('sensor.p1_meter_energy_import') | float(0) - (this.attributes.get('start') | float(0)),
            this.attributes.get('day0') | float(0))
          }}

is also fine. The expression within the double braces yields a float value which is output to the YAML.

This still makes sense to me, it is expected.

This however:

attr1: {{ this.attributes.get('attr2') | float(0) }}

is not accepted. The configuration check says "invalid key", seemingly refering to 'attr2' which does exist on the this object.

It needs to be:

attr1: "{{ this.attributes.get('attr2') | float(0) }}"

Then all is well.

Why?

If I wrap the second example in double quotes, the quotes are included in the attribute value (I get a string rather than a float). That makes sense. In the last example though, the quoted curly braces do not result in a string, I get a float, which was the idea, but strikes me as odd.

Can anyone explain this on a parser/interpreter level?

2

You need to surround the template quotes if it is on a single line.
The result will be a float (not a string).

3

Oh that's just great. So

day1: "{{ this.attributes.get(iif(trigger.id=='start','day0','day1')) | float(0) }}"

yields a float,

day1: >
  {{ this.attributes.get(iif(trigger.id=='start','day0','day1')) | float(0) }}

also yields a float and

day1: >
  "{{ this.attributes.get(iif(trigger.id=='start','day0','day1')) | float(0) }}"

yields a string.

When on a single line, double quotes are basically meaningless. I should pretend they're not there yet I must apply them or it won't work.

This helps in the sense that I now know how to deal with it. So thanks for pointing that out. The question remains: what were they thinking? It seems unnecessary.

4

I think it's related to the fact that YAML accepts JSON as values... so the quoting helps discriminate between invalid JSON objects and Jinja templates.

5

JSON uses single curly braces, Jinja double ones. That should sufficiently set them apart.

I imagine it may have something to do with prior art. Or legacy, incremental developments that warranted some hacks. It looks like a HA thing more than anything else, there is no issue with the YAML or the Jinja, it's just interpreted inconsistently.

6

As is often the case, the devil is in the excruciating detail.

YAML is a data serialisation language that normally uses a block structure to represent scalar (atomic data), sequences (lists/arrays) and mappings (dictionaries/objects).

This block-structure is extended to permit flow-structure, similar to JSON. Hence the ability to use [ ] and { }.

Whilst this is JSON-like it is not as strict as JSON. For a valid JSON object {"key": "value"} the key must be a string. The value can be an array/object as well as a valid primitive data constant. In YAML there are no quoted strings [at the basic level in block-style], and the mapping-key can be anything, including a sequence or mapping itself. Yes, the mapping key can be a mapping.

This means that {{test: one} : two } becomes a valid YAML mapping, taking the inner mapping {test: one} as the outer key. [yes, I tested this on YAMLlint and it is valid YAML...]

As Jinja templating is a string-substitution process, either the entire YAML is regarded as a "string" for parsing, looking for {{, or sub-sections of the YAML are regarded as "strings" and only these are passed for parsing. As {{ is valid YAML, the latter approach is required.

Since YAML does not use string quotation normally, the need and use of quotation is rather complicated, and varies between block-style and flow-style.

In the block-style, for scalar strings (string constants) there are the multi-line operators | [literal block scalar] and > [folded block scalar]. These mark what follows as strings.

For flow-style scalars, the single line string would normally be unquoted
- all good programmers should come to the party

Where special escaped characters are used, quoting the entire 'string' is required
- "the double quote \" needs to be escaped"

and this is the approach that has to be used on the single line, as a flow-style, so as to delimit "{{" as a string rather than valid YAML. Once the string has been passed to Jinja, it will return as a valid YAML and all is well.

In short - the difference in approach is entirely YAML, being the difference between a block-style scalar string (that needs no quotes to mark it as a string) and a flow-style scalar string, where (in this case) the 'string' holds invalid YAML. The block-style scalar string is seen as a string, whereas the flow-style scalar string is seen as YAML first and string second, and must be quoted to force to a scalar string.

Simple really.

YAML flow style

YAML quoting rules

The YAML parser, will see

day1: >
  {{ this.attributes.get(iif(trigger.id=='start','day0','day1')) | float(0) }}

as a mapping, taking day1 as the key. The value is marked as a folded block scalar, which is a string over many lines. Since it is a string, YAML stops at this point. Inside HA, the 'string' will be passed to Jinja templating, which processes the bit inside {{ }} and returns a string value, which will be placed back into the YAML

Alternatively

attr1: {{ this.attributes.get('attr2') | float(0) }}

is seen also as a mapping, with attr1 as the key. However, this is flow-style and not block-style. YAML will now expect what follows to be a continuation of valid YAML that returns a value for the mapping key.

YAML sees { which is the start of a mapping in flow style, and commences to evaluate the mapping looking for a key.
{ is the start of a mapping, so the key is a mapping, which needs to be evaluated to look for a key.

this.attributes.get('attr2') | float(0) }

is parsed to find the key: value before the closing mapping }. This will cause an error, either because there is no : or because | is a block style scalar command [you can't nest flow and block styles].

The YAML documentation has the explicit answer:-

When not to use Plain Scalars

Because a plain scalar without quotes can conflict with YAML syntax elements, there are some exceptions where you can not use it.

Characters that cannot be used at the beginning of a plain scalar:

  • ! Tag like !!null
  • & Anchor like &mapping_for_later_use
    • Alias like *mapping_for_later_use
  • - Block sequence entry
  • : Block mapping entry
  • ? Explicit mapping key
  • {, }, [, ] Flow mapping or sequence
  • , Flow Collection entry seperator
  • # Comment
  • |, > Block Scalar
  • @, ` (backtick) Reserved characters
  • ", ' Double and single quote
  • <whitespace>
  • % Directive

The single line is a plain scalar without quotes, and, as the documentation says, this can conflict with YAML syntax elements. And, there in the list, is {. Hence you can't start a plain scalar with {, let alone {{, and the answer is to quote the plain scalar.

7

Thank you very much for this comprehensive answer.