Jinja Weirdness with "in" vs. "is in"

Configuration
1

I chased down an odd error in my logs and have narrowed it down to a test case that can be examined in the Template Editor. I’m hoping someone can tell me that I’m missing some semantic nuance.

Otherwise, I think it’s probably a bug, and then I’d like to know whether it should be submitted to Home Assistant or the Jinja folks.

The following code works exactly as you’d expect, and the results are indicated in the comments.

{% set l = ['a']   %}
{% set v = 'a'     %}

{# returns true    #}
{{ v in l          }} 

{# returns true    #}
{{ v is in l       }}

Great… that should just be basic Jinja.

But things get weird if I apply a list filter.

{% set l = ['a']    %}
{% set v = 'a'      %}

{# returns true     #}
{{ v in    l|list   }}

{# generates error! #}
{{ v is in l|list   }}

The error is:

TypeError: 'bool' object is not iterable

Is there an actual Jinja semantics reason for this? I don’t see any reason why that expression should throw an error, let alone why in behaves differently than is in.

2

My suggestion.

1 Like
3

:thinking:

{{ v is in (l|list) }}
4

Yes, that does work but seemingly only deepens the mystery.

5

No, I think it explains it. Tests and | both have a high precedence. Most likely is has a higher precedence then |, So it does:

{{ (v is in l) | list }}

Which corresponds to the error that a bool is not iterable, so you cannot convert a boolean to a list:

{{ True | list }}
1 Like
6

I think you’re right. I will respond later with a test case to either prove or disprove it.

Either way, though, it seems odd that adding the arguably superfluous is construct to an expression would alter how other operators get applied. I’ll look for documentation on Jinja’s operator precedence rules (a quick search right now does not come up with anything, and that’s all the time I have until later).

7

Precedence isn’t defined in the jinja docs but can be determined by experimentation.

is and | (use to apply tests and filters, respectively) appear to have equal precedence and will be applied from left to right. in is a test and has lower precedence and will be applied after.

I’ve always used the my_var in my_list form instead of my_var is in my_list but after reviewing the docs I am now convinced the latter is the “correct” form and the weirdness is that my_var in my_list doesn’t result in an error but it should.

Per the docs, in is listed as a test, and quoting from the docs:

To test a variable or expression, you add is plus the name of the test after the variable

So for example, my_var defined results in an error, but my_var is defined renders correctly. Same with a eq b versus a is eq b or likewise with every other test I’ve tried, except in

In summary, in is the unicorn test in Jinja that can be invoked without utilizing the is operator. That doesn’t mean it is good syntax though.

Edit: see below from Geoff. a in b is proper syntax also, but it uses in as a comparison rather than a test. This has implications on the order of operations.

8

In Jinja2, in is an overloaded keyword being both an infix operator and a test function.

As an operator

in

Perform a sequence / mapping containment test. Returns true if the left operand is contained in the right. {{ 1 in [1, 2, 3] }} would, for example, return true.

Hence foo in bar is treated as <foo: any> in <bar: list> -> bool

and in v in l|list the pipe and list filter is applied first to the right hand list variable, returning a list, which is then consumed as the right operand for in.

As a test function for is

To test a variable or expression, you add is plus the name of the test after the variable.

Hence foo is in bar is treated as test.in(<foo: any>, <bar: list>) -> bool

and in v is in l|list the is consumes the in as the applied test function, v as the first operand with l as the second operand for the test. The pipe is applied later.

Precedence in this case appears to be

is - pipe - in

and it appears that generally the pipe precedence is not strongly defined since chaining should normally be applied left to right, with the left expression “fully” evaluated first, as <foo> <op> <bar> <pipe> <filter> applying the filter only after evaluating the operator.

https://github.com/pallets/jinja/issues/379

Of course, since l = ['a'] is a list to start with, l = l|list returns true and the list filter appears unnecessary.

1 Like
9

This explanation makes sense to me. Seems like it should be treated like an odd (from a linguistic perspective) quirk of the language.

10

I think the “unicornity” must come from it being overloaded as an operator in addition to a test.

I’ve grown to prefer the is in notation because I always like my code to read as easily as possible; even though the is is superfluous, I find it makes logic read more easily, especially when I’m using in alongside other tests where the is is required.

11

The list filter is unnecessary, for sure–the test case is contrived. I ran into the issue in a context where I needed the list, but it was not the best example for purposes of posting here.

12

This is all part of the grammar of a language. In BASIC, when I first started to use it over 50 years ago, the assignment clause started with keyword LET but this became optional, with the = operator being both assignment and logic comparison.

Thus a = b = c results in the variable a being given the Boolean value of the test b = c. Things get too complicated with a = b = c = d.

The driver for what exactly happens, if anyone is interested, is the parsing process when the tokens (from lexical analysis) are converted into an Abstract Syntax Tree (AST). Whilst many languages have well defined syntax, Jinja2 appears to be less well defined, however there is access to the AST via the parse() function in the low-level API. After a bit of experimenting I managed to get this to work…

>>> import jinja2
>>> env = jinja2.Environment()
>>> env.parse("{% set v = 'a' %}{% set l = ['a','b','c'] %}{{ v in l}}")

Template(body=[
	Assign(target=Name(name='v', ctx='store'), node=Const(value='a')),
	Assign(target=Name(name='l', ctx='store'), node=List(items=[Const(value='a'), Const(value='b'), Const(value='c')])),
	Output(nodes=[
		Compare(expr=Name(name='v', ctx='load'), ops=[Operand(op='in', expr=Name(name='l', ctx='load'))])
])])

>>> env.parse("{% set v = 'a' %}{% set l = ['a','b','c'] %}{{ v is in l}}")

Template(body=[
	Assign(target=Name(name='v', ctx='store'), node=Const(value='a')),
	Assign(target=Name(name='l', ctx='store'), node=List(items=[Const(value='a'), Const(value='b'), Const(value='c')])),
	Output(nodes=[
		Test(node=Name(name='v', ctx='load'), name='in', args=[Name(name='l', ctx='load')], kwargs=[], dyn_args=None, dyn_kwargs=None)
])])

>>> env.parse("{% set v = 'a' %}{% set l = ['a','b','c'] %}{{ v in l|list}}")

Template(body=[
	Assign(target=Name(name='v', ctx='store'), node=Const(value='a')),
	Assign(target=Name(name='l', ctx='store'), node=List(items=[Const(value='a'), Const(value='b'), Const(value='c')])),
	Output(nodes=[
		Compare(expr=Name(name='v', ctx='load'), ops=[Operand(op='in', expr=Filter(node=
			Name(name='l', ctx='load'), name='list', args=[], kwargs=[], dyn_args=None, dyn_kwargs=None))])
])])

>>> env.parse("{% set v = 'a' %}{% set l = ['a','b','c'] %}{{ v is in l|list}}")

Template(body=[
	Assign(target=Name(name='v', ctx='store'), node=Const(value='a')),
	Assign(target=Name(name='l', ctx='store'), node=List(items=[Const(value='a'), Const(value='b'), Const(value='c')])),
	Output(nodes=[
		Filter(node=
			Test(node=Name(name='v', ctx='load'), name='in', args=[Name(name='l', ctx='load')], kwargs=[], dyn_args=None, dyn_kwargs=None), name='list', args=[], kwargs=[], dyn_args=None, dyn_kwargs=None)
])])

The key bits - the v in l and v is in l show, as expected, that

in is parsed to the Compare node, using the value v, operand in and value l

is in is parsed to the Test node, using the value v, test name in with argument value l

and therefore clearly in is a comparison, and is in is a test using ‘in’

Just to complete the exercise, I followed this up with adding the pipe list filter.

Again, as I surmised,

v in l|list is parsed to Compare( v, op=in, arg= Filter( l, name=list ))

which shows that the list filter is applied first to the variable l

however, v is in l|list is parsed to

Filter ( Test ( v, name=in, arg=l) name=filter)

which shows that the list filter is applied last, to the outcome of the ‘is in’ test.

All of which explains neatly why you get your error.

Cheers!

3 Likes