WTF {% %} {{ }} Jinja2

Archive Month of "What the heck?!" 2022
7

Yes, I’ve tried AppDaemon, pyscript, python_scripts and Node-Red.

  • python_scripts cannot have triggers, so you still have to rely on YAML+Jinja for these
  • Pyscript seems easy and coincise, but I found it way too unstable. After installing I got big fat warnings saying it could break HA. I dismissed them a bit worried, and I proceeded writing my first automation, made of 2 lines: a trigger on a device attribute (temperature of generic_thermostat) and a log message. I spent an hour trying to understand why it wouldn’t trigger, then I’ve tried to restart HA, and it began to work. So I added a third line, and it stopped triggering again. Not a good first look.
  • Node-Red: great interface, great debugging; but I’ve tried it two times, and immediately I found myself writing code inside nodes. If I have to do that, I can do it on a text editor (I guess this more because of how HA internals are exposed to it, not Node-Red itself).
  • I’m trying AppDaemon now, but callbacks and the plethora of “self.” doesn’t make it so friendly - but at least it’s poweful.

I think HA need something between YAML and AppDaemon. I understand removing Jinja would break HA, but it’s not impossible to empower something else at its side.
Thinking about difficulties encountered by new HA users (like me) I’d at least put on official documentation:

  • A better documentation on how Jinja relates with YAML
  • That mysterious Jinja “undocumented” methods and goodies (not the official documentation only)

Then, endorse one between Pyscript or Node-Red, or even AppDaemon, and make operations as smooth as possible, making it easy for casual users.

2 Likes
8

Not sure what you’re referencing here. There’s the HA Jinja documents which cover what HA has added to the Jinja environment, and then there’s Jinjas documents (managed and depolyed by the people who make jinja) which covers all the base jinja stuff. There are no ‘un-documented’ goodies, if there is (that HA added to the env), then it should be added to the HA docs. I’m assuming you’ve glanced over the link to the Jinja Documentation inside the HA template documentation?

All of these are custom and managed by users outside HA. There’s a near zero chance of them getting endorsed as official, they will always remain in the custom universe unless the authors of those custom items want to add them to core. IIRC, appdaemon is built to run without HA and they have no desire to put it into HA. Pyscript would be a good candidate, but again up to the dev. Node-red is another automation engine that doesn’t need to run with HA, so that’s most likely out of the question as well.

9

you can get an entity_id components with .split(“.”) method, but the Jinja documentation contains no mention about this method. You can filter entities by selectattr('attribute','match','[regex]'), but “match” is not described in standard filters

I’m referring to these. If there’s a document about these, of couse my rant has no basis.

10

There are documents about them.

Templating - Home Assistant (home-assistant.io)

My guess here is that you’ve never looked at the selectattr documentation in the jinja docs to understand that the 2nd argument for selectattr is a test.

11

I suggest spending spending time to learn the differences between a filter, a function, and a test and where to use them

12

Sorry Petro, probably it’s me (although I’m not the only one); I’m referring to this Jinja documentation: Template Designer Documentation — Jinja Documentation (3.1.x). There’s no “match” under “List of builtin tests”; and there’s no “split” either. I read there’s a post about search and match tests, but how am I supposed to find it when I develop templates? These kind of information should be collected in a FAQ or something similar at least

1 Like
13

Because ‘match’ is something HA added to HA’s version of Jinja, therefore it’s in the HA docs, not the jinja docs.

HA did not create Jinja. HA uses it, and extends its functionality. You have to use BOTH documentations to write jinja in HA.

As you can see in the document I listed, the match line literally says Test match..., indicating it’s a test.

split is a class function on strings, which is linked from the jinja documents.

This covers the section about python methods → Template Designer Documentation — Jinja Documentation (3.2.x) (palletsprojects.com) [EDIT]: So it’s not linked directly, however it mentions that a python objects methods are available for use, and .split is one of those methods that’s available on strings.

Keep in mind this is an API, so the documents are like any other api’s documents. Also keep in mind, that HA has no say in Jinja’s documents. Only HA’s documents.

14

Writing code in NR is just an option the vast majority of the times, not an imposition, and you can do the most complex flows without touching code. In my whole NR instance I only have one function node and I didn’t write it myself, The rest is pretty much absent of code. NR really is idiot proof.

15

My curse is that I always find things that cannot be done the easy way. :wink:
Last time I was trying to create a trigger based on an attribute change. I had to use JSONata as $entity().attributes.temperature > 19 just for trigger; but I figured out I needed more if I wanted to create a “general” automation.
I don’t remember what previous time’s goal was, but it ended up the same way. I know one variable isn’t “coding”, but what I mean is that 1) You have to dip into objects, methods, and so on 2) there’s no specific documentation, you have to find someone explaining it to you

16

There’s documents for all objects that are usable in templates, but you have to know where to look. If it’s a python object created by python, you have to look in pythons docs. If it’s an object created by HA, you have to look in HA docs and it’s typically linked from the HA template docs (like state objects, which is what you’re referencing).

trigger objects all have documents. Same with context objects. Only objects that don’t are event data that comes directly from devices. However you can get the object structure by listening to the event in the dev tools events page.

17

The majority of fields auto-complete. With the majority that don’t, you can just copy/paste paths. It is so silly proof my mom built a flow to turn a light on with a door sensor without my help and loved it, you don’t really need that much documentation. And there’s in fact documentation for every node in the right side menu. It’s really syntax error proof.

18

I appreciate you’re trying to give me help. However, I wouldn’t be so generic in asking, mine is simply a rant :slight_smile:
Here is how it goes: someone (like me) starts using HA; after simple ones, they desire some automations just beyond “click there>lamp on/off” - like each time a button is pressed, light in a group turn on in sequence, and when they’re all on the begin to turn off the same way. Having several light groups, a reusable automation would be perfect: just pass it a list of lights and it does all the rest.
They read “it’s easy, just read these docs, there’s autofill, even my mom could”. So it’s just a matter of defining some parameters and apply them at the right place; study Jinja syntax, find a good object reference and you’re ready.
After some late evenings of frustration, they conclude that’s not the case. It’s a full time work.

I’d prefer to be advised in advance: “stick to simple automation. There will be redundancy and you may need to revise several things if you just change an id, but it’s still easier than going full power. Otherwise begin here, but that’s not enough, be prepared to learn object programming models, concurrent processing, event-driven calls and so on. Templating can be a ‘lighter’ solution, but it’s not easy too.”

Maybe it’s me, I agree; but when I began working with computers, it was exciting that by following some naming convention, you could delete your old tax documentation by issuing “del 2001*.xls”, and you could remember that even one year to another and learn something different in the meantime. That’s the kind of simplicity I miss.

1 Like
19

That’s quite an advanced automation no one should say this:

Simple automations are point and click.

What you described will require learning. Everyone has different abilities and for some this takes more effort than others. But there are plenty of knowledgeable users on this forum willing to help, for free.

2 Likes
20

Templating YAML with Jinja2 started as a big trend a few years back. It’s not only HA that did it, by far. A lot of other software (mostly open source though) jumped on that train to hell. Ansible, looking at you here… It’s like all the developers of XML/XSLT sat around a table and went Okay guys, we need a new config as code or vice versa mishmash format, but it needs to be even worse than what we have, it needs to be completely unreadable, undebuggable, needs tons of syntactic whitespace, oh and it needs to be turing complete ! Go !..

YAML as a pure config format is not great, but manageable with the right tools. Jinja2 is an abomination. Unfortunately, as petro mentioned, changing this now is next to impossible. It would break literally everything. Some mistakes will follow you forever.

2 Likes
21

I don’t see it as a mistake. I found YAML to be very logical and easy to pick up. Way more human readable than some other markup languages.

Jinja took a little more effort but the simple concepts (e.g. how to retrieve a state) were relatively easy to pick up. I’m still learning.

22

As a pure markup format, it’s okay’ish. I mean, there are not many alternatives anyway if you want human readable. I absolutely hate languages with syntactical significant whitespace, but that’s mostly personal preference.

But config is not code, code is not config. Or at least, it shouldn’t be. That’s what separate scripts are for. The way Jinja2 mashes all this together is horrible. Of course you can learn anything. But learning Jinja2 looks about as fun as trying to pick up COBOL or something from the days of yore. It just doesn’t look and feel like a modern language.

Pyscript is much better and cleaner. It’s still Python (which I don’t like), but subjectivity aside, that is the way HA should go, IMO.

23

Like you, I have very few function nodes in my Node-Red setup. The few I do have were more about me learning how the function node worked than an actual need to do whatever in code. I’m a professional programmer but I prefer keeping my flows visual. That way it’s easier for me to tweak and tune the logic as various edge-cases present themselves.

1 Like
24

That’s quite an advanced automation no one should say this

We’re getting to the point.
It’s just save=state.light[N]; for i in (1, N): state.light[i]=save.state[i-i]; state.light[0]=not(save)
Pseudocode is easy, but in YAML+Jinjia it’s very complicated (I’ve done it in python_script, I’ll post it ASAP).

Another easy automation for me is: turn_off(light.any, climate.any except light.porch_lamp).
[I know it’s close to doable with YAML alone if it wasn’t for the “except”, so I added it - not an uncommon need]

A complicated automation for me is:

  • take (external_temp, internal_temp, climate.furnace.state), feed it to some filtering (kalman?) and estimate (heat_power, heat_dissipation_ratio)
  • given (heat_power, heat_dissipation_ratio), estimate for current (external_temp, internal_temp) how much in advance turn_on(climate.furnace) so that internal_temp reaches given setpoint at a given time.

I’ve endeavored making my whole home smart, so I’d like to play smart things with it!

Templating YAML with Jinja2 started as a big trend a few years back.

Actually, I have nothing against Jinja di per se; but the user base of Ansible and an home automation system are very different: the former work with it everyday, the latter do it from time to time, maybe months apart. That’s why a structured language, with an overload of possibilities, especially with a scattered documentation, becomes a stopgap.

Moreover, an home automation has to be reliable and robust to exceptions and errors (how many people writing templates ask themself when they will be evaluated and if they’ll always yield meaningful results?).

If I had to bet, in such a situation I’d first consider a visual language, like Node-Red or Blocky. However it has to be well integrated; if you have to write hard-to-remember function calls inside graphic items, its usefulness is lost.

I repeat: I’m not pushing for it, I know it would be a though work; I’m just suggesting not to take YAML+Jinja for granted as the best decision.

3 Likes
25

Could you elaborate a little bit? I don’t get it…

26

Now I got it, but that’s more of a physics or statistics problem than a code problem… Depending on the information you are counting with it might just be a rule of three… once you get the formulas figured out it will be irrelevant whether you use NR or jinja or whatever…

Anyways, I’m not sure this way of achieving this is worth the extra effort vs simpler solutions.