AI is scarily good at writing YAML\JINJA, and I'm not sure how to feel about it

Not so much a question but more a comment on my recent experience.

I used to be a coder, back in the day. Like really really back in the day. I used to code in Assembler and I drifted out of coding around the time that Object Orientated languages were becoming mainstream.

When I started tinkering with Home Assistant it was quite a curve as YAML\Jinja “does things differently” from the way that I was taught. Anyone who is used to the older languages can probably relate, but I digress.

A little while back I saw a demo of someone using an LLM to write basic code, and I fed it one of my scripts that I was having difficulty with and it found the problem and fixed my code almost in instantly (It was a rookie mistake), and I’ve been using it on and off ever since. Mostly for times when I need to do something but am missing something simple in the syntax of how to do it.

I’ve been using it as a cross between a debugging engine and a wiki search, and steadily putting in more natural language asking it how to do something and less code asking for how to fix it.

The other day I realized that I’d just completed a really significant capability upgrade on my HA setup that does things that HA wasn’t designed to do, and I’ve got no idea how about 60% of it actually functions because I’ve been becoming more and more reliant on the AI to do the code. I’d mostly stopped reading the code or the explanations, and just copying and pasting it in place, and … it just did what I need it to do. So I didn’t bother reading how it did it.

On one hand, I now have a system that’s significantly more complex and functional than I could have written myself in a fraction of the time that I could have written a much more basic system. It runs, and runs very smoothly.

It’s well coded, not sloppy or patched together, and it uses capabilities that I probably wouldn’t have realized existed without some serious background reading. I’ve seen some horror stories with other languages or environments, and heard about AI churning out badly optimized code for low level systems, but for for a high level environment like YAML, I’m getting really good outputs.

On the other hand, I think that I’ve become a worse coder for it because I’ve forgotten a whole bunch of things that I knew, and I’ve effectively just been telling someone else what I want and getting them to do it for me. Like I was commissioning code some software rather than writing it.

I’m not sure how to feel about this.

I’m absolutely against the dumbing down of systems, I started out as a tinkerer and then became a professional programmer, and without that early tinkering to see how things functioned and what they could have done I’d probably have gotten bored and done something else, but having an AI doing the heavy lifting just makes things so much assessible, which I guess is just another way of saying Easier.

I’m both exited about the future possibilities, and terrified that people who might otherwise have become skilled coders will become dependant on tools like this and just copy and paste rather than figuring things out.

There will always be tinkerers and coders and makers, but with tools like AI there will probably be less than ever.

These are my thoughts, what are your?

5 Likes

My view: it is only as good as you can judge it to be…if you have little knowledge of what the outcome should look like or not the experience to question it… then you could be in for a lot of ‘fun’. Just ask a couple of times how to format a date and check this also on the other platforms…answers galore.

Am I against AI, absolutely not. Does it help, yes. Can it kill your system…that too :slight_smile:

1 Like

If your a little bid older and find learning new stuff that much harder, then AI is incredible. I have some Esp32’s doing stuff I could never have figured out by my self.
:+1:

2 Likes

I guess that’s why we keep backups.

From my perspective HA is a low risk system for most things. I’d trust it to control the heating in my home. The cooling for a nuclear reactor, not so much.

2 Likes

Figuring stuff out just doesn’t seem as much fun now, as it did back then.

2 Likes

Share a sample automation or script.

I can almost guarantee the LLM will have used out of date legacy config it was trained on. Home Assistant moves too quickly for the training to keep up.

Wiil it work?

Probably yes - for now. The devs are pretty good with backward compatibility.

However at some point in the future it may be depreciated (like the recent template syntax update).

If you read the documents and understand what you are doing rather than farming it out to something else you will be in a better position to be prepared for this.

3 Likes

Is fun isn’t it op?

Tom’s right though…

Without any intervention on the user - this is 100% correct. LLMs take MONTHS to train (the stuff releasing today started training last year about this time, maybe older…)

…and HA is a fast moving project. So it leads to things like your LLM probably has ZERO clue that in three months the template platform will deprecate old syntax and BOOM old style templates stop working. Guess what your code is probably spitting out if you didn’t tell it otherwise?

that’s why I did this… Remember the ‘without any intervention’ part - here’s intervention. A whole list of THOU SHALT NOTs.

nathan-curtis/HALMark: Home Assistant LLM Code Safety Benchmark (HALMark) I’m Sorry Dave, I can’t do that.

If you’re going to code with an LLM - do it safely at least. Tell Claude to read this spec and incorporate it into his Claude.md before you ever spit out a single line of HA Jinja2 code. (Same with Codex, GLM, etc.)

That wont fix 100% of your syntax issues but it’ll make it pretty darned good, mine only does really stupid esoteric mistakes now and when I find the back into the list. I’m up to 40 and there’s 6 candidates sitting in my dev platform for submission. DOES NOT absolve you from checking and debugging your stuff.

1 Like

The same for any human who is reading any post on this forum with example code in it from another user.

This forum has thousands of posts on it that give instructions on things for older versions that no longer apply. even the wiki has pages that are out of date. if I’d coded by hand I’d probably have the same issues.

3 Likes

an artist friend of mine had a rant the other day that amounted to them saying that people should pay 100k and 4 years in art school so they could draw their own anime waifus, rather than using ai to do it for them.

I may try my hand at writing an ai integration for home assistant that allows people to directly generate code within ha to extend ha. no more copying and pasting, direct code insertion for scripts, sensors and automations from a prompt.

Nobody will ever need to know how to code yaml ever again. we could even block users from doing it manually.

three… two… one… something sound of furious responses being typed.

There are already hundreds of these. Check the Share your Projects section.

Nobody will ever need to know how to code yaml ever again. we could even block users from doing it manually.

Not sure what your point is. This is still relevant:

Maybe I phrased it badly.

I can go to an AI and ask it to solve a problem for me, and it can give me an answer based on the data that it trained with which may be out of date.

Or I can Google the problem and find a post on a forum like this where a human being solved the same problem, and use their solution, and there’s absolutely no guarantee that this isn’t also just as out of date.

In fact, there’s no guarantee that the AI isn’t basing its answer on the same forum post. So I’m getting the same out of date information from either source.

AI, on the other hand rarely makes typos, it’s great with syntax, and it’s really good at merging code form different sources into a more capable form.

2 Likes

And AI is always trained on data that is months old. HA changes monthly.
If stuff is in the forum, you can tell if it’s kept up to date and when it was posted.
AI code could be based on anything, good and bad, working and not, based on the random selection of bits and bytes.
It’s fine if you can check what it’s doing, but then so is People code.

All Tom was saying is there are a lot of ‘use AI to Vibe Something’ integrations already, look at those first in the Custom Integrations - Home Assistant Community and Share your Projects! - Home Assistant Community and possibly HACS.

I absolutely love it, as I use it to help me learn how to do new tasks and expand my knowledge. I often find it’s wrong, and it’s great to bounce things back and forth to correct the problem. It has saved me a significant amount of time. Sure, programmers likely feel the same way encyclopedia creators felt about the internet, but things are constantly evolving until they die, so I for one am embracing the new era. I’m fortunate to be towards the end of my career, as being in the tech field can be scary, however people just need to provide unique services and you can either be a creative thinker that adapts and learns with AI, or part of a labor force that builds or fixes things (for now). Like it or not, it’s here to stay, and there is no going back from whatever it may bring.

1 Like

You can always read the documentation.

2 Likes

tbh, I often struggle with the documentation. if you have a problem and want a solution then the documentation will tell you how to use a function or feature, but it won’t suggest the best one for your task. if you’ve gone down the wrong route the wiki will tell you how to go down it, but not that you shouldn’t.

I guess that a rookie example is that if someone is trying to trigger an event based on a numerical output from something the wiki will tell you how to capture and compare the output, but an ai will tell you that you need to convert it to a float first and that you may want to trigger if the value goes over a certain amount, rather than being exactly a certain amount. it will also comment the code automatically.

These are things that a human picks up gradually through experience, and often aren’t put in wiki because wiki can’t take your own personal use case into account.

I love that the ai are context aware, they can give exact trigger names or format types depending on my questions. the wiki has placeholders and I may not kniw the correct real world value to enter.

I’ve also found that done ai look up the answers to questions using live data or data from forums, so if we give it a while they’ll probably access the documents directly and keep themselves up to date.

The ones that I’ve been trying out all seem to be on the latest stable build and haven’t given me any depricated code yet.

Yes it is definitely more of a syntax reference than a “how to”.

But you can use it to check your LLM output or examples found on the forum.

1 Like

The coding specific ones are actually really good at that as well. a couple I’ve tried gave very neat code and instructions on how to impliment it. they even noted the recent change in description on the setting menu for apps and said that if you had an older version it wouldn’t be called that.

one I have particular impressed me because I made a mistake in what I was asking for when I wanted it to calculate an amount of rotation and it guessed what I really needed and suggested the correct formula.

in a couple of years time I’m guessing that this will be a a level where llm will be able to build most automation just through a voice input, and then some manual adjustment for things like delays and layout.

5 Likes