Have ESPHome Developers considered adding something like Berry to ESPHome?

ESPHome
1

I’ve been working on developing 3 different devices that use an ESP32 for automation and they work with ESPHome. I’ve had some discussions about learning how to use ESPHome from a newbie’s point of view. For me, someone who had done a lot of work years ago in Perl, has used C++, but not since 2010 (probably not since 2008 or so), but does use Python regularly, I’ve found ESPHome a challenge to learn.

A lot of that is because it’s hard to understand just what the YAML file does - is it a configuration file? An OOP language? Is it part config, part C++? I’ve also run into issues like trying to read variables. YAML reads globals like ${globel_variable}, but if that variable is a number I’m using for anything (even an integer I’m using for timing), then I have to treat it more like I’m working with something in C++ and might have to write a lambda to read it or put it in brackets to get the value of it. (I was shown to do that at least once.)

I’ve started looking at what other platforms are out there. I’m not trying to rate them or make statements that one is bad and another good, but I’ve wanted to learn what’s out there so I can make more informed decisions about what platform or system will work best for different purposes. One platform I’m reading about is Tasmota.

I do NOT want to start an ESPHome vs. Tasmota discussion and I feel, STRONGLY, that anyone saying, “Well, then, maybe you should just go use Tasmota for your work,” is missing the point and avoiding an opportunity to discuss things that could lead to improvements in ESPHome.

While I don’t have anything actually working in Tasmota yet, what I have found is that Berry seems to be a serious advantage for that platform. With an API so Berry scripts can access features of Tasmota (and, therefore, issues in the ESP chips), it seems to me having an actual scripting language like this, where programs and functions can be added to an ESP system easily, is strongly desireable.

I’ve had times where it took 20 minutes to compile something when I finished a YAML file for ESPHome. It’s not just a case of writing a YAML file, then uploading it to the target system - ESPHome requires a recompile with every change in the file. Having a console that can be worked with directly, and which includes or allows the use of a scripting language seems like it would be a major advantage for ESPHome and that it would, in some cases, speed up development and, in other cases, make creating complext applications on ESPHome accessible to a lot more people.

Have there been any discussions about using Berry on ESPHome? (Or any other languages - Berry, specifically, though, is designed for systems like an ESP32 or similar SoCs.)

Is this something others thing is worth discussing? I realize for some, things work fine as they are, but, “Works for me,” does not seem to be a legitimate response. It basically means, “I like it, I really don’t care what helps otehrs.”

The YAML file format and trying to work with a cross between C++ and a configuration file can be confusing. A language like Berry would provide more power than lambdas would and would be a clear, unified, language where, in the language, there issues like, “Reference a variable this way in the logger statements, and this way in these functions.”

2

Yes. You are configuring pre-defined components that are then compiled.

ESPHome feature requests should be made here:

There’s also a Discord server where the devs hang out if you want to ask them directly.

3

Actually, that’s more rhetorical, since it really doesn’t “act” like a configuration file, but more like part configuration, part OOP, but not fully OOP because the programming part is limited. But I don’t think you can say it’s configuration only and no programming included.

I didn’t want to put this out as a feature request here - that’s why I included asking if it’s worth discussing. Clearly, I think it is worth discussing, otherwise I wouldn’t have asked it. I would think it’s certainly worth talking about in terms of what it adds and if that’s enough to look more into it.

4

You can think what you like. Doesn’t change the facts though. Even in ESPHome automations you are just configuring pre-defined automation options like triggers, conditions and actions.

5

For the old-timers, you can think of ESPHome like something similar to RPG or Cobol: There is an implicit loop behind the scenes.

The idea is that MCU programming involves a lot of boilerplate code doing the same thing, all inside a main loop.

The YAML describes what needs to be done inside a loop, e.g. read a I²C sensor, interpret it, and make it available, e.g., to Home assistant. Each component you declare in the YAML generates C++ code that is inserted in the loop.

Tasmota basically does the same, but at an even higher level.

Don’t try to use ESPHome on, e.g. an Rpi. I never had a build taking more than a couple minute on a 10 year old i5 3rd gen.

Unlikely. Lambdas are actually pure compiled C++ code.
Not really sure if scripting is an efficient idea on those resource constrained MCUs.

1 Like
6

I haven’t used COBOL or RPG. When I was in college, I was taking CS classes, but a lot of friends taking IT classes under the Business Dept were taking RPG and COBOL and many of us in CS took COBOL since so many of our friends were getting their first jobs after graduation in COBOL. I have written a number of programs with a GUI, and I’d say it’s also much like that. There’s also the thread I link to in the other part of this post where someone on YouTube is talking about the same thing. It’s very much like that - and it helps for people to know and understand that (But it’s not mentioned in any docs I found, as a noobie coming in and trying to understand ESPHome.)

Agreed. It’s also a lot like an OOP program, where you’re creating a number of objects or classes. When you create a binary sensor and have triggers that tell it what to do on a value change, for example, that’s working much more like writing a series of coded instructions that is programming than writing a config file that is basically listing a lot of settings. in a config file, you can often have sections so you have settings and info put together in groups, but in this case, you do have to put a lot of instructions in specific sequences and are even using lambdas which are actual C++ code, which makes what is being written much more like program code than settings.

Okay, if I have HA running on a Pi, are there tools so I can compile my stuff on a Mac and install it with something other than a browser? Or a way to compile it on my Mac and load it into the ESPHome integration in HA for upload? Just a couple days ago I did my first YAML file for an ESP32 that uses an LCD screen and needs the display info and uses LVGL. First compile was over 20 minutes. Usually my YAML files compile in 1-3 minutes. Sometimes it seems that it’s compiling more object files than others and takes a bit longer.

But I still wonder - why does changing the configuration file require recompiling at all instead of just uploading the new YAML file? I could see, on compiling, storing information on the latest compile and, if a new YAML file uses a library that wasn’t included before, compiling new libraries. But I’m not clear why making small changes in a YAML file requires recompiling at all, as opposed to just uploading the YAML file.

Yes, but with lambdas, that limits the use of C++ code to one line at a time. Admittedly, I’m at a disadvantage here, since I haven’t written in C++ in 15, maybe 20 years, but the use of lambdas has made me wonder about whether it would be possible to include chunks of C++ code within the YAML file or to compile code that would run using the ESPHome API.

Good point, and useful feedback for what I’m asking about. It may not be efficient - but if it were available, wouldn’t it be possible to make it as an add-on option that’s only included when someone wants it? (I do know Berry is designed specifically for MCUs and systems with fewer resources. I don’t know how much that helps in that situation.)

Or, as I mention above, what about the possibility of writing something in C++ that works with the current API? (As opposed to trying to write code that would require a programmer reading the source files - as opposed to an API doc - that could be compiled and linked to the system code?) (Also, again, haven’t used C++ in 15-20 years. I’ve used Java since then, but not much, so my memory of proper terms may be off, since I’ve been working almost exclusively with Python and doing some bash scripting for over a decade.)

I’m not trying to say any of this to create an argument or to be persnickety, but this comment does feel very pedantic and oblivious to the point I’m making here - that a lot of what goes on in ESPHome’s YAML feels more like a program than a configuration file. I know I’m not the only one who feels that and it came up in this thread, in the video in the first post.

A program is a series of steps that must be taken in a specific order to perform a series of operations. That’s very much what’s going on in many cases in the YAML file. Now, two people (me and the person behind the video) does not make a movement, but it would seem that someone with almost half a million followers discussing this very issue in one of his YouTube videos makes the point that, to people coming into the project, it certainly feels like the YAML file includes programming instructions. (Also, if you look up the definition and find Oxford uses " a series of coded software instructions to control the operation of a computer or other machine," that many parts of an ESPHome YAML file do, indeed fit that definition.)

We can argue about it all day, but it comes down to semantics and technicalities (and I’ll throw my weight as a former English teacher and a writer) about the terms, context and meaning, but I think going there and debating a point like this is putting too fine an edge on this issue. I don’t know how many people use ESPHome, but I’ve worked with open source projects since at least the year 2000, and I think I was doing that even before that. In terms of working with users, and in being open to bringing in more users, it’s much more important to listen to what they’re saying and understanding that perspective than to say, “Nope, all wrong. Technically, this is what’s going on,” when they’re talking about the experience of coming in and learning something from the outside and trying to find ways to work with it.

Whether, technically, it’s a program or not, when you have people coming in and describing it as a program, including someone with a lot of YouTube followers, then there’s a perception people get when coming into the project that is worth addressing and listening to, rather than just writing off.

7

Not really. ESPHome is purely declarative. There is not notion of Objects in the OOP sense. When coming into event-driven automations, then it’s “traditional” programming, indeed.

There is zero link between HA and ESPHome (the compiler/uploader), actually.
The link happens between, e.g. the ESP32 and HA.

So you can run ESPHome anywhere. See Getting Started with the ESPHome Command Line — ESPHome for docker, or Installing ESPHome Manually — ESPHome for plain python (including Mac).

Because ESPHome is a code generator + wrapper around a compiler.
It generates C++ code, compiles it into an ESP firmware, and allows you to upload that firmware onto the ESP.

Not sure what you mean by “one line”. You can basically code whole functions in lambdas. See Lambda Magic — ESPHome for examples

2 Likes
8

More a tangential comment, but a typical response I’ve had from some of the key devs over on Discord when my ESPHome logic starts getting “relatively advanced” is that “you should probably be looking at developing a custom component for this”. Which I trust is probably right.

For me, I’ve started a few times to have a proper crack at it but the cross-over to that has always been too hard for me. Especially as there is limited documentation on the required project structure. People who come from the Arduino/ cpp side of things seem to be able to look at a few projects and “figure it out”. Although I’ve seen comments from quite experienced programmers who have struggled a fair bit.

The main “levels” in ESPHome I would say are:
Basic: Just use the config per docs.
Intermediate: Use some basic to advanced lambda’s.
Advanced: Development your own custom components.

2 Likes
9

I don’t know where the line is, but I do get that: At some point, you cross a line to where something needs to be done at a deeper level.

I don’t know just what you’re doing that takes it to a point where it’s beyond YAML. (I’m assuming, and could be wrong, that when you’re talking about a "custom component, that would mean writing it in C++.) This is a good example and a good point to discuss. If it’s a case where a simpler language like Berry would make it easier for non-C++ devs and non-Arduino experienced devs to do the work, it might be worth considering adding a Berry module.

Again, this brings up a line between levels and types of skills and between what can be done in YAML and what should be done in C++. There are some things in projects where the project leaders/devs/members need to say, “Yeah, guess you’re going to have to write your own code for that.” BUT - and I think this is a serious “but” in many, many open source projects, the more people a project draws in, the better it is for the health of the project. Extending that line from “You have to know the more difficult stuff to do this,” to, “Here’s a module or addon or plugin that will help those who have ideas and skills, but perhaps not the more advanced skills we used to make this,” goes a long way to developing the extendability and useability of a project and taking it from requiring advanced skills to working for many more people.

Documentation is always a problem in open source. Nobody really wants to write it and it’s definitely not easy to write. But that touches on another issue in many open source projects: Those who know the project thoroughly find it hard to see it from the viewpoint of an outsider and make sure they have information that the insiders take for granted.

I’m basically self taught. I did put together my own system that did quite well for me and made my clients happy and was thousands and thousands of lines of code. (I never did take time to get what I felt was a good count of that.) But I have had discussions with others online who have found this project difficult to use for various reasons.

1 Like
10

Yes. Not sure if you saw the Custom Components yet? Replaced by external components.

There were two projects from memory where the feedback from the devs were “this would be much easier in a custom component”. Their feedback was along the lines of “there will be way less lambda if you move this to a proper component”.

One was controlling a vintage robot.

The other was controlling a dunny.

Both involved decoding and using relatively involved protocols (one serial, one pronto).

If you scroll through the config you’ll see some pretty big chunks of lambda.