ESPHome teminology mishmash - A maze of twisty little passages

ESPHome
1

I’ve been using HA since the very first RPi was the go-to platform and have recently been trying ESPHome. I’m an experienced HW and SW designer and ESPHome has great potential, especially the Device builder. After a few weeks of experimenting, chasing down blind alleys and dead ends, as well as some satisfying progress a few general impressions have emerged that I haven’t seen mentioned elsewhere so I thought I’d start a discussion here. And, FWIW, in spite of the issues I’m reporting on, I have found ESPHome to be much more approachable, promising and reliable than other frameworks I’ve been trying, including Arduino, Platform.IO, Circuit Python and Micro-Python. In each case I’ve hit roadblocks with missing or defective libraries, unnecessary complexity in describing what I’m trying to do, or other issues.

At the topmost level, ESPHome seems to struggle a lot with finding a coherent nomenclature for important concepts and communicating those to its users in a way that helps them understand exactly what they are constructing and how those concepts relate and interact. I’ll list a few examples but there are so many others it would distract from my main point to list them.

  1. For example, what a “platform” is, and why it matters whether something is or is not a platform is not clear at all.
  2. I can discern a method behind the madness of “output:s” not being visible to HA, but it’s only a guess inferred from how things like switches, sensors, etc, are described.
  3. Keywords are reused in different contexts (e.g. “inverted:” in Pin Schemas and in “outputs:”) but their descriptions are nearly identical and what happens when they both exist can only be figured out through trial end error.
  4. The “Advanced parameters” for light objects are presented in the GUI without any logical ordering or grouping that would make it easier to infer behavior and how they interact.
  5. When things like “on_boot:” automations occur relative to initialization of e.g. ethernet or WiFi initialization is unknowable except by experimenting with almost random sets of other automations or actions, lambdas, etc.

My key input is that addressing this really need some holistic architectural attention to refactoring and harmonizing the concepts behind I/O classes, disambiguating terms and keywords for each context, and what is executed when. Additional or updated documentation and a lot of probably “little” updates in a lot of places are needed to help users get things working right the first (or second) time.. Overall, it has the feel of something with breakthrough potential but a lot of technical debt. That’s why it needs a top-down approach.

So perhaps I’m missing something here and if my impression is off base I’d be delighted to get pointed in the right direction. I haven’t been able to discover it yet however.

2
  1. They’re “hardware drivers”. They tell ESPhome how to talk to specific hardware.
  2. It’s actually quite logical.
  3. “inverted” means the same thing in both cases.
  4. No idea what you’re talking about.
  5. There’s a documented list of boot priorities.

https://esphome.io/components/esphome/#esphome-on_boot
800: Hardware initialization of vital components (runs very early).
600: Most sensors are set up.
250: Wi-Fi is initialized.
200: Network connections like MQTT or the native API are set up.
-100: Pretty much everything is already initialized (runs last by default)

3

This is a new hobby for me. Another rabbit hole; with a completely new language to learn.

I have been blown away by the development and breadth of supported devices.

And the documentation (and this forum) is pretty good. Just amazing what ESPHome can offer in your home once you get started!

4

Hi Perry, and welcome.

I come from a programming background in the BASIC days and over the past few years have become a huge fan of ESPHome. There is however a paradigm shift from thinking about writing code. Essentially the code has already been written to drive the hardware devices (called Components), and we just need to provide the parameters (component configuration, actions, conditions, etc). It might be frustrating for a while, but I’m sure it won’t take long before it “falls into place”.

I do agree though that it isn’t very user-friendly. Basically I think all Open Source suffers from “by engineers, for engineers (with the same background and level of competence)”. Developers want to get on to the next cool feature, and documentation is a pain.

The official documentation is getting better, but is still very technical with almost no overview or tutorial. Look elsewhere for those :frowning: The “ESPHome” term itself is often used to describe a device running firmware generated by ESPHome, by the ESPHome Device Builder GUI, or even the ESPHome list of devices under the Home Assistant “Devices and services”.

It took me a while before discovering that the ESPHome runtime during on_boot: launches a separate thread to initiate WiFi or Ethernet, and consequently sometimes other on_boot: actions will happen before or after WiFi connects. With poor reception at my workbench sometimes my ESP32-S3 would run 5 minutes and go back to deep_sleep before WiFi had even connected !

Over the past year or two there has been a lot of effort behind the scenes to refactor code, create the Device Builder GUI to make it more approachable, and so very many little updates.

I think your impression is understandable, and it will just take a little time for things to fall into place. In the meantime this forum is a great resource - both for searching and for asking questions. Be aware however that some of the older posts have become very dated with the speed which ESPHome has been / is being developed.

5

ESPHome is like Lego blocks. You pick the length and color and assemble them to make your finished product.

It pulls together code from PlatformIO to support a host of devices and integrates it into HomeAssistant. Open source collaboration at its best.

The learning curve while you futz around is steep, but the results are powerful.

Recently had a GUI overlay to support the underlying YAML code.

Maybe start here. Stand on the shoulders of giants - build great things!

You probably started here, hence too much too soon.

Compare to Tasmota for interest. Both have extensive documentation and a bewildering choice of options, based on flexibility. Both amazing. Both free and open source.

6

I appreciate everyone’s thoughts on this. There are many good points brought up. While I digest them all and compose some thoughts, I wanted to avoid going dark and make a couple key observations.

The ESPHome is in no way an “Ugly Baby”. My initial application was simply replacing a wall switch with a PoE based version using a decorator switch. It has to fit in a standard NA wall box. A Webhook would be invoked when the switch changes state, alerting an HA automation to do something. That should have been simple but turned out to be anything but. Between Arduino IDE and libs, Circuit and Micro Python limitations and/or libraries that worked 95% but failed to get the job done, and a steep platform.IO learning curve, I’ve been frustrated at every turn. With ESPHome I finally have something that actually allows me to flip the switch and turn a light on/off.

ESPHome itself has been around a while, while the Builder is relatively new. Looking at the code the Generator/ESPHome builds I could see how much thought had been put into organizing abstraction layers to allow very broad device/MPU support with low execution overhead. I don’t understand the architecture but it’s apparent that it’s there, and it works.

Learning ESPHome reminded me of when I first picked up Home Assistant many years ago. It became famous for its vision, but also unfortunately for the steep learning curve. The basic architecture has withstood the test of time and extreme scaling. But terminology was unfamiliar and sometimes loose, ambiguous, and sometimes internally inconsistent. In order to understand the documentation you had to read between the lines in ways that only the developers could do at the time.

HA has improved on this immensely and ESPHome starts from a stronger base, but there’s still way too much IMHO left unclear, loosely defined, internally inconsistent, and requiring experimentation or reverse engineering to figure out what configuration items actually do, especially when they interact (as happens a lot). My overall impression is this isn’t an architectural issue or something that whacking away at individual component or core configuration forms or keywords will resolve quickly. The top levels of compute platform, configuration and component selection seems to flow well. Where it breaks down is at the middle layers where the nits of actual functionality is specified and success is left too much as an exercise to the user.

7

This is a valid point - “platform” is used in two different ways in ESPHome though there is a some common logic;

  • The various target device families like ESP32, nrf52, rp2040 are called platforms - so ESP32 is a target platform.
  • Hardware-dependent implementations of sensors are also called platforms, so you could have two sensors (a sensor in ESPHome means an entity that has a floating point value) one of which is implemented via the adc (Analog to digital converter) platform, the other uses the Homeassistant platform (it reflects a sensor value in HA.)

The common thread between the two uses is that both mean a “hardware platform”.

To complicate things somewhat, both kinds of platforms are implemented in ESPHome as “components”, but many components are not platforms. A component is essentially something that gets configured in the yaml as a top-level key - e.g. sensor is a component, but to define a sensor you have to provide a platform key which specifies how the sensor is implemented.

There are quite a few other areas in ESPHome where the terminology is confusing - like “template” which usually means a C++ lambda expression embedded in the yaml - but it isn’t really a template in the usual sense.

Then there are confusing issues relating to how ESPHome and HA interact including the point you made about output not being an entity that gets exposed to HA.

The chief gripe I have is that HA creates entity IDs for ESPHome devices by mangling the user-friendly name, instead of using the ID defined in ESPHome. There are continual patches to HA to paper over the cracks that arise from that design decision.

But it’s worth persisting to get your head around the concepts - ESPHome is a far easier way to get embedded code running on an ESP32 than anything else, even if it’s not talking to HA. I have decades of experience with embedded systems (which didn’t stop me being as confused as you to begin with) and ESPHome has a lot to offer compared to the alternatives. Hang in there!

8

You did a good job of detailing some of the points of confusion - much better than I could.

Of all the attempts to create an approachable, “platform” independent framework for constructing embedded designs, ESPHome is by far the biggest leap forward I’ve seen. My first job when I started at HP was to design the HP64000 in-circuit emulator for the 6809. Those were the days:-) So an awful lot has changed since then, but really it’s still knowing what blocks are in the toolbox and how to connect them. Gates then, web servers now. It’s all the same, right? :-1:

Underneath however the same layers exist. When trying to make sense of the configuration, component, device, entity, and keywords in ESPHome I saw glimpses of order applied very inconsistently as you point out. For example, the idea of a physical “output” component being invisible to HA, but making it visible when a “switch” or “dimmer” gets layered on top is something I can grok. The key is consistency so I don’t have to remember corner cases.

A related example is how open-drain outputs are handled. There’s great support for selecting just what kind of “output” is desired, including an in-your-face description of how the selection forces lower level selection of totem pole, pullup, pulldown, choices. Providing the “custom” bailout that makes it clear the lower level choices are up to you is very easy to understand. Where it breaks down however is the linkage to declaring an “Input” pin. allow_other_uses: is a workaround to allow cobbling together the complete pin description. It’s also an option that only shows up after flailing around with web searches using different ways of saying the same thing until a “hit” that might make sense pops up. I can suggest several ways to place an option or button right where the output type is selected that would turn from grey to black when an open drain output is selected to associate an “input” with the "output. But that’s whack-a-mole.

ESPHome is now the first thing I reach for when pulling together something for all those postage stamp and chewing gum sized MCUs I use now instead of 7400 TTL to do stuff. I posted my thoughts here because there didn’t seem to be a more appropriate place to begin the discussion. There does seems to be a general understanding a pass at the abstraction layer definitions and relationships would be useful. If there’s a better place for this discussion I’m all in.

P.S. I actually much prefer to use the RP20xx, STM, etc. based MCUs over the ESP ones. Everything I do is wired (not wireless). So it’s ironic, in a sense, that I’m using “ESPHome”:slight_smile:

9

That takes me back - one of my early jobs was coding fuel pumps for Gilbarco using 8080 (or Z80?) with HP64000 systems, in assembler of course. I recall it had some kind of scripting system that I leveraged to automate the build process.

10

All the retrocomputing interest is causing something of a revival for the 64000. Some folks in Italy were using it to bring a Z80 based video game back to life. No boot? No problem. Just plug in the emulator and peek/poke around. You didn’t even need a clock since the emulator could drive it. Just before their big demo the emulator broke so I helped them fix it. Even today some of the stuff we did transparently in real-time would be considered state of the art.

Another RBI factoid: The Nevada Gaming Commission had a 64000 system. They would show up unannounced at a casino, point to a slot machine, and plug in the emulator. If the firmware didn’t match byte for byte with their copy it was big trouble:-)

Now back to our regularly scheduled programming…

11

I found the UX design in the Open Home Foundation discussion topic yesterday. Would that be the appropriate place to work on the issues I raised in my OP?

12

I have a specific need to read a MAC address/hostname saved in a special flesh memory and use that when initializing the Ethernet interface. The documentation doesn’t provide enough information to tell me how to do this reliably. I’ve looked at the source code and unraveling the order in which things get done based on configuration choices like inhibiting ethernet enable on boot lands me in the maze of twisty passages again.

Doing this allows me to avoid the cumbersome use_other_mas_address: hack to commission a fleet of identical devices, or do an OTA update of the fleet, using a single common ESPHome design. One ESPHome design can “onboard” each device instance to set its MAC and hostname by programming the special flash memory, then use a single ESPHome design of the devices functionality to “commission” each device. Each device will then power up with it’s assigned MAC/hostname but otherwise behave identically to every other device.

Where is the appropriate place to raise this specific question on setting the mac address (and hostname) from the get-go based on info in the f;ash memory?

13

Interesting exercise.

I have previously provided tech support including for wifi routers - but am far from an expert. My understanding is that MAC is a world-wide unique identifier assigned by the device’s manufacturer - not something that a user should (or can) set. Do you really need to spoof the MAC Address or do you just need a unique IP Address ?

Similarly hostname is not necessary for a client device. If you are wanting other devices to connect to it by name, then maybe wifi: ap: is what you are after ?

I assume that firmware which doesn’t specify manual IP addressing will use the router’s DHCP to issue a unique IP Address. That will be OK for the first unit, but I expect that ESPHome Device Builder still associates that firmware file with the allocated IP address. How else can it know which hardware device to update when it comes time for an OTA update ?

Trying to load the same firmware to another device OTA will simply update the first device. Maybe if you loaded the firmware into each via USB ? but ESPHome Device Builder would only know the last unit, and so unable to update the others OTA.

Basically ESPHome was not designed for fleets of identical devices, and there are no concepts of “onboarding” and “commissioning”. Certainly other users have experienced frustration at having to have multiple almost-identical yaml files (in my case for 8 smart power plugs). I believe the best method is putting all the common code into an include file and a small yaml file containing a few substitutions such as:

# I have a number of Arlec PC191HA smart plugs with Power Monitoring,  
#
# This contains the parameters specific to this individual device.
substitutions:
  deviceIP: "111"                      # last octet of the IP Address
  currently_used_for: "chest freezer" 
  update_interval_sensor:  "5 min"     # How often to measure and report other values

packages:
  common_wifi: !include _common_wifi.yaml
  device_base: !include _common_pc191ha-v1.yaml

Recent improvement to ESPHome Device Builder allows selecting multiple devices to update - far from ideal for users with lots of devices, but still better than running each update manually.

FWIW, I have done some investigating with deep_sleep and wi-fi, which reveals network info. I use the BSSID to work out which WLAN node it is connected to

image
image467×351 25.7 KB

Bottom line is that I think the firmware that ESPHome produces may be suitable to run on your devices - but that ESPHome Device Builder is probably not the best suited platform for your objective.

You can certainly raise this query in the ESPHome Discord (if you haven’t already done so) since that is where all the developers disappeared to.

14

Thanks for the thoughtful response. I’ll respond myself in two parts, first to a couple use model points/questions raised, and then to the more technical parts.

The MAC address I want to use is actually globally unique one assigned out of Microchip’s allocation. I am using the Silicognition PoE Featherwing + the Silicognition RP2040 shim, all piggybacked on a standard Decorator wall switch:

image
image964×1256 243 KB

This actually fits in a standard NA single gang wall box:-) The PoE Featherwing includes a W5500 as well as a 2k EEPROM sold my Microchip that has unique MAC (IPV4 and IPV5) addresses in read-only storage. Every EEPROM shipped therefor has a unique MAC address available for reading.

Reading the EEPROM requires use of an ESPHome external component from github://pilotak/esphome-eeprom to get the MAC addresses. This works fine. All need to understand is how to read these bytes and invoke the already public method in the Ethernet class to set the address before Ethernet is enabled. Actually, I need to set the hostname as well since it is made of the device design name and MAC (PoE-wall_switch_)

A broadcast update to all my wall switches would update each wall switch instance with an IDENTICAL binary image. each updated unit would re-register to DHCP using its same MAC. Fleet update for ESPHome may support this, or I could use a script to do a push update to all devices. The big win however is the exact same image I can debug with would be used unchanged by my 50 or so light switches.

The “normal” method in ESPHome using use_mac_address: and moving several configurations around reminds me of solving the Tower of Hanoi. Not quick or reliable.

15

Unfortunately, that is not a view of the world that is compatible with esphome.

There appear to be two diametrically opposed views:

  1. One image to rule them all. This is what you want. If so, use Tasmota. That is consistent with it’s view of the way things should be.
  2. Unique images for each individual device to make sure only what is needed is included.

There are advantages and disadvantages to both views. I started with Tasmota and still use it on all my devices that I created before the last couple of years. I started using esphome because I got frustrated with very long compile times for esp32 devices and the one image for all broke down when there was a need to support many different controllers (all the variants of esp32).

The development experience with esphome is actually better than for Tasmota, but for something simple it might not matter too much.

16

That is normal industry practise, done at manufacturing stage, universally respected, and works well in real life.

The MAC address is hard coded and usually read-only for normal use, baked into the hardware level. Are you wanting to extract that for firmware purposes, not modify it?

Are you sure you need to go to this extreme? The code compiled during normal ESPHome code that supports the W5500 should expose this without any custom external routines. Maybe your YAML code algorithm needs tweaking?

The text sensor in ESPHome Ethernet Info exposes different Ethernet information, including the MAC address. (I note the documentation seems to indicate it can be changed (set), not just read (exposed)???)

I quote:

mac_address (Optional, MAC Address): Set the MAC address of the ethernet interface.

See the official documentation at

Tricking your network into having devices with the same MAC address is probably not wise at a number of different levels, your understanding of the different network layers in the standard OSI level being critical here. What happens if something goes wrong - how are you going to be able to identify which light is which if they all appear identical without physical inspection of the serial code sticker on the side, if that exists? How will you reconnect if two devices are simultaneously demanding a session using the same MAC?

Maybe you should investigate another method of doing your updates? I note Tasmota supports OTA updates by calling a url for each device. So does ESPHome. It is called OTA Update via HTTP Request.

There is a Tasmota mass update app called TasmoAdmin that you may be able to adapt for your needs. This does not change the MAC address. I recommend you don’t either.

You may be able to dynamically create a custom YAML file including your host name, IP Address, and spool it off to a remote ESPHome compiler server and then OTA upload that to each device. This enables you to harness a faster computer to shorten your compile times. You may be able to disconnect the sequential process and pre-compile and OTA separately, the OTA CLI documentation showing you the parameters to update-all once all the YAML files have been generated.
You may also consider raising a change request to add parameter passing for the host name, etc when doing OTA updates, and then your goals would be achieved with little further ado.

17

I don’t see why. ESPHome’s support for lambdas, for example, allows designs to sit almost anywhere on the continuum between the two views you described.

I’d describe ESPHome as a “low code” framework with Arduino, MicroPython, and PlatformIO as increasingly “code centric”. It’s very approachable because of this. Also, the components generally work. which is different from my experience with ethernet libraries for Arduino and MicroPython. (I haven’t used PlatformIO.) Lambdas are a “safety valve” that provides a lot of the flexibility of the code centric frameworks without having to define YAML keys for every possible corner case.

18

I believe you are missing my point. You CAN do either with either. But if the developers view the world one way, they will make that way easy and the other way might or might not be easy/possible.

esphome was designed first and foremost to be individual firmware for individual devices. Works great to support a wide variety of scenarios with minimal firmware bloat. BUT, assumes that you will make custom firmware for each device (even if you have twenty that are identical functionality, so only need a name specified).

Tasmota comes from a simpler time (created well before HA, let alone esphome). It’s original premise is one firmware for multiple different devices, so it has stuff for initial configuration and stores that in NVRAM, which allows easily updating firmware for multiple devices with the exact same image.

I suspect lambda support was added later to allow more complicated customization in a reasonably easy way.

You generally won’t find the developers of esphome on this forum. There is a discord where they might be more available. See what they say.

19

I’m afraid I’ve inadvertently created a digression from the main topic in my OP. But since we’re here and it is actually a fundamental question about what ESPHome is for…

I think a few things are being conflated here but it all boils down to forcing the identity of an IoT device (i.e. mac address) == an ESPHome configuration. The only reason I can see for this for the convenience of ESPHome’s current implementation of OTA updates. This is a big sacrifice to make for the sake of OTA process simplicity, especially for a framework that’s trying to be approachable and used widely.

This disconnect from real world applications is big enough that someone developed Fleet for ESPHome to address this one thing specifically. Fleet for ESPHome looks very polished but I haven’t used it yet. Perhaps it does some of the things suggested, like compiling to an ESPHome YAML file for generation of the final flash image. Frankly, with respect, that would be a hack workaround of a deficiency in ESPHome. Why go to all that trouble when simply allowing a lambda so the design can read a factory provided mac_address from a memory device that’s separate from the main flash memory?

Now to connect this to the main topic, if a lambda were allowed in mac_address: it could easily break the principle of least surprise. Does the lambda code execute before ethernet is enabled? What if enable_on_boot: = false? Would that be necessary to assure the lambda executes before ethernet is enabled? It’s probably the case that a mac_address: lambda would just execute in place of code that reads the static value from the main flash memory, but this kind of key interaction info is often missing or weak in the docs and GUI. Things like this are what have flung me into the Colossal Cave.

20

At the risk of talking past each other, I think I understood your point but didn’t see that evidenced in ESPHome because of the broad lambda support.

I appreciate the pointer. I really wasn’t sure where to put the OP but I’ll find the discord channel and create a posting there. It’ll be much better formulated however because of all the input here:-)