How to setup an IDE to test new espHome components

ESPHome
1

Hello, I read

and I setup an enviroment to compile and test code faster.
I made an ubuntu image on Virtualbox, since seems that only linux is supported.
I installed docker, vsCode
In vsCode I installed PlatformIO extension

Downloaded GitHub - esphome/esphome: ESPHome is a system to control your ESP8266/ESP32 by simple yet powerful configuration files and control them remotely through Home Automation systems. ad described, open with PlatformIO, setup something…
I tried to compile having many errors (esp8266 and esp32-arduino was ok, esp32-idf and another wasn’t ok). I kept only esp32-arduino in platformio.ini

I don’t know after compilation which component is included.
Is there a yaml I can change to perform a compilation as if it is in Home assistant?

Thanks

2

What are exactly trying to achieve?
You can test components with external_components in yaml, e.g.

external_components:
  # use ble_client_hid from this master branch in GitHub
  - source: github://koying/esphome-ble-remote@master
    components: [ ble_client_hid ]
    refresh: 5min

If you just want to compile faster, install esphome under docker in another machine.
Example docker-compose:

version: '2.1'
services:
    esphome:
        container_name: esphome
        volumes:
            - ./cache:/cache
            - ./config:/config
            - /etc/localtime:/etc/localtime:ro
        restart: unless-stopped
        image: esphome/esphome:stable
        #image: esphome/esphome:beta
        network_mode: host
1 Like
3

So at the end the easy solution is to go through gitHub and a real HA? I’ve seen I can also use a local repo that’s probably easier, but always slow since I have a rpi.

Isn’t possible to simulate something directly from the IDE / toolchain?

I don’t fully understand also the compilations phase in VSCode: since the C source code is made by a py script based on the inut yaml, what is compiling VS (with PlatformIO) since there are no yaml?

4

You lost me.

This has nothing to do with HA

ESPhome is the IDE/toolchain.
What “IDE/toolchain” are you talking about? Behind the scene, ESPhome is using platformIO, but I don’t think this is what you’re talking about.

What has VSCode to do with all this. Are you using ESPhome in CLI mode?

Once again, what are you trying to achieve?

5

I don’t know exaclty what i’m using, but i can describe what i’m trying to do.
I have an HA and i want to make an Esp Home external component. I have the plugin in HA and I can build - load component from there, but it’s not suitable for developing purposes.

The main goal is to modify component code, to build a bin and to load it using ota. Runtime errors will be analyzed using logs because i don’t have jtag access and the device must act in an RS485 network i think it’s not easy to simulate.

I landed on External Components — ESPHome - where they link the Contributing — ESPHome page .

To keep all clean, i started in a virtualbox ubuntu system, I downloaded espHome in a folder and I Installed vsCode + PlatformIO after I read

Now you can open ESPHome in your IDE of choice (mine is CLion) with the PlatformIO addons (see PlatformIO docs for more info). Then develop the new feature with the guidelines below.

After opening PlatformIO plugin and project, it blamed me because i haven’t got docker installed. So I installed it using apt as described here .

Now it compiles (hitting the shortcut) and produces a bin (at least for esp32-arduino) after a long compilation time. I really don’t know what he compiled. He made many .o objects but (if I have understood it right) - these could be done only after a py pre-preprocessor have interpreted a yaml file regarding wanted components. I haven’t found the yaml file.

The next step I would try is to add a random git working external component, to test the produced firmware, and if it’s working i can proceed with a local component i can modify.

Thanks

1 Like
6

I think you’re mucho confused :wink:
At least, I don’t understand what you’re trying to do…

Maybe start with

7

I don’t have problems using espHome in Home assistant, or to use already developed esp home components. You linked a general usage of espHome plugin, that’s not my main goal. I’ve already created 3 different devices with a combination of different components and they works well.

A 4th esp device uses a custom component, based on RS485, that’s not working well and i would like to modify the C code of the component and to test it.

The main topic is: how can I modify the component in an enviroment where I can use an IDE or something more practical than a plain text editor?
Probably, but i’ve not tested, I can copy in my HA (that’s in docker) the extension code and to import it directly using

  - source:
      type: local
      path: my_components

but it will be probably be a mess if there are compilation problems.

I suppose that there are different setups meant to develope a custom component, maybe something that allow to perform tests before going to a release environment.
Isn’t it?

I tried to setup that VM as described before, but i’m not understanding where is located the yaml that guide the component compilation (i.e. in homeassistant/esphome/mydevice.yaml for “release environment”)

8

Where is it?
See my very first post here on how to use a custom component from github.
Assumption here is that you have forked the custom component, made your modifications, and push those modifications back to your fork.

As you mentioned, the alternative is to use a local folder.

Not sure how you actually landed on “Contributing”, but that seem irrelevant to your request.

9

This is the component

Second row in External component is:

You can find some basic documentation on creating your own components at Contributing to ESPHome.

I thought the IDE setup was designed also to develop custom components, since they are the most of the code.

I haven’t forked it yet, because I tried another (maybe wrong) path.

How is it supposed to write code before pushing? I can’t believe all has been written without auto completion, syntax check, and so on (even if it’s possible, it’s a return to the stone age)

10

Likely using a “local” external component, possibly with ESPhome as a docker container as I mentioned above, obviously with vscode as IDE for the component itself.

11

Hello @brazoayeye, reading your topic I come across the same problems.
I already knew about platformio when I was still playing with Arduinos to run my sketch code, but now that I use HA + espHome I don’t want to know anything else.
However, over time we are evolving with the use of Home Assistant and its resources. We want to do some more complex things, to integrate our embedded electronics devices that we love (ESP822 and ESP3).

So I started to venture into creating my external component. At this point I identify with your doubts.

I can’t “Build” my projects, and no matter how many times I install the lib dependencies, it always breaks.
One thing is certain, espHome should have a more detailed step-by-step guide on how to install and run it via the platformio.
What I can tell you is that you don’t necessarily need to run it in the environment, if the code is perfect, just upload it to your git branch, declare the external component in YAML and with luck it will run or break, but at least this way you can debug. …

image
image1916×1036 150 KB

image
image969×801 80.2 KB

If anyone who knows how to configure and run the platformio environment for espHome can help, we would be grateful!

12

At the end I found a decent compromise.

I program inside VSCode that can detect some errors
I installed pip install esphome and all worked well creating a starting yaml with esphome wizard myYaml.yaml and then esphome compile myYaml.yaml. I downloaded the component locally and included as described before, That runs the entire compilation phase including py script and so on.

Since i’m working from another network I also made a ssh tunnel to the HA server and then to the device with something like

sudo ssh -L 80:192.168.32.103:80 -L 3232:192.168.32.103:3232 -L 6053:192.168.32.103:6053 [email protected] -p myCustomPort

then added `127.0.0.1 myEspHomeDevice.local to /etc/hosts

This way I can compile in the ide, than compile better in a terminal, then updating the device using ota and testing it runtime with esphome run myYaml.yaml

There probably are better solutions, but i’m very bad on setting up enviroments because I made it few times in my life and, since it’s something set-and-forget, I always forgot it after I had set it.

1 Like
13

Custom Components are being deprecated for External Components. If someone who has their head wrapped around how best to do this would make a Youtube walk-though on the process; that would be fantastic (especially if it included PlatformIO autocomplete and debugging).

Did find this:

14

Unfortunately, developer documentation for ESPHome is pretty much nonexistent. From what I can tell, there are different implementations of Custom Components. One style (older?) is basically a c++ file loaded into your project.

Another version of custom components (newer?) has your custom code located in config/custom_components/my_custom_component. And within that directory, you’ll have .cpp or .h (or both) files, and at least one python file __init__.py. This follows the standard structure for all ESPHome components (built-in and external). If you code this type of custom component properly, it’s easy to convert it to an external component, which can then be loaded from a git repo or locally anywhere on your build system. The ESPHome Contributing To guide shows the basics of that structure.

So, in building my external component, I first got it running from my custom_components directory (as mentioned above). Then I converted it to a bonafide external component. When that was working, I pushed it up to github, and referenced it accordingly from my ESPHome project.

The difficult part here, for me at least, was figuring out what to do in the python file. That’s where documentation is fully missing, and you pretty much have to reverse engineer the ESPHome built-in components to see what’s going on and how it works. And it’s not at all obvious - there are heavy layers of abstraction in using one language to code another language. I’m an experienced software developer, and I spent most of a work week before I had a functioning __init__.py file. I hope the devs put together some docs at some point.

As for build environment, I’m using the official Docker image esphome/esphome. I think they call this “ESPHome CLI”, and HomeAssistant is not part of this and is not required to make use of this. Once you configure and boot up the esphome container, you don’t need to use the CLI to run ESPhome or PlatformIO commands. Instead, use the web GUI to configure, build, and upload your ESPHome projects. For coding your C++ components, use whatever text editor (or fancy IDE) that you want.

EDIT: Regarding testing, I have no idea how to test external components, other than building and running on actual hardware. I’ve been experimenting with platformio unit tests, but I can’t get them to work natively on the desktop. If you want to test components that use ESPHome libraries, it looks like you have to build your tests as actual firmware and upload them to your devices.

UPDATE 2025-02-02: Since my above edit, I figured out how to code and run unit tests on my external component – both on the host system (docker esphome/esphome), and on the esp32 hardware. This is all done using the platformio test on my external component. I have no vscode or any other IDE to make this work, just docker, terminal and a text editor. The basic requirements are:

  • Writing tests in c++ (I’m using the Unity testing framework)
  • Configuring the platformio project directory and platformio.ini file
  • Configuring a docker compose.yml file
  • Installing build-essential within the esphome/esphome container. (I do this every time I bring up the container, but one could easily create a custom docker image based on esphome/esphome with build-essential pre-installed).

As mentioned above, I have two testing environments: native and esp32. The native environment compiles the platform agnostic parts of my component and runs tests on them within the docker container. The esp32 environment compiles the entire component, including the esp32 and esphome code that will only run on the esp32 device. The result is the familiar directory of esphome firmware files. I manually upload the firmware to my device, and the testing output shows up in the log (terminal output, web.esphome.io, or whatever you’re using to view logging output from the esp32).

The difficult part here was figuring out how and where to put the esphome source code so platormio could use it to build the test firmware based on a platformio project. When you build esphome from esphome (cli, vscode, web, whatever), esphome assembles (copies source code into) the platformio project directory behind the scenes, and it dynamically copies only the esphome source that is actually needed for your build. Esphome also configures the platformio.ini file and any env variables that need to be set.

If you manually assemble a platformio project directory and try to build esphome firmware, you have to know what esphome source code to copy into the src/ directory, and you have to manually configure the platformio.ini file and/or env variables. To get the platformio project working correctly, I copied the src/ directory, platformio.ini, post_build.py, and partitions.csv files from my .esphome/build/<my-component>/ directory, where esphome has already built a basic firmware using my component.

My external component directory doubles as a platformio project, so everything I need to build and test the component is in one place and committed to the git repository. My goal is to make this component a good example of how one might go about creating, coding, and testing an esphome external component. I’m pushing my dev branch to github every few days (when I’m working on this), so feel free to poke around.

2 Likes
15

Now I’m pretty new to EspHome, but I’ve been developing software professionally for 30 years. Adding on to this thread, in order to develop espHome devices in a more “traditional” development process, I feel like some things are missing. It may be they exist (and I would REALLY appreciate a how-to!) or they need to be created (and I would be happy to develop some of these components, but I’d need some architectural discussions first).

  • The ability to “compile in the IDE”. That is, transform YAML into C++ and then build it locally. For me, this is VSCode & PlatformIO.
  • The ability to run the compiled code in a simulated runtime environment, not actual hardware.
  • The ability to have a library of unit tests that the IDE can execute against the compiled runtime.
16

That’s possible using the host platform. No idea how you would configure VSCode to compile and run it though.

1 Like
17

I wasn’t familiar with the host platform, so thanks for the tip.

I’ve been using the VSCode terminal, and either running

esphome compile .\wallController.yml

which only compiles the code, or

esphome run .\wallController.yml

which compiles it and uploads it to the target. I guess I just need to change the target from the ESP device I’ve developing against to the host. I’ll post again with the solution if end up figuring it out.

1 Like
18

I would like to help people who are having the same problem.

Steps to write and test compile an external component:

  1. Install Visual Studio Code.
  2. Install Git on your machine.
  3. Install the “Dev Container” extension from the marketplace.
  4. Fork the esphome repository on GitHub.
  5. Create a branch of your fork called SEN5X, for example.
  6. Clone the forked esphome repository to your machine.
  7. Open the cloned repository folder in VS Code.
  8. VS Code should prompt you to use the included dev container. Accept it and wait.
  9. Once the installation is done, open the cloned repository in the dev container by typing: >Reopen in container.
  10. Write your external component.
  11. Create a symbolic link for Python by typing ln -s /usr/bin/python3 /usr/bin/python.
  12. Write YAML test files and put them in the components folder.
  13. Execute the test_build_components script with the -c parameter indicating the folder of your external component.
3 Likes
19

How do you keep track of the changes of your component in this way?

20

I was hoping this thread would provide some useful insights. I was especially hopeful when I saw the link to DrCoolzic/ESPHomeCompDev: ESPHome custom component design tutorial

But, then I looked closer. The information in DrCoolzic’s site is very good, but the most important parts are missing. I remember being excited the first time I saw it, but then also very disappointed when I discovered there was a lot of missing information.

The docs don’t make it clear that Windows is not a supported development environment (at least it was not clear to me). Here was the process that I followed:

  1. Install Python (3 from the Python org) - default install, disable path length limit, does NOT add to path, but usable from Start.
  2. Install VS Code - default/user install, add links for VS Code in explorer, add extensions for Python and Jupyter
  3. Install Git - there could and should be a whole thread on just this, because there are a lot of choices and some of them matter a lot
  4. After a couple of months I wised up and added Python to PATH
  5. Install pipx (using pip) - I have been a longtime fan of pipx for installing tools into their own environment easily. I will likely be moving to uv over the next few months as it is maturing nicely.
  6. Use pipx to install esphome - pipx install esphome

I am doing my development locally using external_components: with the local option. This makes development easy. At some point, I will push it to github and use it the more normal way. That is not really the hard part.

The hard part is understanding what is the “right” way to think about components. The current documentation basically says this is outside the scope of the documentation and just look at other components to see how they have done it. Problem is there are MANY components and it is not clear which one to use and if that one is really doing things in a good way or not.

As an example, I wanted to add vibration to the mpu6886 component. I didn’t have a clear idea of what I really wanted. The Zigbee sensor I was modeling uses a binary value for vibration. I had no idea how to do that with the code in the mpu6886 component and was unable to find any examples of other components.

Okay, it doesn’t have to be binary so I added another (regular) sensor for vibration. It seemed like I should use device_class=DEVICE_CLASS_VIBRATION, That did not go so well. I got a nonsensical error that took me a long time to figure out. This is because DEVICE_CLASS_VIBRATION is ONLY valid for binary sensors (for some unknown to me reason).

Okay, I don’t really care about the device class, since I am not likely using this with HA. With that I was able to make progress. Since I like to understand WHY things are the way they are, I noticed that there are constants for CONF_ACCELERATION_X and the other axises and the same for the gyro, but they are not used (not even in the mpu6850, which likely came first). WHY?

My guess is there is no good reason for this. I didn’t dig into the git history enough to see which came first. But this is where some good documentation (perhaps some thinking is necessary too) about what are the better patterns to follow would be helpful.

@distante I use VS Code and even if you don’t commit to git history it keeps track of changes to the files which is really helpful. Of course, storing the files in a git repo (even if it is not connected to any origin) is better as it allows for helpful commit messages that make it clearer what you were thinking when you made the change.

1 Like