Announcement ESPHome running on Adafruit 52840 over Thread mesh networking using ZephyrRTOS

TLDR

I ported ESPHome to run on top of ZephyrRTOS. Using this I was able to build against the Adafruit Feather 52840 Express, and create an OpenThread mesh network (not WiFi) which is used to communicate with my Home assistant instance.

Slightly Longer

I have long enjoyed how easy it was to make smart home devices using ESPHome, but was always interested in moving away from Wifi (range issues, congested IPs, power consumption, etc). Because of the fantastic job @OttoWinter did in creating abstraction layers in ESPHome (as well as an overall well structured project) I was able to create appropriate shims and project structure builders to make the code produced by ESPHome run on top of ZephyrRTOS, an embedded operating system with its own highly optimized runtime, and low power consumption.

One of the major selling points of Zephyr is that it had native support for using Openthread mesh networking as an L2 transport layer, meaning code sitting “above” could work with a normal network socket system. This means it was possible to make the ESPHome api component “just work” (after many hours of head banging) to communicate with HomeAssistant. The API component can talk using standard sockets which get translated to the L2 thread network, sent across the mesh to a border router (part of the openthread project) which then is visible to the network containing HomeAssistant. The border router also advertises registered services for each microcontroller such that they are discoverable on link local.

Even though I did all this to enable thread networking, it is by no means required. By supporting zephyr as a target platform there are hundreds of boards that can be supported by ESPHome (though of course I have only built and tested against one at the moment).

What Does This look Like

esphome:
  name: zephyrboard

logger:

openthread:
  network_name: NETWORK_NAME
  panID: 0x111
  xpanID: 1111111111111111
  network_channel: 11
  network_key: <NETWORK_KEY>

api:
  password: "example_password"
  reboot_timeout: 0s

zephyr:
  zephyr_base: "<BASE_ZEPHYR_INSTALLATION>"
  board: adafruit_feather_nrf52840
  flash_args: "--runner blackmagicprobe --gdb-serial=/dev/ttyACM0 --elf-file=BUILD_DIR/zephyr/zephyr.signed.hex"


zephyr_ota:

binary_sensor:
  - platform: gpio
    pin:
      number: D9
      mode:
        input: true
        pullup: true
      inverted: true
    name: "zephyr sensor"
    on_press:
      then:
        - switch.turn_on: switch_light
    on_release:
      then:
        - switch.turn_off: switch_light

switch:
  - platform: gpio
    pin: D4
    id: switch_light
    internal: true
    restore_mode: ALWAYS_OFF

How To Use

I’m writing this whole post to see how much interest there is in what I have been working on. I don’t want to spend a long time writing a guide no one will ever look at. Not that there is a lot to it, but I would rather put my time into the code I am going to use if no one else is interested. If there is interest I can do a write up and put it in follow up comments on setting up zephyr, the border router, etc.

The code can be found here.

Who Am I?

Just a person interested in using this for my own projects, I am not associated with ESPHome or Home Assistant, I just enjoy their wonderful work.

Will This Be Upstreamed?

I hope so eventually. I hope to not be sole maintainer for this forever, and I don’t have the energy to keep up tracking upstream development all the time. That said, until more features of ESPHome are supported (devices, interfaces, etc) I dont expect that they will want to pull in partially working project, but I might be wrong.

What Works

Right now I have focused on the basics.

  • Compiling
  • Thread networking
  • GPIO
  • Logging
  • Flashing (locally and OTA)
  • Most standard features not dependent on hardware, such as automations
  • Communication with home assistant

What Doesn’t Work

I am not sure enough to make a list, basically the above list is what I have tried so far. I suspect SPI and I2C probably work in software mode (bit banging), and if they don’t it should be pretty quick to make them. I suspect anything specific to ESP projects (like camera) will take much more time, or not be possible. There is WiFi and Ethernet support support in Zephyr so in principal it should be possible to get the ESPHome components working, but that is not a near term goal for me. Library support is not working, as this is based directly from Zephyr and not platform IO. I can go into reasons if anyone asks.

What Are My Next Plans

I started all this to have some devices around my house, so what I get working next will largely be driven by those projects. I suspect A2D will be high priority so that I can monitor battery levels. Past that I plan on tacking SPI and I2C, as I will need them before long.

What are some drawbacks

The Rom sizes are not as optimized as I would like. Because OTA updates require two slots in memory (one running, and one to update to) sometimes there is a little juggling of space required. This is not so much of a problem with simple ESPHome scripts, but it is difficult to enable some of the features Zephyr provides. Currently Network logging is not working, but should not be hard to get it to.

Long Term Hopes

I would really like to take advantage of Zephyrs real time scheduler and callbacks to eliminate the need to run loop. Events could simply be acted upon as they happen. This could be a big win in terms of power efficiency and responsiveness. Currently all of this happens exactly like it does on an ESP board which helps maintain compatibility, and makes it easier to focus on making other things work.

Is This Code Stable

No it is not. I tried to keep things designed in such a way that it could grow well, but I would classify this code possibly just past “make it work” stage. I suspect there will be refactoring in how all the Zephyr components interact with the configuration and build systems before long. That said it is usable (within what works) right now.

How Can You Help

Well first if you want to play with it, that would be great, bug reports, patches, etc are welcome (just note I have a day job so I might not always be supper prompt).

If you are interested you can donate microcontrollers (or cash to buy them), or run them on your own microcontrollers. I would be happy to support many boards, but I think my partner would give me “a look” If I spent money on many many boards, and I might not be around to keep working on it.

Thumbs up and or likes, knowing people find the work interesting helps with motivation.

Thank You

If you have made it this far in reading THANK YOU. Thank you for your attention and your time reading all of this. I have worked on this for a while and other people showing interest is very heart warming.

10 Likes

Well done. I for one am very interested in this.

Sorry for not writing anything back sooner, it has been a busy week.

Last night I was able to get I2C running for both the dedicated hardware controller, and software GPIO mode. Unfortunately I don’t have any free SPI hardware lying around, so I will not be able to work on that until I am able to order something. I have started on ADC, and UART, and think I will probably have those up and going over the next week or so.

@xAPPO, thanks for expressing interest in this, if you (or anyone else following along) wants to try and reproduce what I have done I will try and make a complete How-to below. Feel free to ask questions on any part that is not clear, or if something does not work, as it is possible I have forgotten details in all my trial and error. Also note, I am new to this forum, so I can not post many links, so I will do my best.

How to


  • You will need at least 2 Adafruit Feather nRF52840 Express boards (I am going to look into adding support for other boards soon, but for now this is the only one supported. One board will function as a network coprocessor for the border router (that bridges thread to your wider network). The other will be the board you develop with.

  • You will need some sort of debug programmer segger jlink (the edu edition is way cheaper, but read the disclaimer, you cant use this for professional development), or a blackmagic probe. I have the blackmagic probe because I like to support open hardware where I can. I think there are a few other programmers that work as well.

  • Create a python virtual env (this is not strictly necessary, but you may get conflicts with other packages and you will get conflict with esphome if you do not do this) I recommend python 3.8 for this, as there is something in the zephyr SDK that seems to be linked against python3.8 libraries. I personally have my venv running on python3.9, and was able to still get everything working by installing python3.8 libraries. The actual ABI seems to be stable, the linker is just not happy otherwise [meaning you might be able to fake it with symlinks too, but that is beyond this write up]). Make sure you have activated this environment in the shell you do the later steps in.

  • Follow the zephyr getting started guide ( docs.zephyrproject[.]org/latest/getting_started ) to make sure you have the build per-requisites and install the environment. You can skip the bits about setting up your local/bin path, and don’t have to install with --user, because you are in a venv. If you install the sdk outside of the recommended paths (what I did, follow the link [install the Zephyr Software Development Kit (SDK)] link to get the environment variables you need to export. I added these to my virtual env bin/activate script, so they would automatically be setup anytime my environment is setup.

  • The zephyr build tree has various samples in it, one of which is the network coprocessor sample. This is used to build software to flash on on of your boards such that it can be used as a radio for some computer, such as a raspberry pi. See: /docs.zephyrproject[.]org/2.6.0/samples/net/openthread/coprocessor/README.html . Make sure you build this with support for thread version 1.2, as this is what the border router now requires. Here is the command I used to build the controller. west build -b adafruit_feather_nrf52840 coprocessor/ -- -DOVERLAY_CONFIG="prj.conf overlay-rcp.conf" -DCONFIG_OPENTHREAD_THREAD_VERSION_1_2=y -DCONFIG_USB_DEVICE_STACK=y -DCONFIG_USB_DEVICE_PRODUCT='"TESTING"' -DCONFIG_USB_CDC_ACM=y -DCONFIG_USB_REQUEST_BUFFER_SIZE=2048 -DCONFIG_UART_LINE_CTRL=y Once you have this build, you can use the west command (see getting started guide) to flash this onto your board, with possible modifications for whatever debugger you are using to flash with. The command I used would look like west flash --runner blackmagicprobe -d <path to where build directory was produced from previous step, likely cwd> --gdb-serial=/dev/ttyACM0 on linux, your serial device will be different on other platforms, see getting started guide

  • Plug in the network coprocessor created in the last step. Install the Openthread Border router: openthread[.]io/guides/border-router . This must be on a device local to your network. I installed mine on the same raspberry pi I have my home assistant on, which makes the networking slightly easier. If you have it on a separate host, you will need to make sure whatever device is running your home assistant has an piv6 address and route to the open thread prefix (created when you form a network via the instructions) via whatever hardware the boarder router is running on.

  • Clone the esphomeZephyr repo from my first post. In a shell with your venv setup, run python setup.py install to install esphome with zephyr support.

  • Create a yaml project from scratch (I dont have the wizard supporting zephyr yet). You can model this on what I have in my first post. Make sure you fill in the network details in the open thread block to match those you had when you formed your network in the boarder router step.

  • plug in your second board to your computer

  • run esphome compile <your-project>.yaml then run esphome flash. Currently I don’t have run option implemented so you need to do the steps separately. The first time you build it will compile and flash a bootloader and your application. When flashing ignore the first prompt about selecting your serial device, whatever you choose does not matter, it will only look at your flash_args. I have not gotten around to changing that yet, because it requires a bit of refactoring in esphome. After the first flash (assuming it all goes well with your network and such) you should be able to use network flashing (it will ask if you want to flash over the network if it finds your device). Your computer will need an ipv6 address, and a route to the prefix mentioned above (on linux this can be done with ip route add via ).

  • Enjoy

I’m sure this will need to be fleshed out more, but I am just writing out everything from the top of my head. After I get more working on the project, I hope to spend a bit more time to streamline this a bit. It may seem like a lot, but it should not actually take too long to do. If anyone gives it a shot, let me know how it goes, and maybe keep some notes so we can work on a better guide. If you do try it, thanks for giving it a shot!

Note:
Please forgive any spelling mistakes you may find, feel free to point them out to me, and I will fix them, and thank you for doing it.

2 Likes

Sorry it’s bee a few weeks since an update, holidays and all.

I thought I had an old spi device laying around, so I started working on that interface. Turns out I miss remembered, and it was an i2c device. I have everything in place for SPI support it all compiles successfully, but no way to test it. I am going to look into getting something cheap.

I am in the middle of A2D support, I paused it to work on SPI support, but hope to have it finished soon.

I am also adding support for the Nordic nRF25840 USB Dongle. its a smaller device, but it still has a decent number of GPIO pins. Best of all, it is much cheaper at only 10$, so hopefully that will make it easier for others to use this for their projects.

Once the above are done, I am going to clean things up, rebase everything on latest esphome, and try to start a discussion about integrating it upstream. If any devs of the project are reading, I would love to hear from you on what you think, and what you would like to see.

2 Likes

I will note I now have 2 devices up and running daily automation tasks running this software, and have been very happy with it so far.

1 Like

I just pushed support for the adc component. I it is successfully reading out the battery voltage attached to my adafruit nrf52840 feather express.

This component supports as many channels as the zephyr board supports, which in the case of the adafruit board is 6 channels exposed on a pin, a separate internal channel for the battery, and a channel for an external reference power. To use multiple channels, just declare multiple adc sensors with different pins.

The component supports a few different options than the ESP version. First, there is an option for which voltage should be used as a reference voltage, by default the component will use the internal reference voltage of 0.6 volts. However, 1/2, 1/3, 1/4 vdd and external reference are also supported.

The adc can only read up to slightly higher than whatever voltage is used as a reference, so there is also support for various gains; 1/6, 1/5, 1/4, 1/3. 1/2, 1, 2, 3, 4, 6, 8, 12, 16, 24, 32, 64, 128. Select whatever gain best maps the voltage to be measured to the reference voltage.

Additionally, you can select between 8,10,12,14 bits of resolution when reading the signal.

Using reading out the on board battery connector as an example; Battery is 4.2v at full charge, the board has a 2x dividing resistance built in making voltage 2.1v. I used the internal reference at 0.6v and selected a 1/6 gain so I had a maximum voltage of 0.35v on the battery read out of the possible 0.6v. Sampling does not make much difference when measuring the battery, as I only care about a course %.

Also as a complete aside, I could not find a good formulation for turning voltage into % for a 4.2v lithium polymer battery, if someone knows a good source please let me know, but in case this helps anyone, this is a pretty good approximation I found,

(3.49962210e+01*atan(8.01705286e+00*x - 3.02904337e+01)/exp(x/1.37267273e+08)+4.92184943e+01)*(0.24443444*x)+3.3667718100000004

The variable x here is in the input voltage. This seems accurate to around 2%, but battery age may affect that, especially near fully discharged.

I’m sorry if this has confused anyone when I have been talking about a2d I am referring to the analog digital converter (adc component), the term a2d comes from another life.

Finally, Just an update I am currently refactoring things and should have support for this device soon.

1 Like

The project looks awesome!!! I’ve come here looking for information about the new ESP-now protocol, but find out a better one: Thread by Google! Being a real fun of ESPHome, HA and Google, for sure I will be testing your project as soon as I have some spare time.

I started 4 years ago with IoT and got so surprised by ESP8266, that bought tens of them. Soon later I discovered that WiFi its not the best to use for sensors. That way, I moved into MySensors (Arduino + NRF24L01). With your project, I think I can now start creating my sensors with just one MC, even with the ESP8266-01 :wink:

Thanks for your time!!!

@jcataluna Thanks for your interest and kind words. I just want to note that Thread communications do no work with ESP devices, you need a micro controller with the correct radio. Currently that means the adafruit 52840, but I am almost done with support for the nordic NRF52840 dongle, which is a bit cheaper.

I hope to be finished up with support for the dongle tonight or over the weekend. I actually have ESPHome produced images running on one right now, but it was a bit hacky to upload the image, and I want to refactor all that code so it is more generic, with an eye to adding support for more boards in the near term. I am also playing around with making images smaller to leave more room for user code.

I just got in the SPI device I ordered, so I hope I get a chance to get that finished up as well. I think the last big piece to support a majority of ESPHome devices will be to get UART working.

A stretch goal will be to get Bluetooth working on the devices, as these boards support both BLE and Thread at the same time, but I have a lot of projects I want to use these boards for, and I dont need BLE anytime soon, so it might take a back seat.

Actually a ESP-H2 was announced last year by espressif which includes thread support.

ESP32-H2 combines important connectivity technologies, such as IEEE 802.15.4 radio connectivity, which is vital to mesh architectures with low power consumption. ESP32-H2 also makes the Thread and Zigbee protocols available, thus addressing a variety of cases in application development. Furthermore, Bluetooth LE supports point-to-point, broadcast and mesh communication topologies

Later the same year


ESP32-H2 Officially Recognized as a “Thread-Certified Component” and a “Zigbee-Compliant Platform”

But it looks like it’s not yet sold publicly :thinking:

Oh I had not seen that, well whenever that IS released, this should be able to support it!

In other news, I am pretty much done re-factoring the code base. I have the NRF52840 dongle able to be a target for ESPHome, it is able to produce images, upload them, and they communicate with Home Assistant etc.

The problem I am having is that they don’t seem to work with OTA updates, which is a pretty big thing. The devices will accept the new images, they just dont seem to swap them successfully on reboot. This may have something to do with the chained bootloader from nordic, or I am just overlooking something who knows.

I am going to keep poking at it today, but I will just push things in whatever state they are tonight in case people want to experiment.