Using ota.http_request.flash with deep sleeping devices

ESPHome
1

ESPHome 2024.6 has been released and with it the new option to poll new firmware OTA instead of pushing it from the ESPHome dashboard in HA:
ota.http_request.flash

While this is surely interesting to update devices in foreign networks (i.e. commercial devices with an option to update), my personal interest is mainly to improve my own update procedures for my deep sleep devices. Such devices are usually sleeping when you try to push a new firmware to them and so the update fails. The usual solution is to set some special flag (like an input_boolean or some persistent MQTT message) and then wait until all sleeping devices are awake and waiting for you to push. This is a tedious manual process with each single update and it generally defies the reason for putting MCUs to sleep, like reducing power consumption or avoiding heat to build up.

ota.http_request.flash now allows us to deposit the new firmware instead of pushing it. The sleeping MCU can then poll the new firmware whenever it wakes up and sees fit to do so.

I open this thread to discuss with you any best in breed solutions to …

  1. Compile all your ESP nodes’ YAMLs on the HA side, create MD5 hashes, and copy the resulting binaries and checksum files to some location that can be reached through HTTP(S) requests (like /homeassistant/www/)
  2. Poll the firmware regularly from the ESP side, but not too often to reduce power consumption
  3. Do any other ingenious tricks that you came up with to distribute ESPHome firmware updates to your MCUs through the new polling mechanism.
3 Likes
2

Attention, this is largely untested!
But some first steps to solve topic 1. from the list above could be as follows (all done on HA OS):

In ESPHome dashboard click “UPDATE ALL” (upper right corner).
This will compile all ESPHome nodes that are not yet up to date and will try to OTA them. The MCUs who sleep will fail in the process.

After that all firmware.ota.bin files can be found in the ESPHome docker container in paths with this name pattern:

/data/build/NODENAME/.pioenvs/NODENAME/firmware.ota.bin

(Hint: This pattern might diverge, if you used the build_path parameter in your YAML!)

To get access to the docker containers file structure you must install the HA add-on “Advanced SSH & Web Terminal” and turn off protected mode. Then open a terminal, log in to HA with

> ssh [email protected]

and use this command to open a shell in the docker instance of ESPHome:

docker exec -it addon_5c53de3b_esphome bash

Then copy all firmware files to /config/www/... to make them available over HTTP in your network.

mkdir /config/www/otabin
cd /data/build
find -name "*.ota.bin" | xargs cp --parents -t /config/www/otabin

MD5s can be created as described in the OTA docs with:

md5sum firmware.ota.bin > firmware.md5

…but I am not Linuxoid enough to add that to the find command. Maybe someone can help.
(Update: See the next posting for a solution to combine find, cp and md5sum.)

From that point on any sleeping MCUs could check in on their firmwares to update them.
The path would be:

http://homeassistant.local:8123/local/otabin/NODENAME/.pioenvs/NODENAME/firmware.ota.bin

It would great if someone could also change the find command above to remove the redundant segments from that path.

In this post @hpi has shown how to script the whole ESPHome compilation process. Maybe an extension of that would be the most elegant way to do all of the above. But again, I am not familiar enough with Linux to do that.
But it’s a start…

3

Ok, I guess, find, cp, and md5sum can be combined like this:

mkdir /config/www/otabin
cd /data/build
find -name "*.ota.bin" -print0 | xargs -0 -I myfile sh -c 'md5sum myfile > myfile.md5; cp --parents myfile* /config/www/otabin/;'

# For my own reference as a Linux noob:
# -print0 = no newline chars in output
# -0 = take care of file names with blank spaces
# -I = define a name that will be replaced by the arguments
# --parents = recreate the original path structure in the copy

Would still be nice if someone could help to remove the redunant segments from the copied paths (i.e. /NODENAME/.pioenvs/NODENAME/firmware.ota.bin → /NODENAME/firmware.ota.bin).

4

By chatgpt:

find /data/build -type f -path '*/.pioenvs/*/firmware.ota.bin' -exec bash -c '
for f; do
    full_path="$f"
    xxx=$(echo "$full_path" | awk -F/ '\''{print $(NF-3)}'\'')
    echo "Full path: $full_path"
    echo "XXX directory: $xxx"
    target_dir="/config/www/otabin/$xxx"
    echo "Target directory: $target_dir"
    mkdir -p "$target_dir"
    cp "$f" "$target_dir/firmware.ota.bin"
    md5sum "$target_dir/firmware.ota.bin" > "$target_dir/firmware.md5"
done' bash {} +