SUCCESS: ESPHome ESP32-C3, bme280, mqtt, and deep_sleep with HomeAssistant

ESPHome
1

I’ve just got my first ESPHome project working to my satisfaction and I thought I’d post a success story along with a bunch of tips, tricks, and traps I’d encountered along the way. I’m using an Adafruit QT Py ESP32-C3 board with a PiicoDev Atmospheric Sensor BME280. I also tried Adafruit AHT20 and Adafruit HTU31 Temperature & Humidity Sensors but they don’t seem to be supported by ESPHome yet (and don’t have the nice pressure sensor).

First, my config (sorry for the size, explanations will follow);

esphome:
  name: outside
  platformio_options:
    board_build.flash_mode: dio

esp32:
# At some point the actual board might be supported.
#  board: adafruit_qt_py_esp32-c3
  board: esp32-c3-devkitm-1
  framework:
    type: esp-idf
  variant: ESP32C3

# Enable logging
logger:

ota:
  password: "<password>"

mqtt:
  broker: 192.168.7.6
  username: "mqtt"
  password: "<password>"
  discovery: true
  discovery_retain: true
  discovery_unique_id_generator: mac
  discovery_object_id_generator: device_name
  birth_message:
  will_message:
  shutdown_message:
  on_message:
    - topic: esp/sleep_mode
      payload: 'OFF'
      then:
        - deep_sleep.prevent: deep_sleep_1
    - topic: esp/sleep_mode
      payload: 'ON'
      then:
        - deep_sleep.allow: deep_sleep_1

wifi:
  ssid: "ngurra"
  password: "<password>"
  fast_connect: true

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Underhouse Fallback Hotspot"
    password: "<password>"

i2c:
  sda: 5
  scl: 6
  scan: true
  id: bus_a

# We need to use globals to share sensor state without circular references.
globals:
   - id: temperature
     type: double
     restore_value: no
     initial_value: 'NAN'
   - id: pressure
     type: double
     restore_value: no
     initial_value: 'NAN'
   - id: humidity
     type: double
     restore_value: no
     initial_value: 'NAN'

# Example configuration entry
sensor:
  - platform: uptime
    name: "system uptime"

  - platform: wifi_signal
    name: "wifi strength"
    update_interval: 60s
    expire_after: 90s

  - platform: template
    name: "Dew Point Temperature"
    id: dewpoint_temperature
    filters:
      - lambda: |-
          return (243.5*(log(id(humidity)/100)+((17.67 * x)/
          (243.5 + x)))/(17.67-log(id(humidity)/100)-
          ((17.67 * x)/(243.5 + x))));
    icon: 'mdi:thermometer-alert'
    unit_of_measurement: °C
    update_interval: never
    expire_after: 90s

  - platform: template
    name: "Sea Level Pressure"
    id: sealevel_pressure
    filters:
      - lambda: |-
          const float STANDARD_ALTITUDE = 34.0; // in meters, see note
          return x / powf(1 - ((0.0065 * STANDARD_ALTITUDE) /
            (id(temperature) + (0.0065 * STANDARD_ALTITUDE) + 273.15)), 5.257); // in hPa
    unit_of_measurement: 'hPa'
    update_interval: never
    expire_after: 90s

  - platform: template
    name: "Absolute Humidity"
    id: absolute_humidity
    filters:
      - lambda: |-
          const float mw = 18.01534;    // molar mass of water g/mol
          const float r = 8.31447215;   // Universal gas constant J/mol/K
          return (6.112 * powf(2.718281828, (17.67 * id(temperature)) /
            (id(temperature) + 243.5)) * x * mw) /
            ((273.15 + id(temperature)) * r); // in grams/m^3
    accuracy_decimals: 2
    icon: 'mdi:water'
    unit_of_measurement: 'g/m³'
    update_interval: never
    expire_after: 90s

  - platform: bme280
    temperature:
      name: "Temperature"
      id: bme280_temperature
      oversampling: 4x
      expire_after: 90s
      on_value:
        then:
        - lambda: |-
            id(temperature) = x;
    pressure:
      name: "Pressure"
      id: bme280_pressure
      oversampling: 4x
      expire_after: 90s
      on_value:
        then:
        - lambda: |-
            id(pressure) = x;
    humidity:
      name: "Humidity"
      id: bme280_humidity
      oversampling: 4x
      expire_after: 90s
      on_value:
        then:
        - lambda: |-
            id(humidity) = x;
            id(dewpoint_temperature).publish_state(id(temperature));
            id(sealevel_pressure).publish_state(id(pressure));
            id(absolute_humidity).publish_state(id(humidity));
    address: 0x77
    update_interval: 60s

deep_sleep:
  id: deep_sleep_1
  run_duration: 1s
  sleep_duration: 54s

The first problem I encountered is the ESP32-C3 platform is pretty new and not that smoothly supported yet. I like the RISCV architecture from an academic/license/etc point of view, like the low power, and the Adafruit QT Py ESP32-C3 boards are nice. I quickly realised I needed the dev version of ESPHome (maybe it’s changed already). The ESPHome wizard didn’t know about my particular board: so I had to pick a generic one with the ESP32-C3, and it took a bit of googling to find the platformio_options:, framework:, and variant: settings to make it work.

I have no idea how to setup the dev version of ESPHome inside Home Assistant, but even if I did, I have it running on a RasberryPi3, and apparently the esp-idf framework is not quite ready to run on arm64 and even the dev version doesn’t work yet. So I have ESPHome setup and running on my Debian intel laptop, which also made plugging into the board for the initial upload easy. I run the esphome dashboard and connect to it from the browser on the same machine. The ability to edit, upload, and see logs is really nice.

Apparently you can also use the arduino framework: and it’s necessary to make the LED work on this board because it uses some neo-bus thingy that the idf framework doesn’t support yet. However, I got the impression that the esp-idf framework is the future and I didn’t really care about making the LED work.

Next was getting i2c and the bme280 working. It took a bit of digging though datasheets to find the particular i2c pins on this board for the i2c: settings, and adding the basic platform: bme280 sensor entry. I then found the neat bme280 environment suggestion and added the platform: template entries it suggests for Absolute Humidity, Sea Level Pressure, and and Dew Point Temperature. At this point is was basically working. I set up two of them and Home Assistant could see them both.

At this point I moved the board and sensors into a case I’d designed in onshape and 3D printed. This quickly revealed that without using deep_sleep to reduce power there was a definite ~1degC temperature offset. I guess I could redesign the case to thermally partition the sensor and micro, but I was also considering battery-power and deep_sleep seemed like the right answer. This added a heap of gotchas.

First, you quickly discover that the Home Assistant API is deep_sleep-angry. This API has HA polling the device, so the device needs to stay awake long enough for HA to catch it, which pretty much undermines the whole point of deep_sleep. The alternative API with the right “device-polls-server” needed for deep_sleep is mqtt. So the api: setting had to go and was replaced by mqtt:.

At this point, for some strange reason my ESPHome started having problems uploading the new firmware. I was getting some kind of “timeout verifying firmware size” type error. I have no idea what caused it. I initially thought maybe the firmware was too big, but that can’t be right. In the end fiddling with the config, disabling things and retrying until it worked seemed to resolve it, and I’ve had no problems since.

Setting up HA with mosquitto and the mqtt addon was a bit fiddly, but mostly went according to the instructions. These instructions said to add a user for mqtt, so I did, and setup the mqtt addon to login using that user, but later on I removed/re-added the addon (trying to clear stale things in HA) and it was setup using a completely different login. both worked. Initially I setup the mqtt: entry with empty username: and password: entries but that gave errors connecting to the mqtt server. This is a private/firewalled/unimportant wifi network so I initially didn’t really want the hassle of username/password logins, but I have no idea if it’s actually possible to setup mqtt without it.

HA gets pretty messed up when you remove, add, and rename sensors, and cleaning out stale entries seems to be mostly a case of fiddling around, restarting, trying stuff, restarting. removing and reinstalling stuff, restarting…ugh. It’s complicated by mqtt “retained” messages. That discovery_retain: true setting means discovery messages sent by devices for sensors are retained and regurgitated by the mqtt server whenever HA reconnects. This is great for quickly restoring sensors after HA restarts, but it also means it never forgets old sensors.

This is where I really needed mqtt debugging tools. In HA under “Settings; Devices & Services; Mosquitto broker; configure” you can publish and listen to messages. Listening to “#” will show you everything, “homeassistant/#” shows discovery messages, and <device>/# shows you status and sensor messages from a particular device. Note the “Retained: true” status on retained messages. The publishing support on HA doesn’t seem to support sending retained messages or specifying the qos. For that I discovered and used the following for deleting retained messages. The topic you’ll mostly want to delete is old homeassistant/sensor/<device>/<sensor>/config autodiscovery messages.

$ apt-get install mosquitto-clients
$ mosquito_pub -h homeassistant.localnet -u mqtt -P <password> -t <topic> -r -n -d

I’m pretty sure the mosquitto-clients package includes some kind of mqtt listener, but I’ve just been using the one in HA. You’ll pretty quickly get sick of repeating most of these arguments, so set up a `~/.config/mosquitto_pub file with the arguments you keep repeating. Make sure you set it readable only by yourself to keep the password secure.

The main thing that was not working at this point is HA was refusing to see the sensors on my second device. The mqtt addon was seeing the device, but none of it’s sensors. Eventually I discovered HA’s “Settings; System; Logs” which told me HA was complaining about duplicate sensors. Note I thought this might have been related to mqtt retained messages and was looking for the mosquitto logs, and these are not that. I never did find the mosquitto logs, but the duplicate sensors hint led me to the discovery_unique_id_generator: mac and discovery_object_id_generator: device_name settings.

The problem is the default sensor id generation just uses the sensor’s name: prefixed with “ESP<component_type>” as the sensor identifier, so multiple devices with the same config have sensors that clash names. Note that the device name takes the esphome: name: setting which is different, so HA sees the different devices, just not the other device’s sensors. Using discovery_unique_id_generator: mac puts the device MAC address in the name, and discovery_object_id_generator: device_name seems to result in HA putting the device name into the sensor’s “Entity ID” instead of appending incrementing numeric suffixes onto each one. So sensor.outside_wifi_strength instead of sensor.wifi_strength_2, but note that HA seems to remember the entity names so you’ll need to do the HA delete/restore addins/devices/sensors and mqtt-delete-retained dance to make it pick them up.

This is getting long… perhaps I’ll post this now and followup with more in other posts…

4 Likes
2

The next step was deep_sleep. Adding this is potentially scary, because if you mess it up you can have get your device sleeping so much you can’t update the firmware any more (at least no OTA, I assume USB would still work). This is where the mqtt: on_message: topics that trigger deep_sleep.prevent: deep_sleep_1 are really important, and you need to add them at the same time that you add the deep_sleep: entry. Also make sure your run_duration: is long and your sleep_duration: short initially to be safe. You can then toggle the deep_sleep mode OFF for firmware updates and then back on with;

$ mosquitto_pub -t esp/sleep_mode -m "OFF" -r
<wait for device to wake up and stay awake, and then update firmware>
$ mosquitto_pub -t esp/sleep_mode -m "ON" -r

Note these are -r retained so the device gets them when it wakes up and connects to the mqtt server. Note that turning sleep mode back on typically immediately triggers sleeping because the “run” time is well past expired when you do it. It’s really nice the that ESPHome dashboard logs viewer just works through the whole sleep/wakeup process and you can watch it happening. I was half expecting it to drop-off and not recover when deep_sleep started.

There appears to be some kind of 5min “that firmware update worked, remember it’s OK” thing that gets logged, so maybe its a good idea to wait for that message after firmware updates before turning sleep_mode back on. However, I occasionally got impatient and it didn’t seem to matter.

Then I noticed that all my sensors become “unavailable” when the device enters deep_sleep. This is why you need to add the birth_message: and will_message: settings to turn off availability reporting. My first attempt to do this I thought it would be neat to change the <device>/status message to sleeping instead of offline but I swear it kept giving me extra offline messages after whatever I’d set in the will_message:. At the time I was trying this I hadn’t yet fully grokked the implications of mqtt “retained” messages and was floundering a bit, but even thinking back now it doesn’t make sense. I seemed to still get it for a few attempts after setting birth_message: and will_message: empty to turn them off. In the end after floundering around and retrying things a few times they went away, and I gave up any sort of attempt for meaningful status messages without triggering “unavailable” sensors. Just turn them off… it’s easier.

Throughout the whole experience there were quite a few moments where “try it again” seemed to fix the problem and it was never clear what was different about the second attempt that made it work. On a few occasions doing the build would spit out errors that seemed related to features I’d turned off, and “Clean Build Files” and retrying would fix it. I started to suspect that perhaps not cleaning the build didn’t always break it, but maybe it left vestigial features from the old config in the generated firmware. Maybe I still got offline messages because I hadn’t cleaned them out of the build? I started to develop a superstitious “Clean Build FIles” twitch after any moderately significant config change or mysterious behaviour.

Related to the “unavailable” problem I added the update_interval: 60s and expire_after: 90s settings. TBH I don’t fully understand exactly what expire_after does or how it works. Is there some kind of TTL or Expiry for mqtt “retained” messages? In any case what I wanted was updates every 60s that would be live on the HA dash for up to 90s, so samples were valid for 1.5x my polling interval to avoid any “unavailable” gaps unless the device went away for more than the polling interval.

After this the final tuning was to figure out how long to run and sleep for. The docs suggested you need to run for long enough for it to do everything you want it to before sleeping, and this can take a while. There are lots of docs on how to minimize wakup time and power used, but many of them are not ESPHome focused. The main one that was easy was the wifi: fast_connect: true setting which avoids rescanning wireless networks on wakeup by storing the last connection in rtc memory. Using a static IP for the mqtt server avoids DNS lookups. I decided against using a static IP for the devices to avoid DHCP setup because I wanted to avoid having to manually manage them all. I looked for and couldn’t find any sort DNS and DHCP cache in rtc memory that could also avoid these overheads. Someone should totally implement that. I noticed that the total wakeup-to-next-wakeup time was run+sleep+5secs, so there seems to be almost exactly 5secs setup/shutdown time overheads on top of the run and sleep time. So for a 60s output interval I needed run+sleep=55secs.

After that general setup tuning, we need enough to to read and publish the sensors. Looking at the logging it seemed like the sensor outputs were taking sometimes more than 30s after the device had finished “waking up”. That seemed pretty terrible, to get all the sensors I’d have to sleep less than 50% of the 60s polling interval? I though maybe setting a minimal update_interval: 1s would reveal the fastest the sensors could be polled. To my surprise the answer seemed to be “faster than 1s”, and checking the bme280 data sheet confirmed it should be easily polled multiple times per second even with 16x oversampling. Note the datasheet also gave recommended oversampling and irr settings for my kind of application (weather monitoring) which made me tune my bme280 settings down.

So why was it taking so long to spit out the first samples with update_interval: 60s? Looking closer at the logs I noticed that actually the bme280 and wifi_signal sensors were spitting out their first reading very fast, before even the rest of the device had finished its “wakeup” procedure. It was only the platform: template derived sensors and the uptime sensors that seemed to start sending outputs at a random time after startup. This kind of random-offset starting is often a good idea to avoid a “synchronized spike” of traffic when all devices/sensors try to send their data at the same instant. However, if I set my deep_sleep run_duration: anywhere less than 10s I pretty much never got to read those sensors.

So I started to try and figure out ways to make the platform: template derived sensors spit out their results ASAP after wakeup. I could just set update_interval: 1s but that made them spammy as hell when deep_sleep was turned off. What I really wanted was for them to only spit out a value immediately after the bme280 sensor values they depended on where read.

I stumbled onto id(<sensor>).publish_state(<value>); and on_value: in the docs and decided I could change the platform: template derived sensors to be a lambda-filter that eg calculates the dew-point-temperature from the current temperature, and have the actual temperature sensor publish_state(x) when it had a value. Unfortunately this didn’t work and triggered a circular dependency; the bme280 sensors depended on the template sensors to publish_state on them, and the template sensors depended on the bme280 sensors to get the other values (eg humidity) needed to calculate the template value.

The circular dependency was fixed by using globals: to store the bme280 sensor values. They get set on_value: by the bme280 sensors, and get read by the template sensor’s filters when the bme280 sensors publish_state() to them. Note that the template sensors actually depend on multiple bme280 sensor values, and the filters will not produce a valid value unless all the values they depend on have been read. I wanted to initialise the globals to NaN so the filtered template value would be NaN if any of the dependency values had not yet been read. I first tried nan and then NaN and got build errors. A C++ search found nan("") which worked, but I then discovered NAN worked which seemed more ESPHome-consistent.

At first I considered making every bme280 sensor triggering publish_state() on every template sensor that depended on them and adding a debounce: 0.1s filter to make it only publish the last value after all the bme280 sensors it depended on had been read. However, I noticed that the bme280 sensors were always read/published in the order listed, so I only needed the last sensor to publish_state() on all the template sensors. I could have made the filters ignore the published value and just use the globals as inputs, but left it the way it was. Adding update_interval: never means the template sensors never publish values on their own, only when publish_state() is called by the last bme280 sensor.

This finally meant the template sensors were being published immediately, and I could set the run_duration: 1s and still get all the sensors. I suspect I could even use run_duration: 0s and it would still work, since they seem to be published before the “wakeup” is complete. It’s now running for only 6s per minute or 10% of the time, and the temp readings in the case are pretty good.

So… that’s it. I’m happy to answer any questions and am wide open to suggestions on how to do this better.

3

Oh Yeah, a final tip. After all this, to make your sensors appear in the auto-generated HA overview, go to “Settings; Devices & Services; Mosquitto broker; N devices” and open the device. You should see all the sensor readings there. Edit the device by clicking on the pencil and assign it to an “Area”. That should do the trick. Note you can leave the “Name” field blank/alone and it will just stay as the device name.

4

So after initially posting this success story, I moved one of the devices outside where the wifi was weaker, and it started having problems.

It turns out that 5s startup time I observed is highly dependent on wifi signal strength, and when it’s weak that 1s sleep time that worked fine before started to be insufficient. It also looks like the power usage, and thus heat generated is higher when the wifi is struggling too, as I observed a bigger temp delta when running with sleep disabled. In the end the wifi was so weak I couldn’t even push firmware updates, so I solved it by putting another access point closer.

However, in trying to debug this I realised that all that nice logging in the ESPHome dashboard is coming over mqtt, and it’s chatty as hell! All the stuff about static IP’s and to minimise DNS and DHCP traffic to speed up startup and minimise power is pretty irrelevant when there’s a massive amount of mqtt chatter happening all the time. I know, a bit of a Doh! moment.

So I started trying to add the ability to turn logging on/off with a mqtt topic, so I could set them nice and quiet by default, but still turn it on if I needed it. This is when all the strange intermittent oddities started to come back. I added this under the mqtt: on_message: section after my existing esp/sleep_mode handlers;

  on_message:
   ...
    - topic: esp/log_level
      payload: 'NONE'
      then:
        - lambda: |-
            ESP_LOGD("main", "Setting log level NONE.");
            id(mqtt_client).set_log_level(ESP_LOG_NONE);
    - topic: esp/log_level
      payload: 'DEBUG'
      then:
        - lambda: |-
            ESP_LOGD("main", "Setting log level DEBUG.");
            id(mqtt_client).set_log_level(ESP_LOG_DEBUG);

And all of a sudden, my esp/sleep_mode handlers stopped working! I couldn’t turn off sleep mode any more! In a panic I changed my sleep mode to run for 50secs, sleep for 10secs and after repeated attempts managed to catch the tiny 1s run window and push this new firmware. I then started playing with it to try and figure out what was wrong.

Strangely, my esp/log_level on_message: handlers appeared to be working, only the esp/sleep_mode handlers weren’t. I thought “maybe there’s a limit on how many handlers you can have and only the last two count?”. So I re-ordered my handlers added logging for all handlers and now the esp/sleep_mode handlers worked and the esp/log_level handlers didn’t. Then I started playing around with retained vs unretained messages, and it appears that all the handers are working when you send messages randomly while the device is running, just not all of them for retained messages at boot/wake time. The other thing I noticed is the 2 working handlers were actually being triggered twice at boot/wake time.

After looking at the bme280 code for a bit to try and figure out another problem I saw there was quite a bit of extra verbose logging I was missing out on, so I bumped up the logging_level to VERBOSE to see more. This started to reveal what I think is going on.

First, the multiple triggering of the working on_message: handlers seems to be because the topic subscriptions are sent per - topic: entry, even if there are multiple entries for the same topic with different payload: values. This means the retained esp/sleep_mode message gets sent by the mqtt server and handled twice. The non-working handlers also get subscribed twice, but never seem to get the retained message or get handled. But there is an interesting bit of verbose logging in between the subscribe requests and the working subscribe messages being handled;

[15:53:47][V][component:199]: Component bme280.sensor took a long time for an operation (0.14 s).
[15:53:47][V][component:200]: Components should block for at most 20-30ms.

So I suspect what is happening here is the bme280 sensor is blocking for too long when doing the initial readings which means some incoming mqtt messages get lost. There’s probably a receive buffer that fits the first two subscribe messages in, but because they don’t get processed while the bme280.sensor is blocking they get overwritten by the second to subscribe messages. This is why the last two - topics: handlers in the config worked.

Note that most of the time the slow bme280 sensor is not a problem, since after boot fetching and sending sensors is unlikely to collide with incoming mqtt messages. It’s only during boot that it unfortunately ends up causing some incoming subscription messages to be lost.

So, how to fix? Well, for starters maybe ESPHome should be fixed so that it only subscribes to topics once, even if there are multiple - topic: entries for the same topic. This would help reduce the amount of unnecessary outgoing and incoming mqtt chatter, and reduce the chance of incoming messages being dropped when there are too many to buffer and the processing gets a bit backed up.

Second, the bme280 sensor should probably not block for that long. Looking at the code it seems that it triggers the measurement in “forced mode” and then tries to immediately read the result with a timeout set based on how long the measurement should take. I believe the sensor effectively blocks the i2c read until the measurement is done, and this is the source of the blocking. It would be good if instead this code did a “sleep” (where sleep means return control to other threads/processes/whatever) for the min-measurement time (which is in the datasheet) before trying to read the result. I’m not sure if the ESPHome codebase supports that kind of operation, but I think defer() might be it.

Also note that the blocking time is dependent on the oversampling, so minimising the oversampling will also minimise the blocking and probably reduce this problem.

Another workaround to minimise the problem would be to interleave the - topic: handlers so the subscription calls for different topics are interleaved and thus less likely for both responses for the same topic to be dropped together. Or, only have a single - topic: entry for each topic so we only subscribe to each topic once and use a lambda to look at the message contents to do different things.

We could turn off the sensors by default (update_interval: never) and only turn them on manually after the bootup sequence is done so they don’t interfere with receiving the subscription messages.

We could never retain mqtt topics set to the default behaviour, and instead clear them and only retain them when set different from the default. This would mean we only get responses when subscribing for the topics that are set, so theres less chance of them being dropped.

We could change the qos on the retained mqtt messages to require an “ack”. So qos=1 will I think mean the server repeatedly sends the messages until the client responds with an “ack”.

After I get this sorted I’ll re-tackle how to turn off logging in a nice way, and re-publish my updated config.

1 Like
5

nice write up! I picked up a few tips.

6

Time for an update. I’ve got the logging levels set by mgtt setup, but there are still some issues that appear to be esphome bugs that I’m going to start working on and submit some pull requests.

My config is currently this;

substitutions:
  devicename: outside

esphome:
  name: ${devicename}
  platformio_options:
    board_build.flash_mode: dio
  includes:
    - loglevels.h

esp32:
# At some point the actual board might be supported.
#  board: adafruit_qt_py_esp32-c3
  board: esp32-c3-devkitm-1
  framework:
    type: esp-idf
  variant: ESP32C3

# Enable logging
logger:
  level: VERBOSE

ota:
  password: "<secret>"

# We need to use globals to share sensor state without circular references.
globals:
   - id: log_level
     type: int
     restore_value: yes
     initial_value: ESPHOME_LOG_LEVEL_WARN
   - id: temperature
     type: double
     restore_value: no
     initial_value: 'NAN'
   - id: pressure
     type: double
     restore_value: no
     initial_value: 'NAN'
   - id: humidity
     type: double
     restore_value: no
     initial_value: 'NAN'

mqtt:
  id: mqtt_client
  broker: 192.168.7.6
  username: "mqtt"
  password: "<secret>"
  discovery: true
  discovery_retain: true
  discovery_unique_id_generator: mac
  discovery_object_id_generator: device_name
  birth_message:
  will_message:
  shutdown_message:
  on_connect:
    - lambda: |-
        ESP_LOGW("main", "Restoring log level %s.", LOG_LEVELS[id(log_level)]);
        id(mqtt_client).set_log_level(id(log_level));
  on_message:
    - topic: ${devicename}/log_level
      then:
        - lambda: |-
            ESP_LOGW("main", "Setting log level %s.", x.c_str());
            for (int i=0; i < LOG_LEVELS_LEN; i++) {
              if (x == LOG_LEVELS[i]) {
                // Save the log level for restoring after wakeup.
                id(log_level) = i;
                id(mqtt_client).set_log_level(i);
                break;
              }
            }
    - topic: ${devicename}/sleep_mode
      then:
        - lambda: |-
            ESP_LOGW("main", "Setting sleep mode %s.", x.c_str());
            if (x == "ON") {
              id(deep_sleep_1).allow_deep_sleep();
            } else {
              id(deep_sleep_1).prevent_deep_sleep();
            };

wifi:
  ssid: "ngurra"
  password: "<secret>"
  fast_connect: true

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Underhouse Fallback Hotspot"
    password: "<secret>"

i2c:
  sda: 5
  scl: 6
  scan: true
  id: bus_a

# Example configuration entry
sensor:
  - platform: uptime
    name: "system uptime"

  - platform: wifi_signal
    name: "wifi strength"
    update_interval: 60s
    expire_after: 150s

  - platform: template
    name: "Dew Point Temperature"
    id: dewpoint_temperature
    filters:
      - lambda: |-
          return (243.5*(log(id(humidity)/100)+((17.67 * x)/
          (243.5 + x)))/(17.67-log(id(humidity)/100)-
          ((17.67 * x)/(243.5 + x))));
    icon: 'mdi:thermometer-alert'
    unit_of_measurement: °C
    update_interval: never
    expire_after: 150s

  - platform: template
    name: "Sea Level Pressure"
    id: sealevel_pressure
    filters:
      - lambda: |-
          const float STANDARD_ALTITUDE = 34.0; // in meters, see note
          return x / powf(1 - ((0.0065 * STANDARD_ALTITUDE) /
            (id(temperature) + (0.0065 * STANDARD_ALTITUDE) + 273.15)), 5.257); // in hPa
    unit_of_measurement: 'hPa'
    update_interval: never
    expire_after: 150s

  - platform: template
    name: "Absolute Humidity"
    id: absolute_humidity
    filters:
      - lambda: |-
          const float mw = 18.01534;    // molar mass of water g/mol
          const float r = 8.31447215;   // Universal gas constant J/mol/K
          return (6.112 * powf(2.718281828, (17.67 * id(temperature)) /
            (id(temperature) + 243.5)) * x * mw) /
            ((273.15 + id(temperature)) * r); // in grams/m^3
    accuracy_decimals: 2
    icon: 'mdi:water'
    unit_of_measurement: 'g/m³'
    update_interval: never
    expire_after: 150s

  - platform: bme280
    temperature:
      name: "Temperature"
      id: bme280_temperature
      oversampling: 2x
      expire_after: 150s
      on_value:
        then:
        - lambda: |-
            id(temperature) = x;
    pressure:
      name: "Pressure"
      id: bme280_pressure
      oversampling: 2x
      expire_after: 150s
      on_value:
        then:
        - lambda: |-
            id(pressure) = x;
    humidity:
      name: "Humidity"
      id: bme280_humidity
      oversampling: 2x
      expire_after: 150s
      on_value:
        then:
        - lambda: |-
            id(humidity) = x;
            id(dewpoint_temperature).publish_state(id(temperature));
            id(sealevel_pressure).publish_state(id(pressure));
            id(absolute_humidity).publish_state(id(humidity));
    address: 0x77
    update_interval: 60s

deep_sleep:
  id: deep_sleep_1
  run_duration: 1s
  sleep_duration: 54s

Which also depends on this loglevels.h file:

// This is defined in logger.cpp but is not publicly visible, so we need to redefine it. Sigh.
const char *const LOG_LEVELS[] = {"NONE", "ERROR", "WARN", "INFO", "CONFIG", "DEBUG", "VERBOSE", "VERY_VERBOSE"};
const int LOG_LEVELS_LEN = sizeof(LOG_LEVELS)/sizeof(char*);

One of the bigger changes is I’ve changed the mqtt messages to be prefixed with the device name instead of esp/.... This lets me toggle sleep_mode and set log_level independently per device with this (note I have -h, -u, and -P arguments in ~/.config/mosquitto_pub);

$ mosquito_pub -t <device>/sleep_mode -m "OFF" -r
$ mosquito_pub -t <device>/log_level -m "DEBUG" -r

In order to implement this without having the device name scattered everywhere in the config I’m now using substitutions: and ${devicename} throughout the config. This is a pattern I think I’ll always use, and I may even switch to using the new include with vars feature to reduce duplication even more.

I changed the mqtt: setup to have only a single on_message: message entry per topic in an attempt to fix the problem with receiving duplicate and missing retained messages. This involved switching from yaml topic handlers to C++ lambdas checking the message contents. I also added an on_connect: entry to set the initial logging level (reasons why later). This resulted in me needing to share a constant string array between the the on_connect: and on_message: handlers. Unfortunately you can’t just declare something in one lambda and use it in a following lambda, as they seem to have different scopes. I could have shoved them into globals: but that seems designed for variables which means it would end up in RAM. A const array of const char* strings is something compilers can optimize into the code, which means ends up in FLASH and doesn’t eat RAM. Note that this particular array is already declared inside the mqtt C++ code but it’s private, which means we need to re-declare it. It turns out you can use includes: to include arbitrary *.h files and link in *.cpp files, and these are visible to all lambdas, so the loglevels.h include was added. I know putting string array in a header is a bit hacky, but it works.

In general I’m finding myself move more and more away from coding in yaml to C++ in lambda’s and now C++ includes. I expect I’ll probably start doing that more and more, despite my desire to try and stick to the simplest way to use esphome.

Adding the logging stuff was… interesting.

The first trap for beginners I hit was I’m an old C guy with less C++ experience and I stumbled into the difference between char * and string. It turns out the ‘x’ message value argument is a string, so you need to x.c_str() it when logging it with a "%s" format string. It’s also nice but a little surprising that == has been overloaded to compare the string contents, not just the pointers, and can even compare with a char * strings.

The next nasty trap is there are ESP_LOG_LEVEL_DEBUG style constants for the log levels. These are defined by the ESP-IDF libraries. These are almost but not quite the same as the ESPHOME_LOG_LEVEL_DEBUG constants used by ESPHome, which adds a CONFIG level in there. It took me ages to figure out why switching to DEBUG level turned off all my debug logging. In the end I stopped using these constants at all and use the LOG_LEVELS array used internally by mqtt logging, which also gives a handy mapping from string to int levels.

The other potentially confusing thing is there are actually 2 layers to the logging; the underlying logger: component, and the logging support in mqtt:. In logger: you can set the logging level on a per “component” basis, and it doesn’t appear to support easily changing the default level for all components. The logger: level: setting sets both the default level and the highest level enabled in the compiled binary (it optimizes away all the logging entirely for lower levels). So if you are dynamically setting log levels via mqtt you need to set this to the lowest level you want to support.

The mqtt: logging works by adding a mqtt message sending hook into the logger: handler, and it supports setting it’s own global log level. Note that this means that the logger: level is the level you would see using the USB interface, and the mqtt: level can “prune” those logs to only send higher levels over mqtt. You can set the default mqtt log level in yaml, but it also requires setting the log_topic: which I just wanted to leave as the default. In the end I chose a different way to set the default because…

Despite switching to only one on_message: handler per topic, I just could not make anything other than the last handler correctly receive/handle a retained mqtt message. The log_level handler just never worked…well… I think maybe I saw it work once, but I’m not sure. Playing around with mqtt qos didn’t help, it didn’t seem to do anything… well… I think I maybe once saw the sleep_mode handler get triggered twice which kinda looks like something qos=1 might have caused, but I’m not sure. I’m not sure my theory of the slow bme280 code is enough to explain this problem and suspect there’s some more fundimental problem with multiple subscribe topics. I’ll be doing some more debugging and filing a bug (and maybe a pull request) soon. Always put your sleep mode on_message: handler last, because it’s the only one that works.

This meant the log_level could only be changed by re-sending the <device>/log_level message after turning off sleep_mode, which would be immediately forgotten and reset by the sleep/wakeup when sleep_mode was turned back on again. Then I had the brainwave of putting the log_level into RTC memory by using globals: restore_values: yes and restoring it in on_connect:. This meant the log level could be changed after toggling sleep_mode off, and then it would “stick”, which is great for turning on detailed logging early in the boot process.

There are still some extra slight questions, like why does the logging in on_connect: never show, but otherwise this is working pretty well. Except for the other minor problem BME280 suddenly starts reporting pressures around 700hPa instead of 1000hPa after deep sleep. · Issue #3383 · esphome/issues · GitHub which I’ll also be debugging and submitting a pull request for.

1 Like
7

FTR, I’ve filed 2 bugs and submitted one pull request as a result of these efforts;

I’ve not yet debugged/fixed the mqtt problem, but I have fixed theBME280 problem. I did have a look at the mqtt code but it’s much more complicated than the BME280 stuff so I haven’t tried to tackle it in detail though. What I do have is a repeatable test-case that shows the problem though.

However, with my BME280 fix applied it’s now working fine with the configs and workarounds I posted above. I’m using it to drive an under-house ventilation fan depending on inside/outside humidity values. If anyone want’s more details on the HomeAssistant setup for that just ask here.

1 Like
8

This is helpful, thank you. I am not able to get my QT PY C3 going with your yaml. Can you let me know if you have updated anything now?

9

This works better:

esphome:
  name: ktype
  platformio_options:
    board_build.f_flash: 40000000L
    board_build.flash_mode: dio
    board_build.flash_size: 4MB
#  includes:
#    - loglevels.h
    
#esp32:
#  board: esp32-c3-devkitm-1
#  framework:
#    type: esp-idf

#works better
esp32:
  board: esp32-c3-devkitm-1
  # adafruit_qtpy_esp32s3_nopsram  #5.0.1
  framework:
    type: esp-idf
#    type: arduino
#    version: 2.0.4
#    platform_version: 5.1.1
  variant: ESP32C3


# Enable logging
logger:
#  level: VERBOSE
 
spi:
  miso_pin: GPIO8
  clk_pin: GPIO10

sensor:
  - platform: max31855
    name: "KType Temp"
    cs_pin: GPIO20
    update_interval: 60s
10

Thank you for sharing. It seems to me C3 is excluded from ext1/touch wake-up. Am I missing something?

11

For the record, I’m using these devices;

This is the ESP32-C3, BME280, and cable I used;

And here is another ESP32-C3 board that I also bought but haven’t yet used that includes a LiPo charger so it could be easily battery powered, but some soldering would be required;

Different boards may require slightly different yaml configs, but the ones posted worked for me. Note that for the initial firmware load I connected the device via USB3 directly to my laptop running ESPHome, but after that I could use OTA updates. I also found the ESPHome running on the Raspberry Pi could not compile the firmware because the esp-idf stuff didn’t work for cross-compiling on ARM, but that might have already been fixed.

My fix for the BME280 has already been accepted upstream and pushed to Home Assistant. I’ve not yet figured out a fix for the mqtt subscribe problem… the code there is a bit more complicated. Thankfully the workarounds I’ve figured out do the job.

I think there is something about touch wakeup being missing from the C3, but it might depend on the board you use. I’ve not missed it because I don’t use it.

12

Time for another update;

The cross-compiling esp-idf stuff for RISCV on ARM now definitely works, and it’s possible to compile and push updates over wifi from ESPHome running under HomeAssistant on a RaspberryPi. I’ve been doing it on a RaspberryPi3 and it’s a bit slow (particularly the first time when it pulls updates and compiles everything), but it works.

There is a fix for the problem where mqtt subscribe only works for the last topic: in your config on wakeup from deep sleep in Fix use of dangling pointers in esp-idf MQTT backend by aaliddell · Pull Request #4239 · esphome/esphome · GitHub. This means you won’t have to turn off deep_sleep mode before you can toggle any other mqtt topic settings with mosquitto_pub.

This fix is not yet merged so until it is merged and live you’ll have to explicitly include it by following the instructions at mqtt on_message: handling misses retained messages on wakeup/reboot. · Issue #3406 · esphome/issues · GitHub and put a special external_components: entry in your config.

1 Like
13

Hello, great job. Please could you post a picture of the 3d printed case?

14

Unfortunately both the cases I printed are in hard-to-get-to places so I’d have to print out another just to take photos. In the mean time you can go to this URL and see/pan/spin around the 3d model (you don’t need an account or anything, just a capable enough browser);

https://cad.onshape.com/documents/baf51b8ac02b7b9f4861e5d3/w/5772eb2dea0568e6ffa50594/e/892ed19264eaa753e6f9f057

I also put the STL files on thingyverse here;

1 Like
15

Really appreciated your dedication to the project. Would there be a way you can post the final code along with a schematic/parts list. Just for those who may be intimidated to try this. It really was alot, im expetenced to a point in these matters, and it was alot to digest. I would like to simply COPY or duplicate with EXACT parts. Also the Seed Xiao c3 is great. Has battery management also ext antenna, and is highly recommended for deep sleep projects. Thank you again for your hard work.!!

16

I’ve been meaning to clean it all up and put it on github, but I’m a bit busy doing a bunch of other stuff right now, so I’m not sure when I’ll get to it. When I do I’ll link to it from here.

FTR the mqtt fix for missing retained messages on wakeup/reboot is now merged, so you don’t need to worry about that any more.