ESPHome Memory issue force cycle reboots

Joao-Sousa-71 · October 15, 2023, 7:40pm

Hello all,

I’ve been facing an issue with an ESP32 (ESP32-WROOM-32) that has the following sensors connected:

CM1106 Single Beam NDIR CO2 Sensor Module (CO2 Sensor)
CUBIC PM2105 Laser Particle Sensor Module (PM1/PM2.5/PM10 Sensor)
TEXAS HDC1080 (Temperature and Humidity sensor)
SENSIRION SGP30 (TVOC/eCO2 sensor)

Everything works fine however the ESP reboots every ~5000 seconds

And after adding a memory sensor I’ve confirmed that is the memory is not released and cause the ESP to reboot

This is the ESPHome yaml:

substitutions:
  devicename: daikin-air-sensor-floor-1
  friendly_name: Daikin Air Sensor Floor 1
  device_description: DAIKIN Air Sensor BRY88AB151K

esphome:
  name: $devicename
  comment: ${device_description}
  project:
    name: "DAIKIN.BRY88AB151K"
    version: "BRY88AB151K"
  # name_add_mac_suffix: true
  includes:
    - cm1106.h
    - pm2105.h          # For PM2105 module

esp32:
  board: esp32dev
  framework:
    type: arduino

# Enable logging
logger:
  baud_rate: 0
  level: INFO

# Enable Home Assistant API
api:
  encryption:
    key: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

ota:
  password: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

wifi:
  ssid: !secret ls_ac1750_wifi_ssid
  password: !secret ls_ac1750_wifi_password
  manual_ip:
      static_ip: 192.168.1.123
      gateway: 192.168.1.254
      subnet: 255.255.255.0

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "$devicename"
    password: "xxxxxxxxx"

captive_portal:

web_server:
  port: 80
  version: 2
  
uart:
- id: cm1106_uart
  rx_pin: GPIO16
  tx_pin: GPIO17
  baud_rate: 9600

sensor:
- platform: wifi_signal
  name: "${friendly_name} Wifi Signal"
  update_interval: 30s
- platform: uptime
  name: "${friendly_name} Uptime"
- platform: template
  id: esp_memory
  icon: mdi:memory
  name: "${friendly_name} Free Memory"
  lambda: return heap_caps_get_free_size(MALLOC_CAP_INTERNAL) / 1024;
  unit_of_measurement: 'kB'
  state_class: measurement
  entity_category: "diagnostic"  
- platform: custom
  lambda: |-
    auto cm1106Sensor = new CM1106Sensor(id(cm1106_uart), 10000);
    App.register_component(cm1106Sensor);
    return {cm1106Sensor};
  sensors:
  - name: "${friendly_name} CO2"
    id: daikinco2
    accuracy_decimals: 0
    unit_of_measurement: "ppm"
    device_class: carbon_dioxide
    on_value_range:                       #CO2 LED always on when CO2 is below 1000
      - below: 1000       
        then:
          - output.turn_on: ledco2
      - above: 1000                       #CO2 LED blink slowly when CO2 is above 1000 but below 2000
        below: 2000 
        then:
        - while:
            condition:
              sensor.in_range:
                id: daikinco2
                above: 1000
                below: 2000 
            then:
            - output.turn_on: ledco2
            - delay: 500ms
            - output.turn_off: ledco2
            - delay: 500ms
      - above: 2000                       #CO2 LED blink quickly when CO2 is above 2000
        then:
        - while:
            condition:
              sensor.in_range:
                id: daikinco2
                above: 2000
            then:
            - output.turn_on: ledco2
            - delay: 200ms
            - output.turn_off: ledco2
            - delay: 200ms

- platform: custom
  lambda: |-
    auto pmsensor = new pm2005();
    App.register_component(pmsensor);
    return {pmsensor->vpm1, pmsensor->vpm25, pmsensor->vpm10};
  sensors:
  - name: "${friendly_name} PM1"
    id: daikinpm1
    unit_of_measurement: "µg/m³"
    accuracy_decimals: 0
    device_class: pm1
  - name: "${friendly_name} PM2.5"
    id: daikinpm25
    unit_of_measurement: "µg/m³"
    accuracy_decimals: 0
    device_class: pm25
    on_value_range:                       #PM2.5 LED always on when PM2.5 is below 35
      - below: 35       
        then:
          - output.turn_on: ledpm25
      - above: 35                         #PM2.5 LED blink slowly when PM2.5 is above 35 but below 75
        below: 75 
        then:
        - while:
            condition:
              sensor.in_range:
                id: daikinpm25
                above: 35
                below: 75 
            then:
            - output.turn_on: ledpm25
            - delay: 500ms
            - output.turn_off: ledpm25
            - delay: 500ms
      - above: 75                         #PM2.5 LED blink quickly when PM2.5 is above 75
        then:
        - while:
            condition:
              sensor.in_range:
                id: daikinpm25
                above: 75
            then:
            - output.turn_on: ledpm25
            - delay: 200ms
            - output.turn_off: ledpm25
            - delay: 200ms
  - name: "${friendly_name} PM10"
    id: daikinpm10
    unit_of_measurement: "µg/m³"
    accuracy_decimals: 0
    device_class: pm10

- platform: hdc1080
  i2c_id: sgp30_bus
  temperature:
    name: "${friendly_name} Temperature"
    filters:
      - offset: -5.0 # original was -4.0
    id: temperature
  humidity:
    name: "${friendly_name} Humidity"
    filters:
      - offset: 6.0
    id: humidity
  update_interval: 30s
- platform: sgp30
  i2c_id: sgp30_bus
  eco2:
    name: "${friendly_name} eCO2"
    accuracy_decimals: 1
#    internal: true
  tvoc:
    name: "${friendly_name} TVOC"
    id: daikintvocppb
    filters:
#      - lambda: return x * 0.0023;        #ppd -> mg/m³
      - lambda: return x * 2.3;        #ppd -> µg/m³
      - exponential_moving_average:
          alpha: 0.1
          send_every: 10
          send_first_at: 1
#    unit_of_measurement: "mg/m³"
    unit_of_measurement: "µg/m³"
    device_class: volatile_organic_compounds  
#    accuracy_decimals: 2
    accuracy_decimals: 0
    on_value_range:                       #TVOC LED always on when TVOC is below 35
      - below: 500       
        then:
          - output.turn_on: ledtvoc
      - above: 500                         #TVOC LED blink slowly when TVOC is above 35 but below 75
        below: 700 
        then:
        - while:
            condition:
              sensor.in_range:
                id: daikinpm25
                above: 500
                below: 700 
            then:
            - output.turn_on: ledtvoc
            - delay: 500ms
            - output.turn_off: ledtvoc
            - delay: 500ms
      - above: 700                         #TVOC LED blink quickly when TVOC is above 75
        then:
        - while:
            condition:
              sensor.in_range:
                id: daikinpm25
                above: 700
            then:
            - output.turn_on: ledtvoc
            - delay: 200ms
            - output.turn_off: ledtvoc
            - delay: 200ms
  compensation:
    temperature_source: temperature
    humidity_source: humidity
  store_baseline: yes
  address: 0x58
  update_interval: 1s

switch:
- platform: custom
  lambda: |-
    auto cm1106Calib = new CM1106CalibrateSwitch(id(cm1106_uart));
    App.register_component(cm1106Calib);
    return {cm1106Calib};
  switches:
  - id: calibration
    internal: true

status_led:
  pin: 
    number: 25
    inverted: True

button:
  - platform: restart
    name: "${friendly_name} Restart"
  - platform: template
    name: "${friendly_name} CO2 Calibration"
    on_press:
      then:
        - switch.turn_on: calibration

binary_sensor:
  - platform: gpio
    pin: GPIO34
    id: RESET
    internal: true
    on_press:  # After pressing the RESET button on the back, the CO2 indicator light is on, and after ten minutes, set the ambient CO2 concentration as the reference 400ppm and turn off the CO2 indicator light
      then:
        - output.turn_on: ledco2
        - delay: 10min
        - switch.turn_on: calibration
        - delay: 10s
        - output.turn_on: ledco2

i2c:
- id: PM25_bus
  sda: 23
  scl: 22
  scan: true
- id: sgp30_bus
  sda: 19
  scl: 21
  scan: true  
  
output:
  - platform: gpio  #DAIKIN Air Sensor CO2 LED
    pin: GPIO4
    id: ledco2
  - platform: gpio  #DAIKIN Air Sensor TVOC LED
    pin: GPIO32
    id: ledtvoc
  - platform: gpio  #DAIKIN Air Sensor PM2.5 LED
    pin: GPIO33
    id: ledpm25    
    
text_sensor:
  - platform: wifi_info
    ip_address:
      name: "${friendly_name} IP Address"      
      icon: "mdi:ip-outline"
    ssid:
      name: "${friendly_name} Connected SSID"
      icon: "mdi:router-wireless"
    mac_address:
      name: "${friendly_name} MAC Address"
      icon: "mdi:lan"
  - platform: version
    name: "${friendly_name} FW Version"

An the code of the two libraries used:

cm1106.h

// put this file in your esphome folder
// protocol implemented as described in https://en.gassensor.com.cn/Product_files/Specifications/CM1106-C%20Single%20Beam%20NDIR%20CO2%20Sensor%20Module%20Specification.pdf

#include "esphome.h"

class CM1106 : public UARTDevice {
  public:
    CM1106(UARTComponent *parent) : UARTDevice(parent) {}

    void setCo2CalibValue(uint16_t ppm = 400) {
        uint8_t cmd[6];
        memcpy(cmd, CM1106_CMD_SET_CO2_CALIB, sizeof(cmd));
        cmd[3] = ppm >> 8;
        cmd[4] = ppm & 0xFF;
        uint8_t response[4] = {0};
        bool success = sendUartCommand(cmd, sizeof(cmd), response, sizeof(response));

        if(!success) {
            ESP_LOGW(TAG, "Reading data from CM1106 failed!");
            return;
        }

        // check if correct response received
        if(memcmp(response, CM1106_CMD_SET_CO2_CALIB_RESPONSE, sizeof(response)) != 0) {
            ESP_LOGW(TAG, "Got wrong UART response: %02X %02X %02X %02X", response[0], response[1], response[2], response[3]);
            return;
        }

        ESP_LOGD(TAG, "CM1106 Successfully calibrated sensor to %uppm", ppm);

    }

    int16_t getCo2PPM() {
        uint8_t response[8] = {0}; // response: 0x16, 0x05, 0x01, DF1, DF2, DF3, DF4, CRC. PPM: DF1*256+DF2
        bool success = sendUartCommand(CM1106_CMD_GET_CO2, sizeof(CM1106_CMD_GET_CO2), response, sizeof(response));
        
        if(!success) {
            ESP_LOGW(TAG, "Reading data from CM1106 failed!");
            return -1;
        }

        if(!(response[0] == 0x16 && response[1] == 0x05 && response[2] == 0x01)) {
            ESP_LOGW(TAG, "Got wrong UART response: %02X %02X %02X %02X...", response[0], response[1], response[2], response[3]);
            return -1;
        }

        uint8_t checksum = calcCRC(response, sizeof(response));
        if(response[7] != checksum) {
            ESP_LOGW(TAG, "Got wrong UART checksum: 0x%02X - Calculated: 0x%02X", response[7], checksum);
            return -1;
        }

        int16_t ppm = response[3] << 8 | response[4];
        ESP_LOGD(TAG, "CM1106 Received COâ‚‚=%uppm DF3=%02X DF4=%02X", ppm, response[5], response[6]);
        return ppm;
    }

  private:
    const char *TAG = "cm1106";
    uint8_t CM1106_CMD_GET_CO2[4] = {0x11, 0x01, 0x01, 0xED}; // head, len, cmd, [data], crc
    uint8_t CM1106_CMD_SET_CO2_CALIB[6] = {0x11, 0x03, 0x03, 0x00, 0x00, 0x00};
    uint8_t CM1106_CMD_SET_CO2_CALIB_RESPONSE[4] = {0x16, 0x01, 0x03, 0xE6};
    
    // Checksum: 256-(HEAD+LEN+CMD+DATA)%256
    uint8_t calcCRC(uint8_t* response, size_t len) {
        uint8_t crc = 0;
        // last byte of response is checksum, don't calculate it
        for(int i = 0; i < len - 1; i++) {
            crc -= response[i];
        }
        return crc;
    }

    bool sendUartCommand(uint8_t *command, size_t commandLen, uint8_t *response = nullptr, size_t responseLen = 0) {
        // Empty RX Buffer
        while (available()) {
            read();
        }

        // calculate CRC
        command[commandLen - 1] = calcCRC(command, commandLen);

        write_array(command, commandLen);
        flush();

        if(response == nullptr) {
            return true;
        }

        return read_array(response, responseLen);
    }
};

class CM1106Sensor : public PollingComponent, public Sensor {
  private:
    CM1106 *cm1106;

  public:
    CM1106Sensor(UARTComponent *parent, uint32_t update_interval) : PollingComponent(update_interval) { 
        cm1106 = new CM1106(parent);
    }

    float get_setup_priority() const { return setup_priority::DATA; }

    void setup() override {
    }

    void update() override {
        int16_t ppm = cm1106->getCo2PPM();
        if(ppm > -1) {
            publish_state(ppm);
        }
    }
};

class CM1106CalibrateSwitch : public Component, public Switch {
  private:
    CM1106 *cm1106;

  public:
    CM1106CalibrateSwitch(UARTComponent *parent) { 
        cm1106 = new CM1106(parent);
    }

    void write_state(bool state) override {
        if(state) {
            publish_state(state);
            cm1106->setCo2CalibValue();
            turn_off();
        }
        else {
            publish_state(state);
        }
    }
};

pm2015.h

#include "esphome.h"

uint16_t pm1;				//PM1.0
uint16_t pm25;				//PM2.5
uint16_t pm10;				//PM10
uint8_t Sensor_Situation = 0;				// Close: 1 Malfunction : 2 Under detecting : 3 Detecting completed: 0x80; other data is invalid.
uint16_t Sensor_MeasuringMode = 0;	// Single measuring mode: 2 Continuous measuring mode: 3 Dynamic measuring mode: 5 Timing measuring mode: >= 300 (means measuring time)
bool Updated = 0;			//
    
class pm2005 : public PollingComponent {//, public Sensor {
 public:
 	Sensor *vpm1 = new Sensor();
  Sensor *vpm25 = new Sensor();
  Sensor *vpm10 = new Sensor();
  
  pm2005() : PollingComponent(1000) {}

  void setup() override {
    // This will be called by App.setup()
    //Wire.begin(23, 22);
    //delay(100);
	  //Wire.beginTransmission(0x28);
	  //Wire.write(0x16);        	//Frame header
	  //Wire.write(0x01);        	//Number of byte, not including length of device address (From P1 to P7, 7 bytes in total)
	  //Wire.write(0x05);         //Data 1
	  //Wire.write(0x00);         //Data 2, high byte
	  //Wire.write(0x24);         //Data 2 , low byte
	  //Wire.write(0x00);         //Reserved
	  //Wire.write(0x16^0x01^0x03^0x24);         //Data check code
	  //Wire.endTransmission(true);
  }
  
  void update() override {
    // This will be called every "update_interval" milliseconds.
    
		byte* buf = new byte[12];		
  	Wire.requestFrom(0x28, 12);
  	Wire.readBytes(buf, 12);
    /*
  	ESP_LOGD("custom", "P1 0x16                : %02X", buf[1]);
  	ESP_LOGD("custom", "P3 Sensor situation    : %02X", buf[2]);
  	ESP_LOGD("custom", "P4 high byte PM1.0     : %02X", buf[3]);
  	ESP_LOGD("custom", "P5 low byte PM1.0      : %02X", buf[4]);
  	ESP_LOGD("custom", "P6 high byte PM2.5     : %02X", buf[5]);
  	ESP_LOGD("custom", "P7 low byte PM2.5      : %02X", buf[6]);
  	ESP_LOGD("custom", "P8 high byte PM10      : %02X", buf[7]);  	
  	ESP_LOGD("custom", "P9 low byte PM10       : %02X", buf[8]);
  	ESP_LOGD("custom", "P10 high byte M mode   : %02X", buf[9]);
  	ESP_LOGD("custom", "P11 low byte M mode    : %02X", buf[10]);
    */
  	if(Sensor_Situation != buf[2])
  	{
	  	Sensor_Situation = buf[2];
	  	if(Sensor_Situation==1)	ESP_LOGD("custom", "Sensor situation: Close.");
	  	else if(Sensor_Situation==2)	ESP_LOGD("custom", "Sensor situation: Malfunction.");
	  	else if(Sensor_Situation==3)	ESP_LOGD("custom", "Sensor situation: Under detecting.");
	  	else if(Sensor_Situation==0x80)
	  	{
	  		Updated = 1;	
	  		ESP_LOGD("custom", "Sensor situation: Detecting completed.");
	  	}	
	  }
  	pm1 = buf[3] * 0x100 + buf[4];
  	pm25 = buf[5] * 0x100 + buf[6];
  	pm10 = buf[7] * 0x100 + buf[8];
  	Sensor_MeasuringMode = buf[9] * 0x100 + buf[10];
  	
		if(Updated)
	  {
	 		ESP_LOGD("custom", "PM1.0: %d", pm1);  		
	 		ESP_LOGD("custom", "PM2.5: %d", pm25);  		
	   	ESP_LOGD("custom", "PM10 : %d", pm10);  		
	   	vpm1 -> publish_state(pm1);		//PM1.0
	   	vpm25 -> publish_state(pm25);	//PM2.5		
	   	vpm10 -> publish_state(pm10);	//PM10
	   	
	  	if(Sensor_MeasuringMode==2)	ESP_LOGD("custom", "The measuring mode of sensor: Single measuring mode.");
	  	else if(Sensor_MeasuringMode==3)	ESP_LOGD("custom", "The measuring mode of sensor: Continuous measuring mode.");
	  	else if(Sensor_MeasuringMode==5)	ESP_LOGD("custom", "The measuring mode of sensor: Dynamic measuring mode.");
	  		
	   	Updated = 0;
	  }	
  }
};

I’m currently using ESPHome 2023.6.3 (but noticed the issue with prior versions).
Any help or comment is more than welcome.

Spiro · October 15, 2023, 8:08pm

Start removing sections and see where the problem lies and reupload the code. Process of elimination. I would suggest start with webserver or captive portal. I mean to show which section is affecting memory so you can then work on the problem section rather than just leave it out of final product.

Joao-Sousa-71 · October 15, 2023, 8:16pm

I did that already and the result was having more free memory after boot that increased some minutes the reboot cycle. Also tried to remove loggers, fallback hotsopt, use a fixed IP address.

Spiro · October 15, 2023, 8:21pm

Good thinking. What about starting from bare config and add in sections until memory leak occurs. Having the memory sensor should at least reduce the time to finding fault as don’t have to wait hours if you see memory usage going up and up.

Joao-Sousa-71 · October 15, 2023, 8:33pm

Yes, removing sensors one by one might show which one is causing the memory issue, I don’t know if there is any debug option that can be added to the code to show where the memory is being used. I’m far from being an expert in ESPHome…

Spiro · October 15, 2023, 8:36pm

I’d seen some where people were suspecting the API encryption was causing memory leaks on ESPs.

Joao-Sousa-71 · October 15, 2023, 8:45pm

It is possible to disable?

Spiro · October 15, 2023, 9:02pm

Just to test you could try comment out the API encryption and key section to see if it works and is the problem. Don’t know if advisable to leave it out in long term.

Joao-Sousa-71 · October 15, 2023, 9:05pm

I can try that
I need to upgrade my ESPHome docker installation to the latest version to fix this, and will give it a try.

jawaligt · February 9, 2024, 4:18pm

Hi, did you ever manage to fix this? I’m running into the same issue and I haven’t been able to find a fix.

I’m talking to the guy who wrote that code, but he doesn’t seem to know either. He just sent me his own .yaml which is functionally identical.

Joao-Sousa-71 · February 9, 2024, 11:42pm

I didn’t not! Removing the encryption doesn’t fix it either. I guesse the only way is to try to delete sensors and check if the problem continues or stop.

clydebarrow · February 10, 2024, 10:48am

That’s the problem. Should simply be:

byte buf[12];

The new operator allocates from the heap each time the function is called and I see nowhere it’s released.

Joao-Sousa-71 · February 11, 2024, 1:40pm

I will try it! Thank you!

Joao-Sousa-71 · February 11, 2024, 2:08pm

@clydebarrow I’ve tagged your answer as the solution since definitely solve the problem! Thank you again.
@jawaligt just make the change on the pm2105.h library, compile again and profit.

yanzeyuan · February 13, 2024, 2:34am

@clydebarrow You’re correct, the problem is the buf is not released.

Just add:

free(buf);

at the end of update().

clydebarrow · February 13, 2024, 8:09am

No, just use what I suggested. There is no need to allocate from the heap (which is much more time-consuming) for something used only within the lifetime of a single function - that’s exactly what auto variables are intended for.

Joao-Sousa-71 · February 13, 2024, 1:55pm

I don’t want to abuse from your kindness but also have this warning that is not really serious and can be also ignored from the log:

|Time|level|Tag|Message|
| --- | --- | --- | --- |
|13:42:05|[W]|[component:214]|Component hdc1080.sensor took a long time for an operation (0.06 s).|
|13:42:05|[W]|[component:214]|Component hdc1080.sensor took a long time for an operation (0.06 s).|
|13:42:05|[W]|[component:215]|Components should block for at most 20-30ms.|
|13:42:35|[W]|[component:214]|Component hdc1080.sensor took a long time for an operation (0.06 s).|
|13:42:35|[W]|[component:214]|Component hdc1080.sensor took a long time for an operation (0.06 s).|
|13:42:35|[W]|[component:215]|Components should block for at most 20-30ms.|

In the meantime I found this thread but didn’t figure out any conclusion.

github.com/esphome/issues

Component xxxxxx took a long time for an operation

opened 08:57PM - 19 Jul 23 UTC

jesserockz

## Do not report new issues related to this message, instead please comment on t…his issue with the logs that you see and the config for that component. ### The problem ``` [00:00:00][W][component:204]: Component xxxxxx took a long time for an operation (x.xx s). [00:00:00][W][component:205]: Components should block for at most 20-30ms. ``` These 2 log lines may show up in the most recent version of ESPHome due to the log level being changed from `verbose` to `warning`. I made this change because changing the device log level to verbose just to see if these lines show up significantly slowed down the device due to all the extra logging it had to do. ### Which version of ESPHome has the issue? 2023.7.0 ### Can I hide this message? Yes you can hide the message, but it wont change the fact that the component is taking a long time. ```yaml logger: logs: component: ERROR ``` ### Is it expected? For some components this is actually expected like displays which have a lot of drawing and pixel/data moving. For the majority of sensors, it means that they are doing too much processing or waiting for responses instead of letting ESPHome get on with business until the data is ready.

I can open a new topic for this if it is adequate. Thank you.

jawaligt · February 13, 2024, 4:58pm

I can’t really comment on the validity of this, but does it make any practical difference for the operation of the sensor? Or is it more of a ‘correct code’ thing.

jawaligt · February 13, 2024, 5:02pm

I’m also curious what you guys think about the PM1/2.5/10 sensor. Does it need any calibration? And if so, outside or inside?

It’s weird that the eCO2/TVOC sensor needs 12 hours to establish a baseline.

clydebarrow · February 14, 2024, 12:36am

Either way can be “correct” - only if there is no memory leak! - but only one is good code.