Global array size limitations

GrahamS · January 10, 2025, 10:02pm

I want to use global array of bytes.

globals:
  - id: TIDE_TABLE
    <<: !include .tide_table_jan.yaml

inside the include file is:

    type: std::array<uint8_t, 396>
    restore_value: no
    initial_value: '{
        0x00, 0x00, 0x00,
        0x00, 0x00, 0x00,
        0x00, 0x00, 0x00,
        0x00, 0x00, 0x00,
        0x01, 0xd8, 0x80,
        ...

If I increase the array size to 4404 (tide table for entire year) the WIFI connection dies.
I thought this was just a ESP8266 flash/RAM limitation so I ported to a
esp32-s3-devkitc-1. Same problem using the ESP32.
If I create 12 global variable arrays, each 396 bytes, no problem on both boards.
There must be an internal limit for array sizes set by ESPHome.
Anyone know of a limit? I could combine 3 byte groups into an integer to get the array size down to 1468.

GrahamS · January 11, 2025, 5:31am

I split my large array into 2196 and 2208 byte arrays and the firmwave does not crash the WiFi. In an experiment 4095 byte array failed. So limit is somewhere between 2209 and less than 4095. You would think a ESP32-S3 N16R8 could handle larger arrays.

WallyR · January 11, 2025, 5:52am

You might need extra lines to get the full benefit of the extra hardware on your board.

GrahamS · January 11, 2025, 6:07am

Using this to get the “max”:

esphome:
  name: desal
  friendly_name: Desalination Controller
  platformio_options:
    board_build.arduino.memory_type: qio_opi
    board_upload.maximum_ram_size: 524288

esp32:
  board: esp32-s3-devkitc-1
  variant: esp32s3
  flash_size: 16MB
  partitions: "/config/esphome/default_16MB.csv"
  framework:
    type: arduino

psram:
  mode: octal
  speed: 80MHz

Mahko_Mahko · January 11, 2025, 6:23am

Does debug give you any insights?

GrahamS · January 11, 2025, 6:39am

Cannot use Debug because the Wifi connection is lost immediately after the download reset. The board green LED blinks. I think it is the USB receive data monitor.

WallyR · January 11, 2025, 6:41am

What does the serial output tell you?

GrahamS · January 11, 2025, 6:55am

Something more interesting. Some repetitive reboot?

[22:50:51]ESP-ROM:esp32s3-20210327
[22:50:51]ESP-ROM:esp32s3-20210327
[22:50:51]Build:Mar 27 2021
[22:50:51]rst:0xc (RTC_SW_CPU_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
[22:50:51]Saved PC:0x4209d712
[22:50:51]SPIWP:0xee
[22:50:51]mode:DIO, clock div:1
[22:50:51]load:0x3fce3808,len:0x43c
[22:50:51]load:0x403c9700,len:0xbec
[22:50:51]load:0x403cc700,len:0x2a3c
[22:50:51]entry 0x403c98d8
[22:50:51]ESP-ROM:esp32s3-20210327
[22:50:51]Build:Mar 27 2021
[22:50:51]rst:0xc (RTC_SW_CPU_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
[22:50:51]Saved PC:0x4209d712
[22:50:51]SPIWP:0xee
[22:50:51]mode:DIO, clock div:1
[22:50:51]load:0x3fce3808,len:0x43c
[22:50:51]load:0x403c9700,len:0xbec
[22:50:51]load:0x403cc700,len:0x2a3c
[22:50:51]entry 0x403c98d8
[22:50:52]ESP-ROM:esp32s3-20210327
[22:50:52]Build:Mar 27 2021
[22:50:52]rst:0xc (RTC_SW_CPU_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
[22:50:52]Saved PC:0x4209d712
[22:50:52]SPIWP:0xee
[22:50:52]mode:DIO, clock div:1
[22:50:52]load:0x3fce3808,len:0x43c
[22:50:52]load:0x403c9700,len:0xbec
[22:50:52]load:0x403cc700,len:0x2a3c
[22:50:52]entry 0x403c98d8
[22:50:52]ESP-ROM:esp32s3-20210327
[22:50:52]Build:Mar 27 2021
[22:50:52]rst:0xc (RTC_SW_CPU_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
[22:50:52]Saved PC:0x4209d712
[22:50:52]SPIWP:0xee
[22:50:52]mode:DIO, clock div:1
[22:50:52]load:0x3fce3808,len:0x43c
[22:50:52]load:0x403c9700,len:0xbec
[22:50:52]load:0x403cc700,len:0x2a3c
[22:50:52]entry 0x403c98d8
[22:50:52]ESP-ROM:esp32s3-20210327
[22:50:52]Build:Mar 27 2021
[22:50:52]rst:0xc (RTC_SW_CPU_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
[22:50:52]Saved PC:0x4209d712
[22:50:52]SPIWP:0xee
[22:50:52]mode:DIO, clock div:1
[22:50:52]load:0x3fce3808,len:0x43c
[22:50:52]load:0x403c9700,len:0xbec
[22:50:52]load:0x403cc700,len:0x2a3c

GrahamS · January 11, 2025, 7:16am

I little more tracing using the right-side USB port. I think this error is due to a stack overflow. Same problem with the ESP8266 so probably an ESPHome core issue.


[23:12:19]Rebooting...
[23:12:19]ESP-ROM:esp32s3-20210327
[23:12:19]Build:Mar 27 2021
[23:12:19]rst:0xc (RTC_SW_CPU_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
[23:12:19]Saved PC:0x40377680
[23:12:19]SPIWP:0xee
[23:12:19]mode:DIO, clock div:1
[23:12:19]load:0x3fce3808,len:0x43c
[23:12:19]load:0x403c9700,len:0xbec
[23:12:19]load:0x403cc700,len:0x2a3c
[23:12:19]entry 0x403c98d8
[23:12:19]Guru Meditation Error: Core  0 panic'ed (Unhandled debug exception). 
[23:12:19]Debug exception reason: Stack canary watchpoint triggered (IDLE) 
[23:12:19]Core  0 register dump:
[23:12:19]PC      : 0x42028545  PS      : 0x00060036  A0      : 0x82028577  A1      : 0x3fcf4970  
[23:12:19]A2      : 0x00000047  A3      : 0x001f8000  A4      : 0x000007d0  A5      : 0x601f8000  
[23:12:19]A6      : 0x3fc9a2d4  A7      : 0x00000000  A8      : 0x00000000  A9      : 0x60000000  
[23:12:19]A10     : 0x3fc953d0  A11     : 0x00000000  A12     : 0x00000008  A13     : 0x3fcf49a8  
[23:12:19]A14     : 0x42027f0c  A15     : 0x00000000  SAR     : 0x00000020  EXCCAUSE: 0x00000001  
[23:12:19]EXCVADDR: 0xb33fffff  LBEG    : 0x400570e8  LEND    : 0x400570f3  LCOUNT  : 0xffffffff  
[23:12:19]
[23:12:19]
[23:12:19]Backtrace:0x42028542:0x3fcf49700x42028574:0x3fcf49b0 0x42028748:0x3fcf49d0 0x42028b36:0x3fcf4a50 0x40377b19:0x3fcf4aa0 0x40378a24:0x3fcf4ac0 0x42027ad1:0x3fcf4b80 0x42027ed9:0x3fcf4ba0 0x42027f0f:0x3fcf4bd0 0x420283b6:0x3fcf4bf0 0x4037e7f4:0x3fcf4c10 
[23:12:19]
[23:12:19]
[23:12:19]
[23:12:19]
[23:12:19]ELF file SHA256: 0000000000000000
[23:12:19]

WallyR · January 11, 2025, 8:01am

I had the same error, but I could just disable a few things and get around it.
That solution might not be possible for you, but now you have something to show the next ones that try to help.

Mahko_Mahko · January 11, 2025, 8:45am

Same result with a bare-bones config?

S3 support is pretty new too. I don’t know if it is worth trying with a common esp32.

I’m just taking some wild stabs here.

GrahamS · January 11, 2025, 8:37pm

Here is my solution. It appears that array initialization takes too long and this disrupts the boot process. This is a problem for BOTH ESP8266 and ESP32 boards. It is not a problem with the allocation size of the memory.

First define the array with a maximum size:

globals:
  - id: TIDE_TABLE
    type: std::array<uint8_t, 4416>

Then initialize at boot with low priority:

  on_boot:
    - priority: 200
      then:
        - lambda: |-
            id(TIDE_TABLE)[0] = 0x00;
            id(TIDE_TABLE)[1] = 0x01;
            id(TIDE_TABLE)[2] = 0x20;
            id(TIDE_TABLE)[3] = 0x00;
            up to 
            id(TIDE_TABLE)[4415] = 0x50;

Used:

  <<: !include .boot.yaml

I used a Python program to generate to verbose boot file.
It works but does take longer to compile (2-3 minutes).
Enjoy