Using a binary sensor to sleep and wake an ESP32 rather than time

Are you using a cheap esp clone by chance?

How are you powering it? Try the best supply and cables you have. Are you connections soldered? How long are your wires?

You could try adding a on_boot delay (and wire begin if using I2C).

You could also try this.