I’ve always had this problem in my Raspberry Pi 3 B+, but i dont want to change it only for this problem. Everything works great except this.
I found a clue about it: In ESPHome you can change platformio options: https://esphome.io/components/esphome.html#platformio-options
Maybe, if you tell platformio that you are using a raspberry pi 3b, compiler adjust to rpi’s hardware not to causing freeze. Parameters we have to put are described here: https://docs.platformio.org/en/latest/boards/linux_arm/raspberrypi_3b.html