@nickrout id you build this yourself with the kit or get the reassembled version?
I ask because it looks like the preassembled version is pre-flashed, so you only need to connect it to Wi-Fi and add it using the ESPhome integration. If not, then you’ll need to flash it following their instructions (I didn’t follow all the way through but it looks like they’re using the ESPhome web flasher) to flash through your browser on a regular PC.
More abstractly, here’s a primer to (hopefully) help figure out how the pieces fit together.
EDIT: Reread and see where @nickrout was going with this. I would follow his suggestion of temporarily connecting it to the same network as HA to see if it clears the problem up. The link above may still help.