Well not really, esphome uploads over USB/serial too.
I assume by ‘download board’ you refer to the addon board supplied with these devices. Yes you do need this to flash it. On the download board there are two buttons, I think you need to hold one down while flashing.
EDIT: Also a bit of searching this forum finds a number of threads eg Flashing a LilyGo T-ETH-POE, LILYGO Internet POE board integration