Alright, let’s wrap this up…
So the Blakadder way didn’t work for me. Not sure why. It may work for you, but since you are reading this, you are probably in the same boat.
I tried flashing again multiple times today but with no success. After digging into that flash read error (see above), I had an epiphany (turned out I needed multiple today btw). So I opened up web.esphome again and then flash to the “Prepare for first use” image. That worked and was a starting point for me. I then decided to flash a minimal image to the device, just to get it into my ESPHome dashboard and then build it up. That also worked. After several rebuilds, and adding options every time, the device was up and running. I thought I was done and happy. But there wasn’t a whole lot of activity on the device and it wasn’t discovering any Bluetooth devices.
So started Googling around and found the reason why it uses the ESP-EDIF framework instead of the Arduino I was using. ESP-EDIF is much faster (that’s what the website said).
Then I decided to flash an image that was built on the ESP-EDIF framework. But, as said before, that didn’t work either. Turned out I was running into this issue. The actual fix can be found here. But, I run Hassio and had no idea where to find that platformio folder that I needed to delete.
Luckily, I already have Advanced SSH & Web Terminal installed, and if you run the addon in privileged mode, you can use the docker command…
So SSH into HA, and issue
docker ps
to find the name of your ESPHome Docker env.
Then use
docker exec -ti <containername> bash
to get into that container.
The folder you are looking for is in the /data/cache/
You can delete the platformio folder.
Just to be sure, I restarted the ESPHome addon.
Now you can try and build the firmware using the ESP-EDIF framework. This may take a long time, but it will succeed. 
DONE!
Well almost… because I now needed to make sure I could still update my firmware of the other ESP devices I use. But they run the Arduino framework. So I tested that with an easy one; CO-meter. It needs to download all dependencies, and especially installing the Arduino framework will take a long time. But patience is your friend and it successfully finished. So all set, done, and finished!
It just took me a lot of time. It always does 
To be complete, below is the yaml file I use. I remove the I2C part, and I will not use that anyway. There is a bug in Blakadders yaml with the LED-config. I just removed that but didn’t fix it yet.
substitutions:
device_name: bluetooth-proxy
device_description: Bluetooth Proxy
esphome:
name: "bluetooth-proxy"
friendly_name: "bluetooth-proxy"
on_boot:
then:
- output.turn_on: power_led
esp32:
board: esp32dev
framework:
type: esp-idf
# Enable logging
logger:
# Enable Home Assistant API
api:
encryption:
key: "KJ7IEZA0CrWW6djX4QFzg+xWI+455px+yH16nvVuNyk="
ota:
password: "ee1de38d9f7b0af9653a85b4f5ea2baa"
ethernet:
type: IP101
mdc_pin: GPIO23
mdio_pin: GPIO18
clk_mode: GPIO0_IN
phy_addr: 1
power_pin: GPIO5
esp32_ble_tracker:
scan_parameters:
interval: 1100ms
window: 1100ms
active: True
bluetooth_proxy:
active: true
binary_sensor:
- platform: gpio
pin: 33
name: "Reset Button"
status_led:
pin:
number: GPIO32
inverted: true
output:
- platform: gpio
pin: GPIO14
inverted: true
id: power_led
Glad you’ve figured it out.