One of the reasons for failure is a bug in esp-idf: Bluedroid HCI: ACL fragment reassembly failure during rapid BLE indication bursts (IDFGH-17460) · Issue #18414 · espressif/esp-idf · GitHub
It has been fixed on 5.4.x and 6.0.1, but sadly not on 5.5.x (and also not yet on 6.1-dev).
But I was able to use esp-idf 6.0.1 with this configuration (needs latest ESPHome):
esp32:
framework:
type: esp-idf
version: 6.0.1
That bug is fixed, but there seem to be more problems with the commissioning. I'll keep this post updated.