Thanks everyone! Got it working.
I stuck to the GPIO numbering. D numbering is not supported on this one with ESPHome
I followed this doc: FireBeetle_Board_ESP32_E_SKU_DFR0654-DFRobot
and this doc ESP32 Pinout Reference: Which GPIO pins should you use? | Random Nerd Tutorials
I added one by one. A pain but works.
I ended up with 8 push buttons and maybe 10 is possible
If I need more buttons I am going to add an I2C I/O expander
Conclusion is the FireBeetle 2 ESP32-E - DFR0654 works fine with esp32dev platform, but for future projects I am going to be using the Wemos Lolin D32 for battery operated devices.
The next thing is to get it to go in deep sleep and then wake up by button press