Maybe you could look at using something like this? (These would let you add up to 16 buttons and from a quick search on Ali can be bought for a few -insert-currency-of-choice- .)
It was kind of stupid to connect the buttom betwenn 3 Volt and a Pin. If i instead connect the buttom between PIN and ground i can use pullup instead of pulldown.
My board supports pulldwon in just one pin but pullup in many others.
As well You can use PCF8574 (8-port) or even PCF8575 (16-port) IO multiplexer/extension board.
By utilizing 2 pins for I2C bus You obtain 8/16 Input/Output ports - best use to connect more buttons/switches/relay.
Its not stupid, you just had them wired for pull down when the gpios dont have it. It sounds like you understand the differences now so thats not stupid. Pull up, pull down is extremely common in just about every circuit and not just esp based circuits. Its an important concept to understand because youll see it a lot more.