Welp just figured it out,
GPIO pins ID_SC and ID_SD (GPIO0 and GPIO1) are reserved for use solely for board detection / identification. The only allowed connections to the ID_ pins are an ID EEPROM plus 3.9K pull up resistors. Do not connect anything else to these pins!
So connecting these pins to ground is a bad idea 
HA setting them to “pull up” should address the floating problem. It won’t help if you connect a switch to ground and that GPIO enables in OUTPUT mode at boot up though. This is what the resistors are to protect against.
That makes sense, but by default HA sets them all to Pull UP so not sure why GPIO14 is floating…
Now the next question is how do I make other GPIO pins work. Even though they show up in the hardware view if I declare in HA config, they don’t actually work. Anyone else has this issue?