The inverting seemed to fix it as well as switching the pins. Now the only thing I think I am missing is being able to mute it.
I assume for being able to mute the doorbell I will want to use something like this
This condition passes if the given switch is on/off.
# in a trigger: on_...: if: condition: switch.is_on: my_switch # same goes for is_off then: - script.execute: my_script