For switches with an assumed state two buttons are shown (turn off, turn on) instead of a switch. By setting assumed_state to false you will get the default switch icon.
default switch icon isnt supported for MQTT switch?
A lot of my MQTT switches do that after a HA restart. After a few minutes (when they received their status), they show the normal icon. You can force the normal icon on HA restart by having the status as a retained message, or as said above, assumed_state. I don’t care for that small amount of time that I have other icons.