Adding my First Zigbee Devices

Yes, you can but again, they either need to be wired to provide permanent power to the bulb (you use automations to capture the button press and toggle the lights) or they need to have a detached mode built in.

Your cheapest way would be to use something like the Sonoff ZBMINIR2 behind your existing switches. There’s a good thread here: https://community.home-assistant.io/t/zha-advanced-features-of-sonoff-zbminir2/787340.