You really need to integrate it in both directions for full control using two different approaches. To integrate Alexa into Home Assistant, the best way is the Alexa Media Player custom component:
This will bring your Alexa devices into Home Assistant, so you can control their volume, do not disturb settings, routines, and make announcements using text to speech through them.
To integrate Home Assistant devices into Alexa in that direction - ie you tell Alexa to turn a Home Assistant connected switch on, the best way would be Home Assistant Cloud aka Nabu Casa and the Home Assistant Alexa Skill
No, they will be difficult to control that way and cloud dependent. You ideally want to get a zigbee USB stick, and then bring your zigbee devices into Home Assistant using ZHA.
Or if you need advanced options and want mqtt, zigbee2mqtt.
If you have Home Assistant cloud, your zigbee devices integrated to Home Assistant would then be connected to Amazon. There is a small monthly fee for the cloud, and initial one time fee to buy the zigbee stick (recommended ones here Supported Adapters | Zigbee2MQTT ) - but to get local control of the zigbee devices it is worth it.
If you use the Amazon device as your zigbee hub, you could control the zigbee devices by setting up Amazon routines and using the Amazon Media Player skill to trigger those routines to turn those zigbee devices on and off, but you will not get status of those devices in Home Assistant. For example, if you turn it on through Home Assistant, but you turn it off through Alexa, Home Assistant would have no idea if the switch/device was on or off. Therefore, I wouldn’t recommend trying to integrate zigbee devices directly connected to Alexa to Home Assistant.