The way Z-Wave works you can’t wander around the house with any Z-Wave device, it needs to stay relatively near where you include it, or you have to run a Heal after you move it to another room.
Mine controls a WiFi connected light, and is at the far end of my house from the controller, so at best there’s normally 3 hops to the controller, sometimes 4. Despite this control is fairly responsive, comfortably less than a second from pushing the button on the controller and an action happening - dimming takes a fraction longer, just because the remote doesn’t send the button held Scene immediately.
If you’re seeing longer, I’d look at both the health of your Z-Wave mesh (see this thread for how to generate a map), and the load on your Home Assistant server (which you can do with the top command, or tools like Netdata).