See how I do it here. I think I’m doing it how you envision. I listen for the ZHA hold and release events and use these to start and stop a loop that uses the brightness step service to dim or brighten the light. The service is called four times a second and steps 5% each call, so full sweep in 5 seconds.
It works perfectly with an Ikea two button dimmer remote and a conbee ii coordinator.