This “issue” has been reported dozens of times in this forum and on also on GitHub for many years, over multiple drivers. Some attempts to address this in HA haven’t been accepted, and they have not been good solutions. I’d suggest either live with the UI issue, or set a specific brightness level when you turn it on to avoid it. Polling the device to avoid a temporary UI issue seems like overkill to me. If you really wanted to do that, you could try using the value updated trigger to react to the targetValue property change (to 255), instead of the entity state.
When you turn on the switch w/o a set level, the driver does not know what the expected level will be, so it doesn’t update until 5 seconds later (+ N duration). This delay is not configurable unless you run the driver yourself, and it applies globally, not per-device. The UI reverts to the previous state if it doesn’t get an update before 2 seconds. When you turn on a device with a specific level, the driver sends an immediate optimistic response, which updates the state in HA.