Perhaps you need a more powerful fan if it’s not reducing the humidity quickly enough? Also, have a look at this blueprint https://community.home-assistant.io/t/bathroom-humidity-exhaust-fan/509992
It does exactly what you want - I’ve been using it for some time now and it works great.