Just tested this out for you and can confirm you're right. Something in the HA command from the dashboard is causing the light.turn_on command to also be sent when simply setting an effect.
Not 100% sure if it's a bug or intentional, due to some lights needing to be powered on to perform the command.
Anyway, you can get around this by calling the mqtt.publish action from within HA. Tested the below on my Z2M LED strip & it turned off correctly after applying the effect:
Yep, can confirm that works. My Effect entity was disabled, which is why I didn't know about it.
You can call the same effect from the Dev Tools multiple times in a row & the light will turn back off.
You can also call it from the device page once & the light will turn back off. Annoyingly, the select on the device page stays on the last selection. There's no none option, so you can't call the same effect twice in a row, even if you refresh the page & clear cache.
Looks like calls via the dev tools are the only way to go for now.
Script is repeatable as many times as you want in Dev tools or scriptws. It's only the Frontend which has issues.
On the device page, you have an effect select entity. If you pick an effect, you can only call it once per effect - you cannot call the same effect twice in a row. Light will go back off when the effect is done, which is what David wanted.
If you add a card to the dashboard, you can call the effect multiple times in a row, but the light will stay on.
So what I'm saying is that basically service calls are working fine, but so far both Frontend options result in unwanted behaviour.