This bugged me as well when I tried this earlier this year, before I switch to SwitchBot bot/curtain/meter/contact/motion MQTT ESP32 bridge - Local control where I just have a toggle button.
I am considering switching back now, with esphome bringing bluetooth proxy functionality, but if that messes with automations it wouldnt be a big help.