Disabling `captive portal`

While messing around with my wifi I noticed that my esphome devices fall into a “captive portal” mode that allows an unauthenticated user to set the device’s wifi config, and potentially even worse, flash it with a new firmware. I understand the convenience aspect, but for sensitive devices this could be pretty dangerous.

How does one disable it? I can’t see any reference to it in my yaml. Alternatively is it possible to password protect it?

EDIT: it turns out it was being imported via another file. Is there a way to override this?

Of course this exact case is documented:

packages:
  - !include common.yaml  # see above

captive_portal: !remove