I have not used alert before. I looked up the docs and it looks like I need to add it to configuration.yaml. Is there a UI way of doing this?
No.
If you do try to install it via the UI, the following warning message appears:
This integration cannot be added from the UI
You can add this integration by including it in your ‘configuration.yaml’. See the documentation for more information.
The Alert integrations’s documentation explains you must add it to configuration.yaml (see the example I posted above).
FWIW, the same is true for the Trigger-based Template Sensor I suggested (it cannot currently be created via the UI and must be defined as YAML).