Your template contains a basic error that has nothing to do with default values. Please move it to a new topic (this topic isn’t meant for solving everyone’s fundamental templating mistakes). Done.
timestamp_custom expects to receive a numeric timestamp value (seconds since the Unix Epoch).
The first one works because it supplies timestamp_custom with a numeric timestamp value.
The second one fails because it supplies timestamp_custom with a time string value.
NOTE
If timestamp_custom is given a value it cannot convert it simply reports the supplied value by default. For example, the result of this is cat
{{ 'cat' | timestamp_custom('%H:%M') }}
This default behavior will change soon.
Starting in version 2021.10.0 it will warn you that you overlooked to provide timestamp_custom with a default value. In 2021.12.0 you will be obligated to supply a default value.
To me, this part suggests (states('sensor.time_utc') is converting time string into numeric timestramp. What is the correct way to extract time string from a sensor?
states('sensor.time_utc') returns the state value of sensor.time_utc which is, like for all entities, a string value. The states() function doesn’t “convert” an entity’s state value.
If you have not read this section in the documentation, I highly recommend it.