Not all “strftime” options are available directly in esphome, one option would be to create a template sensor in HA and display that on the ephome display.
For anyone looking for this in the future, making a template sensor works. This is my template for this. You can then include the sensor in esphome as a text sensor.