Very strange, I guess I must have.
Anyway, with that ticked back on, I was now able to rename my entities as expected (though I had to rename the device back to the default value and rename it again for it to actually rename them as expected)
For anyone else that stumbles on this, would appreciate if you voted for the related feature request I submitted: Prompt for Name When Adding a Device