To get something custom, you’ll need to use a template sensor:
I think you’ll eventually find yourself using template sensors for lots of random things, so definitely worth the time to understand them outside of this specific case. That being said, I really prefer to use existing device classes whenever possible, because they just integrate with everything else in Home Assistant so much better. But I completely get it; there are times where none of the existing classes work for your use case.
Just remember that binary sensors are only on or off (the template must return true or false). They can also be unknown, but that’s a different matter. You don’t have to define a device class, but if one suits, it basically translates the state (e.g. to open or closed) and give you matching icons. That said, you can set your own icon without a device class.
The template sensor approach I was referring to would be done within the Home Assistant configuration YAML file, not the ESPHome YAML (although I guess you could also do it all in ESPHome, but that makes less sense in my mind). Basically, you’d setup a generic binary sensor in ESPHome and then create ANOTHER template sensor within HA that would take the value of your ESPHome sensor and return the text state you’re after. The problem with this approach, is the second sensor is no longer binary and Home Assistant won’t really be able to do anything with it; it will just be a meaningless text value to it.
I noticed it seems that you are making a door contact sensor potentially? Is there a reason the door device class won’t work?
If you can only see the alarm state and can’t actually interact/change it, this probably doesn’t make much sense, but using your binary sensor and an automation, you could also incorporate this to better show your alarm system.
If you’re just displaying the value (set or unset) then the template method would be simplest. Although “on” and “off” should be just as clear. I’ve also seen “armed” and “unarmed” used in alarm systems.
If you go the template route, be sure to exclude the original entities from the Recorder so you don’t end up putting duplicate values in the database.
Think of it as an enumeration: a list of values the item can have. In your case there are two values you want it to have. Technically it isn’t a binary sensor. Come to think of it: it is editable so it is not what you want. The template sensor is your best alternative to a binary sensor. There’s no limitations on what you can do with those, as long as you remember the actual value is the string you want it to show. The value template above is what you need for defining the state of the template sensor. On the top of my head, ad this to the templates of home assistant (untested):
I would do this in Home Assistant and not in the esp. You can also create a text sensor there, but I would not know how to do that without relying on timed updates, which would cause a delay in the sensor value.