The way you can clean them up is by directly editing the entity registry in the .storage directory. Back it up first as if this file gets corrupted by editing it improperly HA will not start.
The API endpoint should allow you to overwrite the state of any entity. So create a template sensor give it a default value as the state (like 0j and a unique id.
Ok, to answer a part of my own question:
For cleaning an entity up from the database, the service purge_entities is the solution. I called it from the UI and the states were gone from the DB.
The entity remained visible in the UI, e.g. in /developer-tools/state, but only until I restarted Home assistant. Afterwards it was completely gone.