The only solution I found for this was to modify the home-assistant_v2.db with SQLite DB Browser, which you can download and install for free. I am running home assistant in a docker instance on a Ubuntu server. The database file is found in the home assistant config folder.
*** Messing with the database is probably the best way to completely screw up your entire install if something goes wrong - so make a backup first. ***
Stop home assistant and make a backup of the database first. Then while home assistant is stopped, you can access the database with the SQLite DB Browser. For me, my orphan entities were caused by creating MQTT sensors in my configuration.yaml. I found these entities referenced in the “states” table of the database. I simply deleted every record I found which referenced the orphaned entities. Once I restarted home assistant, I checked the entities and found the orphan entities were all gone.
I had previously tried restarting home assistant to see if they would disappear, that did not work. I used grep to find any instances of the orphaned entities in the .storage folder. I removed every instance I found in that folder to only see them created by home assistant again. The only file left with references to the entities I wanted to remove was the database file. I found these things so annoying I was willing to risk a complete re-install and re-build if something went wrong in the database modifications. But hey, if it did go that wrong, a clean install would also fix the issue.