I had moved from HASS Discovery to Tasmota Integration a while ago for my active Tasmota devices and forgot the recipe. I recently put into service some previously shelved devices and they show up in both MQTT and Tamota integrations as shown in the screenshots.
Have a look under the “homeassistant” topic. You should see “sensor”, “binary_sensor”, etc corresponding to the entities in HA. These are retained topics, so will cause the entities to be created in HA. Delete the relevant retained topics before deleting them in HA.
I’ve never seen that dialog before. However I have had entities that should be gone continue to “linger” for a while after HA restarts. And then 10 minutes later they disappear all by themselves, so maybe you’ll get lucky.
In playing around with the GUI, I clicked on the 3 vertical dots in the dialog and it popped up a menu asking me if I would like to delete the device from MQTT! Hopefully that will do it.