Remove entities from Scenes configuration

Currently, if I configure a scene using the UI, all the entities are added. For example, in my case, I have a Shelly 2.5 that has two relays. If I add this to the scene, both entities show up. But basically, I only want to set one of the two entities to a specific state. The other entity should not be effected when the scene is activated.

I tried removing the entity from my yaml file, but the entity still shows up in the UI. At leas in 0.115.2 this seems to happen.