Prevent user to remove self refresh token

Referring to https://github.com/home-assistant/frontend/issues/6151 front-end try to prevent user to remove self refresh token which is usual logic and also used in many authentication provider.
But this is not enough, Just the CSS pointer-event prevent is not good to prevent user from an action.
I think this logic should implement in core too.
Or maybe in authenticate integration.