Trying to have a condition when light switch has been trigger manually

Hi have an automation for trigger light automatic on or off, but I try to create an automation that should disable this if the light switch has been touched physically. So I tried to add a condition checking for

{{ trigger.to_state.context.user_id == None }}

what I tought will only be != None if it has been switched by home assistant. So another automation, an input_boolean or something else. And if switch physically then it would be None.
But I get None also when I switch the light from inside HA or if another input_boolean in been triggering the light to switch on/off.

Is there a clean way to check if an light switch has been switch physically or by something from inside HA?