ZHA Sonoff Button (press, double press, hold actions)

I am having issues where the automation only works while on the pairing screen. As soon as i leave the pairing screen the automation stops working. Does anyone have any thoughts?