Maintaining states for multiple remote controls on RF Brigde

I have an RF Bridge and four remote control with A and B buttons. I want to use A button to arm my alarm and the B button to disarm the alarm.
For each remote control there is a binary sensor with a state on, when A is pressed and off when B is pressed. My idea was to create and automation to arm/disarm the alarm, when a state changed. This won’t work since the the states will get out on sync, with four remotes.

Any thoughts on how to solved this would be appreciated