There are blueprints for similar devices. I suggest you review what they do to solve the same problem.
Basically they listen to every event the device produces (just like you do with an Event Trigger that listens for hue_event) but they move the decision-making out of condition and into action.
In action, a choose determines what to do based on which device caused the trigger and which event it produced.
Some of the blueprints might be a bit more complicated than you require because they are designed to handled event_types produced by either Zigbee2MQTT or ZHA plus their different subtypes. Your case is simplified because you only need one event_type, namely hue_event.
Here’s a blueprint you can mine for ideas:
Let me know if you need more help.