This is something that bugs me with existing voice assistants. You want to change the state of a heap of lights, but one of them you want to stay as-is.
Therefore, you should be able to say something like:
set [[the] brightness [of]] [[all | every] (light | lights) in] <area> [brightness] ([except [for] <name>] to <brightness> | to <brightness> [except [for] <name>])