Stumbled upon this and the following thread when searching something different, but this could be a solution to your problem:
Basically you define a Google Home Routine for each sensor starting a ambient sound at the end, which will be used to identify the speaker you have asked the question.