HomeKit bridge asking "Allow 'Device/Entity' to share data with Home Assistant" when using spoken ask Siri

No idea why or really when this started (maybe when I went onto iOS 15) but now if I ask Siri (either on my phone or on HomePod) to access one of the entities or devices I’ve added to Home via HomeKit Bridge, I get asked to Allow or Don’t Allow. Issue is, if I’m trying to use HomePod Siri to control the device, she just keeps repeating it over and over again. She won’t listen if I say “Allow” or any variation of words I’ve tried to convince her I give permission. At least on my phone there’s an Allow button for me to push.
I can’t find any setting in Home or on my iPhone settings to give permission for all devices.
Has anyone else had this issue or at least know the magic words to convince Siri to Allow a device/entity permission?
Edit: finally got her to accept a verbal “Allow” as a response. Only happened once, but hopefully will keep happening.