Gemini Error 1099 When Trying to Control HA Devices

I recently switched from Google Assistant to Gemini for Home on my Nest speakers, which had the unintended consequence of switching to Gemini when I say "hey google" on my phone. I use "hey google" originated commands heavily in my environment, with approximately half of all commands controlling HA devices via Google Home.

I've been using Google Assistant in this way for years, and after switching to Gemini everything was working fine for the first few weeks, but now whenever I try to control an HA device from any of my Android phones via hey google, Gemini returns error 1099 on my phone with no additional information. My Nest speakers work perfectly. If I manually tag "@google home" via a text based command from the Gemini app on my phone it works, it just never works from a hey google prompt.

I've grown really accustomed to this functionality and it's become a huge drag that it's not working. Does anyone have any suggestions?