Constantly lose config and have to "set up voice assistant"

Is anyone else having this? I have a Satellite1 and 3 Voice PE. I run a local Ollama for conversation, use piper and whisper for voice.

Every so often, highly correlated to HA updates, I have to go back and run setup on them again as they’ve switched back to Home Assistant Cloud (or blank) for conversation agent. I have to re-run the setup and set them back to my local LLM and set the voice again.

Otherwise I’m having an OK experience with it. I’ve invested plenty of time in setting this all up so it’s all local and tuning my prompts. But it’s annoying to have my assistant just stop work (spinning ring, doesn’t reply) or error (‘an unknown error has occurred’) or, worse, start giving me the wrong time until I re-run setup on each device.

Yes. Did you “take control” of your devices.

I recently realized this may be result of me making config changes and updates on my voice devices. I’m thinking HA disassociates device from saved setting or in same case I know I update with esphome config that doesn’t support voice and HA dumps settings that no longer apply.

For now I am thinking this is caused my actions and not HA.