Use different languages for STT and TTS in an Assist pipeline

Feature Requests
,
1

The one thing I don’t like about Assist pipelines is that they are fixed to a single language. English only speakers probably don’t care about this, but for everyone else, removing this constraint would open up a lot of new possibilities.

For example I use my voice assistant in Croatian. I could use it in English, but I refuse to, it just doesn’t come naturally.

However, I would really like if the assistant could reply in English. The main reason is that there is really a limited choice of Croatian TTS voices, and they don’t sound so great. And the ultimate goal is to have it reply like e.g. Darth Vader, HAL, Yoda, Bender… Of course none of these voices exist for Croatian. Even if they would, it would just sound stupid.

Currently, I use the Astromech integration for TTS, as an alternative, since it pretends it has support for all languages. And yeah… I even prefer the R2D2 gibberish over the Croatian TTS voices that are available :slight_smile: They aren’t completely terrible, but they sound too robotic and choppy, pronounce some words too fast, can’t swear properly :slight_smile: etc. I would rather use a normal English TTS over them, since it sounds more fluent.

2

Have you found a solution? I am looking for something similar. Piper doesnt have my language so i need it to reply in some other language… Whisper can translate STT into any language, so imo it should be doable… Yes it wont be super fast but atleast usable for people that are not English natives

3

Same problem here - I need output in english :frowning:

4

For everyone still looking for the solution to this problem, let me share mine:

  1. Enabled Piper, Whisper and Ollama integrations.
  2. Uploaded my custom voice files (en_US-glados-medium.onnx, en_US-glados-medium.onnx.json) to /share/piper
  3. Copied en_US-glados-medium.onnx → pl_PL-glados-medium.onnx and en_US-glados-medium.onnx.json → pl_PL-glados-medium.onnx.json. Maybe I could rename, but I am not entirely sure.
  4. Edited pl_PL-glados-medium.onnx.json, just the language.code portion:
    "language": {
        "code": "pl_PL",
        "family": "en",
        "region": "US",
        "name_native": "English",
        "name_english": "English",
        "country_english": "United States"
    },

Other language properties, espeak.voice and dataset are all left as is.

  1. Reboot the entire server to make sure the custom files get loaded on startup.

  2. Then in Settings > Voice Assistant I configured my default assistant main language to Polish, Speech-to-text to faster-whisper in Polish (only available option) and Text-to-speech to pl_PL (again only available). However, the available voice list now includes en_US-glados-medium.

  3. In Conversation agent I selected Ollama Conversation (integration with my local Ollama hosting gpt-oss:20b) and in its configuration I enabled think before responding.

  4. Updated the instructions prompt to let it know the bilingual nature:

You are a voice assistant for Home Assistant.
Answer questions about the world truthfully.
Answer in plain text with no markup.
Assume input is in Polish and has typos.
Always answer in English.
Be sarcastic, cynical and condescending.

Now, the conversations are very funny, the gpt-oss:20b does a great job first at correcting Whisper’s inaccurate STT, secondly at translating to English, thirdly at inferring intent and acting on it (it’s too dark = need to turn on the light; asks about anybody home = report which sensors detected movement) and finally, at being GLaDOS from Portal. The conversations are a real party trick.

1 Like