TTS Proxy: Sits between your LLM and TTS service and allows reg-ex replacements, markdown/emoji stripping and more

Share your Projects! Custom Integrations
1

This integration was made to help with TTS pronounce problems.
It lets you manipulate the text that your LLM assist returns, before it’s handed to the TTS service.

For example some TTS services have problem with units and decimal numbers in languages different to english.
Other TTS languages might have problems with specific names but support tags with phonetic transcriptions.

It also helps to reduce context size, as you don’t have to add rules to your prompt about how the response should be formatted.
Another benefit to not handle this in the prompt: The text returned is not optimized for speech and therefore often better for reading.
That helps in times when you chat to the assistant without speech.

It identifies itself as a TTS integration, so you can choose it as TTS service in the options dialog of your voice assistant.
In the options of the tts-proxy you can select a target TTS service where the manipulated text should be sent to.

It also uses streaming when the selected target TTS service supports it.

Features:

(Regex) Replacements

add literals or regex that should be replaced.
Some examples what’s possible:

°F → degrees
~ → about
(?<=\d)\smm\b → millimeter* (replaces mm only when a number or a number followed by whitespace is detected)

Number to string conversion

Detects numbers (including decimals) and convert them to a string optimized for speech.
Supports multiple languages as output format.

13.23 → thirteen point two three

Markdown Cleanup

You can activate different rules like removing bold/italic, header markings, URLs in links, quote/code markers, table formatting and more.

Emoji handling

You can decide to read the emojis like Siri does on Apple:
:slightly_smiling_face: → smiling face
or to simply remove them.

Dates

This is an advanced feature for German and English, with a fallback to simple convertion to numeric string conversion for other languages.
As I don’t speak any other languages, I couldn’t create the rules for more.

03/01/2022 → January third twenty twenty-two
05/23 → May twenty-third

For German:
31.10.1994 → einunddreißigster Oktober neunzehnhundertvierundneunzig
As German has more complicated grammar rules here, it also supports cases like
am/zum einunddreißigsten …
der einunddreißigste …
and so on.

1 Like
Fast (low latency) cloud models mit OpenAI compatible endpoint: Which do you use and recommend?
2

this is awesome, and a great way to do it. To be honest this feels like something HA should be doing as post processing before TTS is sent since mainly HA pipeline knows if this is a voice or text request, but this is awesome and I’ll definitely be moving to this and removing some context.

2 Likes
3

Some feedback on the number analyzer, I ran into this with my prompting, there needs to be a special case for years. It’ll say something happened in “nineteen thousand seventy one” instead of “nineteen seventy one”.

1 Like
4

When I had the idea, I was also wondered why nobody else created something like this before. :grinning_face_with_smiling_eyes:

About the years:
This might be a more complicated edge case than the ones I already handled, as it will be more hard to identify a single number as a year.

The date formatting kicks in before the basic number formatter.

But if we simply replace everything beside 1900 and 2100, this will also effect numbers that aren’t years.

But most likely this won’t happen as often in a smart home context as the other way around …

5

Yeah I assumed something like a , could indicate that but that depends on language, region, and probably model. I know we look up a lot of facts so it gets brought up a lot more than raw numbers.

1 Like
6

Agreed, thanks for this. I’ll give It a whirl, I had been piping all responses through a jinja macro library.

I was doing the dates and numbers and was replacing newline with space to prevent pause and stutter if the tts was Google translate.

7

Ok, I added this with some checks to identify the most obvious numbers that are most likely no dates, like “1920 Watts” and similar things. You can also set minimum and maximum year that you want to be detected, if you think your fact check are mostly in 1800 - 2100. :wink:

Added also a new option to do newline → space replacements with Linux / Windows linebreaks and whitespace stripping, so only one space as replacement remains.

Also added other replacement types:

Units:
I initially configured these manually, but I think most people would like to use this, so it might make sense to build it in as easy to use feature.

There are also some special cases here like singular / plurar of the units, which are more difficult to handle with regex.
Units are detected with and without space between the number and the unit.
And I added a few common mistakes that the LLMs learned from us humans, like kmh instead of km/h.

Time / Durations / Timespans:
I guess most TTS services will handle that already fine in english.
But I had some problems in German with quite a few services.

And another note:

As the displayed text in the Assist chat won’t change, only the spoken text,
it’s not that easy to see why something sounds wrong in case of problems with this integration.

You can grab the response from your LLM in the debug view of the Voice Assistant,
then paste the text into the preview field of the tts-proxy config dialog:

image
image1132×680 21.8 KB

Might also be handy to just manually test how it reacts to specific cases.

If something isn’t replaced as intended (or you have more suggenstions),
simply post here or create an issue on Github.

2 Likes
8

New release which only applies to german, as this is specific to our habbit to glue all words together to a single one:

You can now set an option to seperate the single words in long numbers with a dash.
At least Elevenlabs Flash / Turbo will read large numbers incorrectly otherwise.
Example: 2395 → zwei-tausend-drei-hundert-fünf-und-neunzig