Fast (low latency) cloud models mit OpenAI compatible endpoint: Which do you use and recommend?

[Thyraz]

Ok, I tried it with default HA prompt (which is close to nothing ) and default assistant.

I can reproduce it by simply telling it to respond all numbers, decimal points and units written out like “four dot three degrees”.

After a few very short follow-up questions, that aren’t written in full sentences, it starts to show doubled answers or leaked reasoning with quite some confusion in its “thoughts”.

The interesting part is, that this happens far less (close to never) when talking english to the assistant.
Sadly that won’t work for my kids within the next years.

But I had an idea about doing “TTS normalization” in an extra step instead of telling the LLM to talk that way.
This has also the benefit, that it doesn’t affect text chats with the LLM instead of using voice (the written-out numbers and units are not that eye-friendly tbh. ).

More info to come once I see if it’s working …

[Thyraz]

Seems to work pretty fine so far.

TTS Proxy: Sits between your LLM and TTS service and allows reg-ex replacements, markdown/emoji stripping and more

This integration was made to help with TTS pronounce problems. It lets you manipulate the text that your LLM assist returns, before it’s handed to the TTS service. For example some TTS services have problem with units and decimal numbers in languages different to english. Other TTS languages might have problems with specific names but support tags with phonetic transcriptions. It also helps to reduce context size, as you don’t have to add rules to your prompt about how the response should be…

[Thyraz]

And more problems with gtp-oss-120b:

When repeating short and very similar questions about music, the same thing happens.

Like

• play radio-abc
• play artist-x
• play song from artist

I also tried with default prompt and also with different music tools / scripts in case something is wrong with the description of the tool or the parameters. Didn’t help.

The problem here is, that this happens quite often in our house.
Requests to music can be issued multiple times without other questions in a row, as the voice assistant’s session is kept active for long enough if you send new requests after every few songs.

So I decided to stop fighting gpt-oss-120b, even that the speed, pricing, and “smartness” of the modell was quite perfect.

Btw. the smaller gpt-oss-20b worked way better here, but can’t handle all the cases I want my assistant to do.

[NathanCu (Nathan Curtis)]

I’m working with Gemma4 E4B right now.

Its good. But…

I think I still need to play with Google default prompt template, they load tools last and it’s causing the cache to miss. If I move tools to the front with the system prompt then, in theory, I solve my one last problem and at least on paper should cut about 7-8 seconds off my TTFT and bring response sub 2 sec..

[Thyraz]

I’m really interested in numbers like this, about how fast can you get local vs. cloud.

Where cloud adds latency on each tool round, while locally you might be limited by GPU power.
Some day, I hope to also switch to a local solution.
Things are just moving so fast at the moment, that I still can’t convince myself that I won’t regret the purchase soon.

But I also tested a few more new models the last days after my problems with gpt-oss-120b.
See next post.

[Thyraz]

So let’s move on with a few fresh recommendations that are fast enough for a really snappy voice agent.

For comparison the numbers OpenRouter lists for gpt-oss-120b hosted on groq that I used so far:

• time to first token (latency): 0.28s
• tokens per second: 314
• input price: 0.15$
• input cached: 0.075$
• output price: 0.60$

MiniMax 2.7, hosted on groq

I already mentioned that above, so just for completeness:
Way smarter than gpt-oss-120b.
Handles every question I throw at it without explanation.

Enterprise model on groq, so has to be used through OpenRouter instead as a private customer.

• time to first token (latency): 0.3s
• tokens per second: 266
• input price: 0.60$
• input cached: -
• output price: 1.80$

Inkling Small, hosted on Baseten

Released July 30.

(Not available through OpenRouter for Baseten yet, so you have to use it through Basetens own API endpoint.)

This is the fastest thing I’ve seen so far and the only model that seems to outperform gpt-oss-120b for me in time to first token (latency).

I tested it against my Alexa+ with a few questions like:

• knowledge that is already available within the models (height of eiffel tower, …)
• tool calls that alexa can also handle like asking for temperature, starting playing radio stations or switching lights

Combined with a fast SST and TTS (Nabu Casa Cloud in my case, or Elevenlabs Flash for something that sounds as good as Alexa), the voice assistant beat Alexa+ every time with tool calls, and was on par for knowledge questions.

This is really impressive, as this was the main gripe my kids had: “It’s slower than Alexa”.

I’t also not the cheapest model, but cheaper than MiniMax2.7, even more as it provides reduced prices for cached input:

As it’s not available on OpenRouter, this values are median values of my own calls today from the Baseten Dashboard:

• time to first token (latency): 0.19s
• tokens per second: 244
• input price: 0.50$
• input cached: 0.10$
• output price: 1.20$

DeepSeek V4 Flash 0731, hosted on Baseten

This was released just a few days ago and might become a new star for applications like this.

It is also capable to answer my more complex smart home questions and has a very interesting price tag based on that.

Time to first token (latency) seems also very promising in times where the providers aren’t under heavy fire at the moment.

OpenRouter lists latency between 0.3 and 0.4s, but most requests error out due to load balancing.

You can get around that when using the model through Basetens own API endpoint, but then you will suffer from high latency at this times.
So we will most likely have to wait a few weeks until things settle …

Tokens per seconds are also varying a lot for Baseten on OpenRouter currently, where I’ve seen values between 40 and about 150 over the time.

• time to first token (latency): ??? most likely values below 0.3s are possible when the load isn’t that high on the providers anymore
• tokens per second: ??? at least 150 seems to be possible, maybe higher
• input price: 0.13$
• input cached: 0.028$
• output price: 0.26$

[rmeissner]

I’ve tried Inkling (normal), as small is, unfortunately, not available from Baseten on OpenRouter.
Inkling normal seems intelligent enough to steer my home, but feels a bit slow.
Before trying out Inkling, I’ve settled on Google Flash 3.5-lite.
Flash 3.5-lite is blazing fast (way faster and cheaper than Inkling) and also eems intelligent enough to steer my home.