Good question
For the LLM I'm currently designing against three tiers.
- A local only LLM setup that can run on a single 16GB modern card. Right now this setup is a little slower than I'd like on my own gear - but if you follow @crzynik's setup his is faster than mine - I just haven't gotten around to incorporating his latest builds in my setups. For me this is on vLLM, Currently, bouncing between the midrange Qwen, and Gemma models.
For anyone serious about this stuff - you're going to read that entire thread...
-
A 'mostly' local but i want a little jazz when I want to show off at a trade show? That setup above but send the frontline to OAI or Anthropic latest 'mini' tool model. (Currently GPT5.4-mini, prob soon to be 5.6 or luna or whatever the small model will be) in this configuration the summary pipeline is all local and the frontline AI 'the one you talk to' is commercial / paid openrouter whatev your comfort level is.
-
Concept Car. DGX Spark (upcoming RTX, Apple M4/5 Mini / STRIX Halo unified RAM box.)
I have a single node DGX with the ultimate goal of fitting the entire build into that plus 1 sidecar NUC... Thats the game of tetris and it's not just HA+ Inference.
Everything that makes the sausage... More on that latter.
MOST of my build once I got the inference done has been in mode two - because the more useful parts happen when the summarizer is unlimited and you simply shouldn't do that on cloud or paid models and is why i ship it 'OFF'
The summarizers fortunately are designed to chop up context and right now I think you could run MOST summary jobs on a card capable of an 8-12K CTX and probably better than an NVidia RTX3xxx series? (Got an old gaming card or a buddy with an old gaming card - make it a summarrizer...)
I'm going to continue to make summarry jobs smaller and more atomic to get them on smaller and smaller cards if i can, Worst it will ever be I hope?
SO -
Cloud fully - you pick your host just think what you're sending please dont cloud summarrizer pipe it WILL hurt.
Cloud / Local Hybrid (capable local summary, chosen trusted provided for speed chonk boost when you need t talk to the agent.
Local fully - You'll need that 16g card at LEAST (but I'm trying to keep my contexts I build small to stay well below this)
For what cloud. Eh thays going to be user preference as long as you get a tool use model that supports your context size (all the big guys far exceed what you send) it's more a matter or read thier terms of use thoroughly and agree before you sign on the dotted line...?
Does that help?