Unfortunately, I wouldn’t expect anything nominally powerful enough for this to be less than $500US at a minimum for a few years.
That’s why I’m doing what I’m doing. Ultimately the best use of local is always churning and burning small tasks.
You’re not going to run a monster back end and flash through context on demand. Instead you’re going to cut up a the work into as a all of a job as possible ad have the result summarized in a way your Frontline ‘actor’ understands.
That one the heavy one that needs a nice voice model put it in the cloud or make sure your local model handles that. Then optimize the rest. For small model summary runs that’s exactly how I’m building.
Because right now as. Of today I’m burning ~$150/mo in tokens. Most of this is the hourly summarizer. It takes a crap ton of tokens and is a 65 second runtime on o3.mini.
If I can peel most of that traffic off and move it local (the 7b model on the NUC) then the hero model is only running when I call it, will need way fewer tokens and even a lighter model. (gpt4o-mini works with a nice vocalizer)
What about the recently announced ASUS NUC 15 PRO+? It’s Core Ultra Lunar Lake with dedicated NPU, the ad says up to 99 TOPS (probably combined, which I’m not sure if Whisper can do, especially on Linux). But still, how this box would perform in your opinion?
I’m really eager to run decent local STT with under second performance that doesn’t cost an arm and leg to buy and then feed monthly.
It’s going to be similar to the 14. I can’t tell if it’s a Battlemage or Alchemist but even though the B series battlemages are better than the A series. (14 is definitely an A Series)
This is all literally about the NPU / GPU and how much fast RAM you can attach to that silicon. Period. Your CPU does NOT matter for inference. (ok, can but should not, it means you’re not using the GPU/NPU packs. )
That said, Nvidia is 100% absolute king of the kingdom. Ollama was originally written for CUDA and everything else was built off that. Including the IPEX core ollama you must use for Intel silicon. (yes I just said you have to pick an entirely different build because team blue.) I’ve posted the docker compose for the because it was such a pain to setup.
So if I were doing midsized local inference (what you’re asking for) I’d actually go more like this: https://www.asus.com/displays-desktops/nucs/nuc-kits/asus-nuc-14-performance/
It’s a NUC with a discrete 4070.
I heard that you can use a coral tpu to run the ai on is this true? If not what option do I have for my ASUS nuc 14? I also have security cameras where I wanted to implement object recognition
You could use an external GPU attached to the nuc just for offloading the LLM on to.
Units? 8 what, for example? Is this the power for just the GPU or the whole system? While actively processing text, or idle?
Thanks.
That Looks like Watts. That would align with what I see from my a770 and my 5070ti…
I got a GMKtec Intel 12th Twin Lake N150–NucBox G3 Plus Mini PC from this summer before I realized that I needed AI 
For Christmas I wished for 32 GB RAM and a extra SSD thinking that would give me the power for basic LLM but reading this thread I understand that wouldn’t even scratch the surface of anything useful!?
The pc got two m.2 slots (a 2280 and a 2242). At least the 2280 is PCIe 3.0 (according to specs).
I googled and found https://hailo.ai/products/ai-accelerators/hailo-8-m2-ai-acceleration-module/#hailo8-m2-overview a NPU that delivers either 13 or 26 TOPS depending on the edition.
Would that make my setup usable or does it need to be ~100 TOPS to be worth the effort?
If it isn’t VRAM then it is useless for any reasonable LLM.
According to Google: The Intel N150 processor doesn’t have dedicated VRAM; it uses Shared System Memory.
I currently got 8 GB and could read that isn’t enough - hence the bump to 32 GB so I have 24 GB for LLM.
I found the Hailo-8 is for vision only. I need something like Hailo-10H for LLMs which also carries some DDR https://hailo.ai/products/ai-accelerators/hailo-10h-m-2-ai-acceleration-module/#hailo10m2-overview
You need real vram.
Please inform yourself well about LLM hardware otherwise you will get yourself into a disappointment.
Yes, it will work… if you wait 25 seconds for an answer instead of 1-2 seconds