Right?
This is the box I intend to run Friday’s backbone on in the future- I already have one on preorder. You will be able to run VERY large context with large param models easily. (it brings >1000 TOPS to the party) The proof will be what one of these looks like costwise v. a PC outfitted with the latest 5090 breed… (These are technically past 5090 territory for AI inference, so I’m interested in cost/TOPS detail and benchmarks and plan to find out.)
THAT SAID. same machine nope. That’s gonna be Inference only baby. This thing is like a Jetson Nano ate your Jetson Nano’s kid brother for breakfast. It’s designed for NPU/GPU stuff not HA.
For me, It will be setup as a local ollama API endpoint and sit NEXT to the NUC 14 already running inference for (8b-12b models) and the rest of the homelab stack on proxmox. This thig is a different beast. And well if you’re putting out that scratch for THAT box (check MSRP) - you can have two.
I wonder what the power consumption is like?
My exact thought too and the reason why it probably is a good idea to split HA from it, so you can shut it down when the home is empty or the residents are sleeping.
It’s going to be basically like a high end gaming pc. I’ve heard numbers from 600-800w avg with max draw around 1300-1400w (makes sense can only get 1500 w out of 110/15 US) but don’t have a confirmation yet on the psu. My math says I should expect it to about equal my desktop pc running full out all the time if I’m running the workloads I expect.
Unless they have a nuclear powered option, then it could also heat your house!
I’m more interested in the “standby” power. I’ve no problem with ramping up the processing power to 11 when it is needed but at idle?
I think that’s totally going to be based on the PSU. I do t have any indication it won’t be able to throttle down. Just in most of the circles I’m playing in they’re trying to get these things to run (literally) as hot as possible as long as possible… So how much power does it use on standby is kinda hard to find out yet… My gut says it’ll be on part with a mac mini on standby base do what I’m looking at but that’s a wild guess based on tentative spec.
Mine won’t be in standby. I’m working on a work breakdown that is constantly sending small jobs that’s the only way these things can be cost efficient. You can pay for OAI API access and beat the heck out of openai models for pennies. Currently I run Friday with a very good feature set for about 40usd/mo
Looks like this for lifespan etc averages about 100usd/mo so I need to get 60$ worth of extra AI goodness just for owning it.
The difference is I hit an oai model about 1/2 times an hour with a heavy model to hit that
Id be able to run a comparable model (high bit qwen) and the only cost is power and the machine. The difference is I can run it all the time 24/7 if I want.
If you’re doing lightweight work not cost effective. At ALL. But if you want to see what it would be like to have an intelligent agent that uses tools hanging out i the house… Yeah this’ll do it.