I have recently “upgraded” from a HA blue (aka Odroid N2+) to a HA yellow with a CM5 and also bought a HA Voice Preview.
I would like to set up a local voice assistant and am wondering whether adding a, low-power, AI board to my hardware stack would help making the experience snappier. As far as I can see, the only viable option at the moment would be the Raspberry Pi - AI KIT with 13 TOPS.
Would that make sense? Does anybody have such as setup? What is your experience?
It support tensorflow, but it looks like it shared memory with the Pi and for LLM memory is also a big requirement, where 8Gb is often too little, so I would suggest looking for other solutions.
What are you trying to solve for in your voice pipeline?
Local inference - standalone CUDA based inference server
Local voice? There’s already Cpu local voice available as addon so it would have tk be better than that so hardware acceleration on local voice, in which case make sure your voice platform of choice supports your hardware.
Personally I’m either using builtin voice or swinging all the way over to hardware accelerated Ai inference which I’ll carve off a portion of the inference server. Voice isn’t one of those just add a… Things
13 tops isn’t enough for inference and honestly I don’t see where it’s going to do anything but shave a second (over local voice unaccel) so nope doesn’t make a lot of sense and I would be staying put.
But that’s me I’m more into the what my Ai is saying than the how and I’ll spent my $ there.