I currently have Llama3 running on a server (lenovo p520) with a 12GB 3060. Works great with Home Assistant and is really responsive. Now that I have some Voice Assistant PEs I’m looking at accelerating and improving Whisper and Piper. So I’m considering adding a second 3060 to the server. Anyone know if it’s possible to segment the GPUs (1 for Llama3, 1/2 for Whisper, and 1/2 for Piper)? Home Assistant is running on a separate server.
I have a 3060 12gb as well, I’m running a dedicated machine with Ubuntu. I have whisper, kokoro tts and ollama running all together.
It’s a squeeze with 12gb of vram but it works. I find llama useless. Qwen2.5 7b is minimum, 14b runs but is slow if I have whisper and kokoro running as its more than 12gb.
You can specify which gpu number is your docker configs for whisper and piper.
Hey, I’m setting up Whisper with GPU support on my RTX 3060-equipped computer.
How exactly did you get the GPU exposed to the container?
I’ve been spending 1-2 hours on this every single night, but I always end up with the error: “CUDA failed with error named symbol not found.” I’m really stuck and don’t understand what’s missing.
Hi Gurrasv985,
I just went through this a couple of months ago and am struggling to remember. In short, I think there were two ways to install the NVidia drivers, stack ect. The tutorial I followed led me to that path. Apparently there are two ways to get there and the one I picked, didn’t work. I had to rip it all out and reinstall…the problem, I can’t remember the paths. Took me awhile to give up and start over…consider this encouragement to move that up the suggestion list.
For anybody else tracking nvidia cards and capabilities:
I’m on a 1080ti running
- Whisper in docker, with GPU support, distil-large-v3
- Piper in docker, with GPU support, en_US-ryan-high (I’m looking around for other voices, but low on my priority list).
- My LLM at the moment is Ollama using llama3.1 driving an article summarization tool and a ConversationBot for Home Assistant.