I have HAOS with the Ollama add-on hosting my models on the same machine. I want to make Ollama able to use the GPU so that the models are faster. But I don’t think HA even has GPU drivers. Does anyone know how to do this?
The Ollama add-on github makes utilizing the GPU sound possible, but doesn’t explain how to do it.
Any help is appreciated!
I was experimenting with ollama but I used it in docker container. Yeah, it can use gpu. For some reason it loves nvidia gpu but it uses a lot of gpu power. Basically to use this locally you will need an expensive nvidia gpu.
It will run with some moderate gpu but it is very slow. So i abandoned it all.
The readme of the addon says that by default it only uses cpu acceleration. My gpu isn’t super expensive, but right now it isn’t being used at all, and I think the models’ speed would be greatly improved if they could use it.
I didnt use this in addon, as I use docker compose. I tried it docker compose but it is slow and it is using a lot of gpu power. Using just cpu will probably kill your comp with ease.