I have an old GPU - a Radeon R9 380 2GB - that is sitting unused in a cabinet. Are there any worthwhile AI models I could use on that tied to Home Assistant?
I’m currently running HA as a VM (HAOS) under Proxmox. The proxmox host is an i7-6700k CPU. I have two cores assigned to HA, along with 4GB of memory (I have more available I could assign if needed). I could toss this GPU in the Proxmox host and pass it through to HA.
I know in general AI is a huge memory (VRAM) hog and needs beefy processing. But if I’m willing to accept slow responses, are there models that would suffice with this hardware? I’m thinking less along the lines of live voice conversations and more like occasional tasks in the back-end like an automation to generate a block of text that gets displayed on my dashboard.
Its not like a little slow it’s like won’t start error and splatter. You must have the vram to support the model or don’t bother.
Models can be quantized to be smaller but think of that like a diffusion filter on a picture. Sooner or later the filter is too heavy, not doing your friend any good and blurs the picture to the point it’s unrecognizable. (read someone might be able to quantized a model small enough to fit but goodness help you with the result.)
In short small model mayyyyyybe? Like the qwen. 5b but I have zero clue if it will function and if so at what kind of speed. You’re probably better taking a pass unless you want to dabble. That size won’t drive HA and will frustrate you.