Run LLM on another machine?

[Sawtaytoes]

I want to use the Home Assistant smart speakers, but I don’t wanna change from running a Raspberry Pi 4.

I don’t think I’d get much benefit from upgrading, and I’d have to find a suitable rackmount and PoE solution. Not in the cards for me.

On the other hand, the Home Assistant smart speakers let’s you optionally use a cloud LLM. Does that also mean I could install that on my NAS which has an Eypc server CPU with plenty of horsepower? Or is it possible I could connect a USB Coral to my Raspberry Pi and connect that way?

What are my options for running the LLM without using the Raspberry Pi but keeping the entire thing local in my network?

EDIT: How I set up Ollama in TrueNAS SCALE: Run LLM on another machine? - #7 by Sawtaytoes

[Sir_Goodenough ((SG) WhatAreWeFixing.Today)]

Hi Sawtaytoes,

Look at Ollama.

Ollama

[AshaiRey]

Hsving ram isn’t that important it is vram that you need. A good graphic card will do.
You can play with it with disturbing you local HA system.
Search for the forum for ’ local llm for dummies’
Local LLM for dummies

[Sawtaytoes]

@Sir_Goodenough

Look at Ollama.

Ollama - Home Assistant

Cool, Ollama is available as a community app on TrueNAS.

@AshaiRey

A good graphic card will do.

I have an RTX 4060 low-profile on my NAS. Sadly, it’s only GDDR6 and only 8GB . That’s all I could fit in this 2U server chassis.

I have NVIDIA GPU-passthrough working for Plex on the latest TrueNAS SCALE 24, so it should work for Ollama as well. Even if not, this is a 16-core Eypc, so it should be more powerful than the RPi4 regardless.

---

I can go through those guides tonight (thanks for the links!), but I’m curious, are there any good videos on the topic? I’ve seen a few, but it just looked really complicated.

Now that the Home Assistant Smart Speaker is a thing, I’m assuming this’ll be easier? Or is it still a pain and a half to get any local AI working in Home Assistant?

[Sir_Goodenough ((SG) WhatAreWeFixing.Today)]

@Sawtaytoes

Or is it still a pain and a half to get any local AI working in Home Assistant?

This was pretty painless for mt, I’m running it mare metal on my gaming machine. The Docker install confused me with the passing the GPU thru and all that, but native worked better (for me).

[v35 (Jordan)]

OMG! i have a 1080ti i have been trying to get the Plex GUI to recognize. I have Proxmox running Truenas, I Passthrough the GPU through proxmox to Truenas. Plex runs in a container within Trunas. In the Plex website GUI under setting>Transcoder Hardware Transcoding Device is empty. If i pass throught my older Radeon R9 380x, Plex will recognize it and i can select the GPU… but streaming/transcoding 4k video is sloooow.

How did you get your 1080Ti to be reconized in trunas scale plex?

[Sawtaytoes]

Set up Nvidia GPUs in TrueNAS

1. Install drivers

Make sure you install the NVIDIA drivers checked in your app settings:

2. Configure GPU-passthrough

Then you need to configure your app to use the NVIDIA GPU-passthrough:

This is the same for Ollama as it is Plex.

Once complete, you still won’t see it in Plex, but when you enable “Auto” for “Hardware Acceleration”, it’ll work:

Verifying it works

You can verify it works with both Plex and Ollama by running nvidia-smi in the Shell.

This is what it looks like with nothing running:

This is when transcoding:

This is when using Ollama in Home Assistant:

[Sawtaytoes]

@Sir_Goodenough

This was pretty painless for me, I’m running it bare metal on my gaming machine. The Docker install confused me with the passing the GPU thru and all that, but native worked better (for me).

I got it working in Ollama, but I haven’t tried Piper, Whisper, or any of that stuff. I’d love to get those working too!