How to connect Llama.cpp server to Home Assistant?

Configuration
, , , ,
1

Hi, I was able to get Ollama to connect to home assistant to serve a local LLM, but I’ve moved to using more llama.cpp for fine grain control. However, when I go to try to connect things together again, I haven’t found a way to connect a home assistant to my local llama server.

Has anyone been able to get to work? I see a thread that seems to use a specific llama.cpp python docker container. Was wondering if there is a way by just linking into the the llama-server? I’m able to get open webui to connect that way via a local lan ip address and port, but have tried using extended openai and custom conversations without any success…

Any help is appreciated!

2

I was able to connect using the Custom Conversation integration, selecting OpenAI an using the llama.cpp server address: http://llamaserverip:8080/v1 and anything for Api key

3

Is this still the best integration methodology?

4

Why not use LM Studio as the frontend for Llama.cpp? Then you would use extended open conversation from HACS.

5

This hacs intergration Local OpenAI LLM, GitHub - skye-harris/hass_local_openai_llm: Home Assistant LLM integration for local OpenAI-compatible services (llamacpp, vllm, etc) · GitHub, function great with llama.cpp and home assistant.

2 Likes
6

The same author har made the integration, Tools for Assist GitHub - skye-harris/llm_intents: Exposes internet search tools for use by LLM-backed Assist in Home Assistant · GitHub, which gives
Additional tools for LLM-backed Assist for Home Assistant:

  • Web Search powered by your choice of Brave or SearXNG
  • Location Search powered by Google Places
  • Wikipedia
  • Weather Forecast
  • YouTube Search and Playback
2 Likes
7

I’m also using Local OpenAI LLM now and it works really well as it works as the native integration with better support for vision, structured output, AI Tasks.
@haraldov, I didn’t know about the Tools for Assist, thanks for sharing, I will definitely try it.

1 Like