Ollama function calling

Configuration Voice Assistant
1

First of all: great addition in the 2024.8 release to add function calling to Ollama! I can see there has been a lot of work put into evaluating the different models, that’s really great!

I am currently testing it by asking it to turn the lights on and off, and for me it is only able to run the first command I send to it, and subsequent messages fail.
This seems to be independent of the prompt and the max. number of history messages parameter (I set it to 0).

I have yet to inspect the Ollama logs in detail, but one entry caught my attention. This only appears on subsequent messages, not on the first command I send to the LLM.

level=DEBUGsource=prompt.go:51

msg=truncating input messages which exceed context

lengthtruncated=2

could this have anything to do with the function calls not being triggered by the LLM?
Does anyone have a similar experience?

Aside from this message, there is no error in the logs regarding entities not found or similar. The LLM just responds that it cannot fulfill the request.

2

I’m going to answer my own question here.

After having a closer look, it seems indeed that for subsequent calls the prompt is longer, as the function definitions are added to the prompt (not on the first call).

I then found this thread which states that the default context length for the llama3.1 model is set to 2048, and the Home Assistant prompt seems to exceed this length on subsequent calls. The system prompt then gets ignored.

Until there is an option to set the context length in Home Assistant and add it to the API call to ollama, a workaround is to create a ollama Modelfile with the increased context length.

You can follow the example here ollama/docs/modelfile.md at 023451ce471e7781bee65505011c48b9e5541811 · ollama/ollama · GitHub

I created a Modelfile with

FROM llama3.1:latest

PARAMETER num_ctx 8196

That seems to work.

2 Likes