[Custom Component] extended_openai_conversation: Let's control entities via ChatGPT

EDIT: SOLVED! Just needed to run different model. With Luna it worked. Not as reliably as GPT 3.5 Turbo, but pretty well.

This component it awesome! I have it working with OpenAI and GPT 3.5 Turbo, but I’d like to have it work locally. I’ve installed LocalAI on a server and have it successfully run the mistral-7b-openorca.Q6_K.gguf model.

The Mistral model responds really fast to queries and all is peachy. Except - when it is about to call a function, I get this reply from the Home Assistant UI:

Something went wrong: function ‘None’ does not exist

And in the LocalAI log I get this:

8:52PM DBG Function return: { “arguments”: { “message”: “Slå på kontorlampa” }, “function”: “answer”} map
8:52PM DBG Response: {“created”:1707486366,“object”:“chat.completion”,“id”:“1eb2d6a5-ced8-40ae-9883-fd3fdd599df4”,“model”:“mistral”,“choices”:[{“index”:0,“finish_reason”:“function_call”,“message”:{“role”:“assistant”,“content”:null,“function_call”:{“arguments”:“null”,“name”:null}}}],“usage”:{“prompt_tokens”:0,“completion_tokens”:0,“total_tokens”:0}}

The exact same call works with GPT 3.5 Turbo and I use the same prompt, functions definition and other parameters as far as I can see.

If I enable “Use tools” for the Mistral model, I get a confirmation that the requested action is carried out, but nothing actually happened to the state of the referenced entity.

Any pointers? Doesn’t the Mistral model support function calls properly, or is there something that needs tweaking from my side?

1 Like

Yes, I was able to get this working. Here’s the final version of it that I used, you can see I added copious comments to try to nudge it in the right direction:

- spec:
    name: play_music
    description: >
      Use this to construct the input to play_music function Search for
      and play some music. If the user wants a song, then use only song and
      artist. If the user wants an album, use the album name and artist name. If
      the user doesn't want a song or an album, probably they are looking for a
      playlist, so use only playlist in that case. The user might also want to
      just hear the top tracks from an artist, in which case use only artist. 
      Use category only if you are sure the user is looking for a particular "vibe" 
      rather than something specific. 
    parameters:
      type: object
      properties:
        song_name:
          type: string
          description: >
            The name of a song to be in the search query. 
            Correct any mistakes from the user, e.g. "lept it go" becomes "Let It Go".
        album_name:
          type: string
          description: >
            The name of an album to be in the search query. 
            Correct any user mistakes, e.g. "Chery Tee" becomes "Cherry Tree".
        artist_name:
          type: string
          description: >
            The name of a musical artist. 
            Correct user mistakes and complete the name if the user shortened it, e.g. "Elvis" becomes "Elvis Presley", "mbv" becomes "My Bloody Valentine", etc.
        playlist_name:
          type: string
          description: >
            The name of a playlist. 
            Prefer Spotify's generated playlists and Spotify's playlists in general.
        category_name:
          type: string
          description: >
            Use categories if you want something like a genre, for example,
            Hip-Hop, Indie, In the car. If you don't want a random playlist with
            this category, then don't set this variable. This name needs to match an exact category on Spotify
            so don't set this variable unless you are very sure it will be a hit, e.g. "pop" will be a hit, "dance" will be a hit,
            but for a query like "sleepy music", instead come up with an appropriate album or artist using your knowledge and search for that instead.
  function:
    type: script
    sequence:
      - service: spotcast.start
        data:
          limit: 20
          force_playback: false
          repeat: "off"
          offset: 0
          entity_id: media_player.<your spotify media player>
          # optional - I specify a device ID here which I got from https://developer.spotify.com/documentation/web-api/reference/get-a-users-available-devices so that it always plays back from a certain device. If you don't include this, it will playback from the current Spotify source device
          spotify_device_id: "<your device id>"
          category: "{{ category_name | default('') }}"
          search: "{% if song_name is defined %}{{ song_name }}{% endif %}{% if album_name is defined %} {{ album_name }}{% endif %}{% if artist_name is defined %} {{ artist_name }}{% endif %}{% if playlist_name is defined %} {{ playlist_name }}{% endif %}"        

And I also add this to the prompt:

For the play_music function, be really smart and correct the user's query, for example, if the user asks "play some Elvis",
you should set artist_name: "Elvis Presley" since you know the user probably means Elvis Presley.

Now with all that, I can ask complex queries such as:

Play that band that always has 2D cartoons of themselves in their music videos

(plays Gorillaz)

and other such fun “smart” tasks.

Otherwise, I was having a constant problem where even a simple query e.g. “play Elvis” would result in it playing a song with a title “Elvis”, etc - lot’s of obvious mistakes, and now it is better, although YMMV.

I am still playing around with this and refining it but it is pretty fun to use already. It does have issues, both with correctness and also doing things like playing music when you did not want it to, that kind of thing.

3 Likes

The state variable is inside the for loop so it evaluates it for each entity. The if statement decides whether to use the actual state or the attribute depending on if the attribute exists or not. Hope that makes sense?

Makes complete sense, if I only I had read your post with more attention.
Thanks for the extra clarification effort though :slight_smile:

Hi Mugga,

I installed Proxmox on one of my computers and activated pci passthrow successfully (Intel i5; 32Gbits memory; RTX 3080) and created a VM using ubuntu 22.04.

LocalAI is installed and I tried both luna-ai-llama2-uncensored and openchat-3.5-1210.

HA connects to the llm but answers are not very good… It seems able to answer my hello but then tries to access the services and fail.
image

Now applying all updates in HA and trying different configurations.

Did you have more success with ollama?

Hey jem,
yeah I also got it working with ollama and tested with a few models. But I also can’t get it really working. I’m not quite sure if the problem is with the models and they need special training or if it is on home assistant side.

Which Luna model are you using? I’m having a similar problem with dolphin-2.6-mistral-7b.Q5_k_M.gguf.

Hello,
before I start to play with this integration, does it has sense, according to you, if HA is running on RPi or it would be too slow?
Thanks for your suggestion

I run on a Home Assistant Yellow (RPi 4 2GB) it runs fine. The processing is done on the OPENAI side.

1 Like

I run on a Home Assistant Yellow (RPi 4 2GB) it runs fine. The processing is done on the OPENAI side.
@Solarflor

Why?
I have noted that the functions copied in the dedicated ‘Assistant for Automation’ are the same that I found in ‘Assistant’

Im running this using a local LLM with the laser dolphin 2x7b model and LocalAI and it works pretty well.

LocalAI is installed on another computer in my home and i point this integration to to that.

1000016518
1000016517

1 Like

Hi everyone, i have this problem with extended conversation …

With standard OpenAI works fine

With “gpt-4-0125-preview” works fine

If I use “gpt-3.5-turbo-0125” or “gpt-3.5-turbo-1106” doesn’t work

Why?

@viperxeno Because gpt-4-0125-preview token limit is 128K while gpt-3.5-turbo-0125 limit is not. (I think 8K or 16K or so?)

1 Like

That was the sense I got from the text of the error message, so I suspect you’re right.

1 Like

You are probably exposing too many entities. Unless you are happy with the token pricing for gpt4, I would recommend to step down on the number of exposed entities and use gpt3.5 instead.

1 Like

Yes I exposed 850 entities :sweat: I’m trying with 100 entities and works fine.

this is epic! Do you have the updated spec I could pinch please?

I expose more than 1000 entities in arears, I think you have a wrong setting. decrease the possible response token to something like to 150-500.
But for costs reduced and speed optimation is useful to reduce the entitis

With gpt 3.5 or 4? If 3.5 as OP stated quite surprising that you are not exceeding max tokens. Have you had a look at number of tokens used in your requests?