OpenAssist - ChatGPT Can Now Control Entities!

[Hassassistant]

OpenAssist Beta - Home Assistant OpenAI GPT4 with Pinecone Index

Video Walkthrough

OpenAssist Project: Controlling Home Assistant with ChatGPT4 & Pinecone!

Overview

Hey there! Just wanted to let you know that this project is still in the beta stage. Sure, it’s functional, but it’s got a couple of quirks (like not always finding the right entities after a query).

This is a one-man show and I’m knee-deep in the development process. I’m open to any ideas from the community. Your suggestions could really help in shaping this into something spectacular. So, dive in!

This Home Assistant custom component creates a Pinecone index containing all your Home Assistant entity details. This allows you to make queries or ask questions using the input_boolean.openassist_prompt, such as “turn my kitchen light off” or “what’s the current state of my kitchen light”.

This query is sent to the Pinecone index to find the closest matching entity, which is then sent to the ChatGPT4 model, returning the necessary data to perform the corresponding service call action.

Integration Install

1. (Manual) Copy the OpenMindsAI folder to your Home Assistant’s custom_components directory. If you don’t have a custom_components directory, create one in the same directory as your configuration.yaml file.

(HACS) Add this repository as a HACS Integration: GitHub - Hassassistant/OpenAssist

2. Restart Home Assistant.

3. Add the following lines to your Home Assistant configuration.yaml file:
   (See below for prerequisites)

input_text:
  openassist_prompt:
    initial: ""
    max: 255

  pinecone_index:
    initial: ""
    max: 255

openassist:
  openai_key: "sk-...s1jz" #YOUR_OPENAI_KEY  
  pinecone_key: "b9a09c6a-...db2" #YOUR_PINECONE_ENVIRONMENT ID
  pinecone_env: "us-west1-gcp-free" #YOUR_PINECONE_ENVIRONMENT ID
  included_domains: "light, weather" #WHICH DOMAINS TO INCLUDE IN PINECONE DB

sensor:
  - platform: openassist
    your_name: "YOUR_NAME" #Optional if you want ChatGPT to know your name.
    mindsdb_model: "gpt4hass" #MINDSDB MODEL NAME.
    mindsdb_email: "[email protected]"
    mindsdb_password: "Your_MindsDB_Password"
    notify_device: "alexa_media_office_echo" #Optional, this sends each ChatGPT response to your notify entity.
    #Can be any of your Notify entities. (Phone, Amazon Echo etc)

# If you need to debug any issues.
logger:
  default: info
  logs:
    custom_components.openassist: debug

4. Restart Home Assistant.

Example Lovelace Card

?raw=true)

square: false
columns: 1
type: grid
cards:
  - type: entities
    entities:
      - entity: input_text.pinecone_index
        name: Index Creation (Please type your ENV ID and hit enter)
  - type: markdown
    content: '{{ state_attr(''sensor.openassist_response'', ''index_status'') }}'
    title: Pinecone Index Status
  - type: entities
    entities:
      - entity: input_text.openassist_prompt
        name: OpenAssist
  - type: markdown
    content: '{{ state_attr(''sensor.openassist_response'', ''message'') }}'
    title: OpenAssist Response

How to use

1. Type in your Pinecone Environment ID in the Pinecone Index input_boolean.
2. Hit enter.
3. Your Pinecone index will be created, this will then upload all your Home Assistant entity data to the index.
   Please allow 10 - 15 minutes for the proccess to complete, dependant on how many entites you have.
   
   
   Be aware: There is a known issue in Pinecone free tier, creation of new indexes getting stuck in a loop. This is out of my control, on one occasion I abandoned the index creation, and started this whole proccess again with a new Pinecone account.
   
   
   
   
   
4. Notifications on the Index creation will be send to the OpenAssist Response entity.
   
   
   
   
   
5. Send a question or query.
   
   Example 1
   
   
   
   Example 2
   
   
   
   Example 3
   
   
   
   Example 4
   
   

Prerequisites

You need to have the following accounts and their corresponding keys:

• Pinecone: You need a Pinecone account with an API key, along with the Environment ID (example: us-west1-gcp-free).
• OpenAI: You need an OpenAI API key, used for embedding.
• MindsDB: You’ll need a MindsDB account, and the name of your MindsDB model.

Creating the AI model in MindsDB

1. Create a free account on MindsDB and login. You can do so HERE.

2. Navigate to the MindsDB editor. You can find it HERE.

3. Create your AI model. In this example, we’re creating an OpenAI GPT4 model named gpt4hass.
   You can replace gpt4hass with your preferred model name. Execute the following SQL query to create your model:

CREATE  MODEL mindsdb.gpt4hass
PREDICT response
USING
  engine  =  'openai',
  max_tokens =  2000,
  model_name =  'gpt-4',
  prompt_template =  '{{text}}';

Click “Run” to execute the query and create your model.

Pinecone API Key and Environment ID

1. Create a free account on Pinecone and login. You can do so HERE.
2. Wait for “Project Initializing” to finish.
3. Navigate to the Pinecone API Key page and take note of the API Key (Value) and Environment ID.
   

OpenAI API Key

1. Create a free account on Openai Playground and login. You can do so HERE.
2. Navigate to the OpenAI API Keys page. You can do so HERE.
   
3. Create a new secret key, and take note of the API Key.

[vingerha]

I am not sure if I like where this is going but I like what you are doing

[Hassassistant]

Only way forward in the AI race is to embrace! haha.

[fugley]

what’s the chance of be REALLY lazy and combining something like alexa to do text to speech for your text input field

[Hassassistant]

Well this is my next Todo.

I think voice inputs is an essential for this project, I’m just figuring out what direction I’ll be attacking this at.
Open to ideas. I’ve just run out of time today to get around to it.

[nickrout]

What about willow?

[poudenes]

is it right that you need a paid MindsDB account? Or can you use also the open source version hosted by yourself using docker?

[Hassassistant]

@nickrout

willow

Just use the free cloud mindsDB env for this project.
I want to do away with using a session cookie for auth on this integration, so i’m going to be looking at using MindsDB SDK as a pose to their API.

But certainly for now, just follow the guide on the github repo, setup the cloud mindsDB environment and use their Web UI SQL Editor to create your GPT4 model.

[Hassassistant]

Assuming you have TTS Piper installed, you can use this automation to have the piper addon/integration read the current OpenAssist sensor message from GPT4.

alias: Piper TTS - Openassist
description: ""
trigger:
  - platform: state
    entity_id:
      - sensor.openassist_response
condition: []
action:
  - service: tts.speak
    data:
      cache: true
      media_player_entity_id: media_player.your_media_player
      message: "{{ state_attr('sensor.openassist_response', 'message') }}"
    target:
      entity_id: tts.piper
mode: single

[mvenus (Moritz)]

Works great for me! Thanks! The main issue i have is that it doesnt really understand when to turn on/off a group.

For Example when i say “Turn off all lights in the hallway” it decides to turn off one light instead of the group.

Sometimes however it decides to try to turn off the group, but makes up some group that doesnt exists (like "group.all_lights_hallway).

i included the group domain. And my groups are named correctly (“light.hallway_light_group”/“Hallway Light Group”)

[Hassassistant]

Yeah so my response here is to go into the sensor.py file for OpenAssist in your custom components, and find the section which includes the initial prompt that gets sent to ChatGPT.

You could tinker with this prompt and adapt it so it’s specific for your home assistant setup as much as you need.

I’ve added my most recent YouTube video which also mentions this prompt section at the end of the video.

I feel I can only go so far with this project before the community users will then need to start adjusting this component themselves slightly to suit there needs.

Hope this somewhat helps.

[vingerha]

My first steps into this topic
Installation went OK and I can send light on / off message.
The weather is not properly detected, all values are “None”
I tried to add/remove an item to the shopping list, it comes back with a good direction but this does not work.

response: {
  "domain": "shopping_list",
  "service": "remove_item",
  "data": {
    "name": "milk"
  },
  "message": "Milk has been removed from your shopping list."
}

So pretty nice for a beta
It is however slow, request<>response 25s

[Hassassistant]

Thanks for the feedback.
Same as above, the Initial Prompt would need to be tinkered with within the sensor.py file.
I think this needs to be an addition to the custom component, easy access to change what is included in the initial prompt, to better tailor fit the GPT4 experience to your needs.

Unfortunately, the response times will be slow while using the free instance of MindsDB, this will be a shared resource, so you could purchase a premium subscription with mindsDB to get around this.

Until GPT4 becomes widely available to the public, I dont see the point in including the OpenAI GPT4 API solution for the Queries.
Also, the potential for using local Chat Completion models is there, but a local model currently struggles to out perform GPT4 all round, and they require a lot of GPU power to use.

[vingerha]

@Hassassistant

purchase a premium subscription

Responsetime is my main issue now to move further. HA is improving its internal service in a nice way too but for the moment needs to be very precise where this one can be commanded’vaguely’…more NLU

Personally I have no problem paying for a service but at my current low level of knowledge/experience I am not sure what to do

• premium: they have “open source” and “starter” and before engaging I need to know if this really helps, paying and then not seeing change is something that happens more than occasional.
• I need to get an idea of the expected cost, recently setup something for aviation data and within 2 days I was at 20$ where I was only interested in 1 flight on one specific day (yes… it is my own doing)

Their price/hour is (for me) quite difficult to interpret, if I have one call each hour is that then 24 hours in a day ( I guess so). I also donot want to pay a hefty base-price for something that hardly has any calls…well, you know what I mean

In the end, maybe I should just live with lowish performance and test it out

[vingerha]

@Hassassistant

would need to be tinkered with within the sensor.py file.

As a beginner, how would I do this? Examples maybe?

[Hassassistant]

Yep lets keep everything free(ish) for now.
This is only a proof of value project at the moment, purely to see what kind of interest i get from the community.

I will build on prompt examples later on when I get time to study this integration more.

[vingerha]

I tried to analyse the sensor a bit … do be patient as this is new to me …
What I seem to see is that you create an instruction to openai via the “prompt”?
For the shoppinglist this then won’t work because this currently enforces the ‘entity_ïd’
If above is true then the prompt could be amended OR the follow up logic could exclude the entity_id, but I am not sure what logic to apply as I donot see added value to “exclude if shooping_list”…it needs to be broader and not hardcoded.

[mrhand]

This is working and fun but yeah in the proof of concept stage and not really for regular use with quick responses yet. My understanding for using MindsDB is that it gives us both a way to try out the openai API for free, and also gives us access to gpt-4 that isn’t even widely available yet. nice!

I looked more into the MindsDB pricing but they’re geared toward businesses with large databases, we don’t really need the SQL based stuff but it’s a convenient demo option at the moment. You could run the open-source version locally, but then you have to supply your own OpenAI API key so that doesn’t help. The “starter” package is $0.70/hr for their service plus the dedicated cloud instance, the recommended one is a g4dn.2xlarge at $0.752/hr. You could shut it down when you’re not using it, but that means ~$1.50/hr to experiment with hopefully faster responses and you probably wouldn’t want to leave that running.

When OpenAI opens up gpt-4 more widely then we’ll have the option to pay for that on-demand API directly. All in all, nice work I’m here for it!

[vingerha]

Thanks and re-iterating it to be nice work

[Hassassistant]

Yeah so I only ran into MindsDB on a separate project I was working on for main stream work.
Figured it’s pretty handy to have MindsDB for the time being to test out GPT4 funcionality.