Hi guys, thanks to the information in this topic I got LocalAI to work nicely with home assistant.
I was already playing around with LocalAI and similar projects and got this to work quite fast and nicely on my desktop, as long as I run the smaller models and keep the prompts short.
To make it all a bit more sustainable I did the following:
dekstop sleeps after 15 minutes
I use an almost empty template, so I can generate templates with automations
reducing the prompt speeds up some LLM’s
the automation checks if the desktop is on, otherwise Wake-On-Lan
then call the service “conversation: process” with your generated prompt
send your llm’s prompt wherever you want (notification, tts media player, etc.).
Sure, this does not replace the conversational assistant, but it does allow you to generate LLM messages with some nice prompt engineering with limited hardware, on your own infrastructure .
I’m pretty green to the whole HACS and LLM thing. I managed (i think) to get the LocalAI running, but after installing the hass-openai-custom-conversation I’m a bit lost. Can someone give me a hint?
Install local-ai
Setup model
Install hass-openai-custom-conversation
Add custom component to your hass installation
Set first field to any string, set second field to the address of local-ai installation
Configure hass assist to use custom openai conversation as conversation agent, set options to contain instructions specific to your setup and model name
The two steps with questionmarks above are where I’m getting lost. I’m guessing the last step is adding the assistant through the regular HA Voice assistant interface.
Hi,
I can’t be sure where you get stuck, but I think you have to go to your integrations page and add the just installed custom component . You can find it by simply looking for custom OpenAI conversation, clicking that will start the wizard where you can “set the first field to any string” and where you have to point to your localAI installation.
I got LocalAI running now and working great. I’d like to incorporate this in HA but since the writing of this, it seems that ggml models are no longer supported for LocalAl and it says to use a gguf model.
Can you please share a model for LocalAI (pref 13b), that would work with you integration and if possible, the settings you used in the files (ie. yaml, tmpl, etc).
I’ve played around with localAI a lot before I implemented it in HA, so I got used to setting up prompt templates and installing new models as instructed at localai.io. I would recommend to follow some of the materials over there.
At the moment I’m still using ggmlV3 files, but it shouldn’t be too hard to get gguf working.
How did you get the extension working with text-gen-webui?
When I put in the api key and address and click submit it says “unknown”
Took me a while to figure out I had to use Samba extension to install the custom addon on the OS version of HA. This is my first time.
Remember to change default “https://” to “http://” from integration url if you don’t use SSL.
I’ve been running text-generation-webui in Raspberry pi 5. Due to hardware limitations it takes a couple of minutes to process outputs, but I use LLM models + piper only to create different non-critical automated announcements from weather, electricity price etc. so it is not really a matter of time. With peak power consumption under 20W it is totally acceptable.
I used LMStudio instead to run the LLM and the API worked.
I setup remote access with Cloudflare and the HA app on my phone.
Now it wont control my devices and I see it is specificaly programmed to refuse.
What obvious thing am I missing now to fix that?
I have a pretty reliable and mostly competent LLM-powered conversation agent using Text-generation WebUI with this new custom component. So far it is working very well, although occasionally the model hallucinates and gives totally wrong answers to questions. For controlling exposed entities it works better than any other component that allows LLMs.
Yes I had time to play with home-llm yesterday. It never failed (to turn devices on and off) in my test. I’m realy impressed. I even got it to stop acting so robotic and have a conversation by slight modification of the prompt. Changing the prompt too much causes weird responses, not that I know what im doing.
The model is small, so it can run on HA hardware, but it would be great if someone could train a 7B model so we can run on GPU. Im using Obbabooga on RTX3060 12G, and HA is on VirtualBox.
Have you tried using the largest model from here? I only have a 1060 but can still handle the largest one, which might give slightly better results when modifying the prompt.
Care to share your modified prompt here? The only change I added was a way to get it to tell me the current time.
i followed this however when i try to set up the integration i put in the details for my local AI instance but it just says “unknown”
Logger: custom_components.custom_openai_conversation.config_flow
Source: custom_components/custom_openai_conversation/config_flow.py:90
Integration: custom_openai_conversation (documentation, issues)
First occurred: 09:51:02 (19 occurrences)
Last logged: 10:32:17
Unexpected exception
Traceback (most recent call last):
File "/config/custom_components/custom_openai_conversation/config_flow.py", line 90, in async_step_user
await validate_input(self.hass, user_input)
File "/config/custom_components/custom_openai_conversation/config_flow.py", line 70, in validate_input
await hass.async_add_executor_job(partial(openai.Model.list))
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/openai/lib/_old_api.py", line 39, in __call__
openai.lib._old_api.APIRemovedInV1:
You tried to access openai.Model, but this is no longer supported in openai>=1.0.0 - see the README at https://github.com/openai/openai-python for the API.
You can run `openai migrate` to automatically upgrade your codebase to use the 1.0.0 interface.
Alternatively, you can pin your installation to the old version, e.g. `pip install openai==0.28`
A detailed migration guide is available here: https://github.com/openai/openai-python/discussions/742
.
. You can find it by simply looking for custom OpenAI conversation, clicking that will start the wizard where you can “set the first field to any string” and where you have to point to your localAI installation.
(running HA bare metal on a NUC)