Thanks for sharing this, I’ll need to test using the localai vs the text-generation-webui that ive been using
If you are using llama.cpp backend in text-generation-webui you can set the context size by defining n_ctx parameter before loading the model
1 Like
bare with me as I’m still pretty new to the LLM stuff.
I’m using an AutoGPTQ model (TheBloke/airoboros-33B-GPT4-m2.0-GPTQ · Hugging Face).
I’m assuming it cannot be done with that one, so I’ll try the model you’ve shared
Yeah, some models only support 2048 tokens sized context unfortunately, so even if you increase the setting, it might not work.
So I’m using your model and selected llamma.cpp as the loader and increased n_ctx to 6144 then loaded the model but I’m still receiving the same error in home assistant.
I’ll probably test with localai later
LocalAI could be used as a fallback for the Assist pipelines: Pipeline chaining or fallback intent
But maybe should be something more generic than LocalAI, so you could use privateGPT or anything else.
Like maybe just use the already implemented Wyoming protocol integration.
What do you think?
So I’ve got this set up using localai now and am no longer getting the error about context length.
I’ve set my vicuna-chat.tmpl to this:
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
USER: {prompt}
ASSISTANT:
And I’ve set the template within home assistant to this:
This smart home is controlled by Home Assistant.
An overview of the areas and the devices in this smart home:
{%- for area in areas() %}
{%- set area_info = namespace(printed=false) %}
{%- for device in area_devices(area) -%}
{%- if not device_attr(device, "disabled_by") and not device_attr(device, "entry_type") and device_attr(device, "name") %}
{%- if not area_info.printed %}
{{ area_name(area) }}:
{%- set area_info.printed = true %}
{%- endif %}
{%- for entity in device_entities(device) %}
{%- if not is_state(entity,'unavailable') and not is_state(entity,'unknown') and not is_hidden_entity(entity) %}
- {{ state_attr(entity, 'friendly_name') }} is {{ states(entity) }}
{%- endif %}
{%- endfor %}
{%- endif %}
{%- endfor %}
{%- endfor %}
Answer the user's questions about the world truthfully.
but am receiving this response whenever I try to interact with the LLM within home assistant:
```vbnet # Prompt /imagine prompt: [Your Prompt], [Your Prompt], [Your Prompt], [Your Prompt] --ar 16:9 --v 5 ```
Not really sure what’s going on here but I think the prompt template is messed up (even though I applied the one directly from huggingface)
Hi guys, thanks to the information in this topic I got LocalAI to work nicely with home assistant.
I was already playing around with LocalAI and similar projects and got this to work quite fast and nicely on my desktop, as long as I run the smaller models and keep the prompts short.
To make it all a bit more sustainable I did the following:
- dekstop sleeps after 15 minutes
- I use an almost empty template, so I can generate templates with automations
- reducing the prompt speeds up some LLM’s
- the automation checks if the desktop is on, otherwise Wake-On-Lan
- then call the service “conversation: process” with your generated prompt
- send your llm’s prompt wherever you want (notification, tts media player, etc.).
Sure, this does not replace the conversational assistant, but it does allow you to generate LLM messages with some nice prompt engineering with limited hardware, on your own infrastructure 
.
I’m pretty green to the whole HACS and LLM thing. I managed (i think) to get the LocalAI running, but after installing the hass-openai-custom-conversation I’m a bit lost. Can someone give me a hint?
- Install local-ai
- Setup model
- Install hass-openai-custom-conversation
- Add custom component to your hass installation
- Set first field to any string, set second field to the address of local-ai installation
- Configure hass assist to use custom openai conversation as conversation agent, set options to contain instructions specific to your setup and model name
The two steps with questionmarks above are where I’m getting lost. I’m guessing the last step is adding the assistant through the regular HA Voice assistant interface.
Thanks 
Hi,
I can’t be sure where you get stuck, but I think you have to go to your integrations page and add the just installed custom component 
. You can find it by simply looking for custom OpenAI conversation, clicking that will start the wizard where you can “set the first field to any string” and where you have to point to your localAI installation.
Hope this gets you going! 
Can LocalAI be run as an add-on in Home Assistant? Trying to avoid using docker if possible 
(running HA bare metal on a NUC)
1 Like
hey there @drndos thanks for sharing!
I got LocalAI running now and working great. I’d like to incorporate this in HA but since the writing of this, it seems that ggml models are no longer supported for LocalAl and it says to use a gguf model.
Can you please share a model for LocalAI (pref 13b), that would work with you integration and if possible, the settings you used in the files (ie. yaml, tmpl, etc).
thanks so much!
Hi, could you share if you used the gguf or ggml model (as above) ? thanks
I don’t think that is realistic and in any way usefull.
I’m running a 13B model (vicuna-13b-v1.5.ggmlv3.q3_K_S.bin)
on the following hardware:
- AMD ryzen 5950X
- 64GB RAM
- GTX1080 (somewhat dated)
Using all these resources gives me a response time of 4~30 seconds, depending on the difficulty and length of the prompt.
Also note that the desktop is completely useless when it is prompted, and as long as the model is kept in memory.
Hey,
I’ve played around with localAI a lot before I implemented it in HA, so I got used to setting up prompt templates and installing new models as instructed at localai.io. I would recommend to follow some of the materials over there.
At the moment I’m still using ggmlV3 files, but it shouldn’t be too hard to get gguf working.
I checked it quickly, over here: Easy Model Setup :: LocalAI documentation they show how to do “easy model setup”, and they updated to the new gguf format. They instruct how to setup the following model from thebloke: TheBloke/Luna-AI-Llama2-Uncensored-GGUF · Hugging Face
This should help get a gguf model working, you still need to setup the prompt templates. Good luck!
How did you get the extension working with text-gen-webui?
When I put in the api key and address and click submit it says “unknown”
Took me a while to figure out I had to use Samba extension to install the custom addon on the OS version of HA. This is my first time.
Thanks sir for the information.
Were you able to get this to work with HA entities?
Remember to change default “https://” to “http://” from integration url if you don’t use SSL.
I’ve been running text-generation-webui in Raspberry pi 5. Due to hardware limitations it takes a couple of minutes to process outputs, but I use LLM models + piper only to create different non-critical automated announcements from weather, electricity price etc. so it is not really a matter of time. With peak power consumption under 20W it is totally acceptable.
I used LMStudio instead to run the LLM and the API worked.
I setup remote access with Cloudflare and the HA app on my phone.
Now it wont control my devices and I see it is specificaly programmed to refuse.
What obvious thing am I missing now to fix that?
Has anyone here had any luck finding a way to use whisper and piper for assist pipelines via localAI api? as opposed to wyoming?
1 Like