LocalAI could be used as a fallback for the Assist pipelines: Pipeline chaining or fallback intent
But maybe should be something more generic than LocalAI, so you could use privateGPT or anything else.
Like maybe just use the already implemented Wyoming protocol integration.
What do you think?
So Iāve got this set up using localai now and am no longer getting the error about context length.
Iāve set my vicuna-chat.tmpl to this:
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
USER: {prompt}
ASSISTANT:
And Iāve set the template within home assistant to this:
This smart home is controlled by Home Assistant.
An overview of the areas and the devices in this smart home:
{%- for area in areas() %}
{%- set area_info = namespace(printed=false) %}
{%- for device in area_devices(area) -%}
{%- if not device_attr(device, "disabled_by") and not device_attr(device, "entry_type") and device_attr(device, "name") %}
{%- if not area_info.printed %}
{{ area_name(area) }}:
{%- set area_info.printed = true %}
{%- endif %}
{%- for entity in device_entities(device) %}
{%- if not is_state(entity,'unavailable') and not is_state(entity,'unknown') and not is_hidden_entity(entity) %}
- {{ state_attr(entity, 'friendly_name') }} is {{ states(entity) }}
{%- endif %}
{%- endfor %}
{%- endif %}
{%- endfor %}
{%- endfor %}
Answer the user's questions about the world truthfully.
but am receiving this response whenever I try to interact with the LLM within home assistant:
```vbnet # Prompt /imagine prompt: [Your Prompt], [Your Prompt], [Your Prompt], [Your Prompt] --ar 16:9 --v 5 ```
Not really sure whatās going on here but I think the prompt template is messed up (even though I applied the one directly from huggingface)
Hi guys, thanks to the information in this topic I got LocalAI to work nicely with home assistant.
I was already playing around with LocalAI and similar projects and got this to work quite fast and nicely on my desktop, as long as I run the smaller models and keep the prompts short.
To make it all a bit more sustainable I did the following:
Sure, this does not replace the conversational assistant, but it does allow you to generate LLM messages with some nice prompt engineering with limited hardware, on your own infrastructure 
.
Iām pretty green to the whole HACS and LLM thing. I managed (i think) to get the LocalAI running, but after installing the hass-openai-custom-conversation Iām a bit lost. Can someone give me a hint?
The two steps with questionmarks above are where Iām getting lost. Iām guessing the last step is adding the assistant through the regular HA Voice assistant interface.
Thanks 
Hi,
I canāt be sure where you get stuck, but I think you have to go to your integrations page and add the just installed custom component 
. You can find it by simply looking for custom OpenAI conversation, clicking that will start the wizard where you can āset the first field to any stringā and where you have to point to your localAI installation.
Hope this gets you going! 
Can LocalAI be run as an add-on in Home Assistant? Trying to avoid using docker if possible 
(running HA bare metal on a NUC)
hey there @drndos thanks for sharing!
I got LocalAI running now and working great. Iād like to incorporate this in HA but since the writing of this, it seems that ggml models are no longer supported for LocalAl and it says to use a gguf model.
Can you please share a model for LocalAI (pref 13b), that would work with you integration and if possible, the settings you used in the files (ie. yaml, tmpl, etc).
thanks so much!
Hi, could you share if you used the gguf or ggml model (as above) ? thanks
I donāt think that is realistic and in any way usefull.
Iām running a 13B model (vicuna-13b-v1.5.ggmlv3.q3_K_S.bin)
on the following hardware:
Using all these resources gives me a response time of 4~30 seconds, depending on the difficulty and length of the prompt.
Also note that the desktop is completely useless when it is prompted, and as long as the model is kept in memory.
Hey,
Iāve played around with localAI a lot before I implemented it in HA, so I got used to setting up prompt templates and installing new models as instructed at localai.io. I would recommend to follow some of the materials over there.
At the moment Iām still using ggmlV3 files, but it shouldnāt be too hard to get gguf working.
I checked it quickly, over here: Easy Model Setup :: LocalAI documentation they show how to do āeasy model setupā, and they updated to the new gguf format. They instruct how to setup the following model from thebloke: TheBloke/Luna-AI-Llama2-Uncensored-GGUF Ā· Hugging Face
This should help get a gguf model working, you still need to setup the prompt templates. Good luck!
How did you get the extension working with text-gen-webui?
When I put in the api key and address and click submit it says āunknownā
Took me a while to figure out I had to use Samba extension to install the custom addon on the OS version of HA. This is my first time.
Thanks sir for the information.
Were you able to get this to work with HA entities?
Remember to change default āhttps://ā to āhttp://ā from integration url if you donāt use SSL.
Iāve been running text-generation-webui in Raspberry pi 5. Due to hardware limitations it takes a couple of minutes to process outputs, but I use LLM models + piper only to create different non-critical automated announcements from weather, electricity price etc. so it is not really a matter of time. With peak power consumption under 20W it is totally acceptable.
I used LMStudio instead to run the LLM and the API worked.
I setup remote access with Cloudflare and the HA app on my phone.
Now it wont control my devices and I see it is specificaly programmed to refuse.
What obvious thing am I missing now to fix that?
Has anyone here had any luck finding a way to use whisper and piper for assist pipelines via localAI api? as opposed to wyoming?
I think you mean controlling entites/ changing states?
No, at the moment I donāt use it to receive/send commands to HA, purely trying to interpret numbers
Where do these tmpl and yaml config files live? Next to the model?
I have a pretty reliable and mostly competent LLM-powered conversation agent using Text-generation WebUI with this new custom component. So far it is working very well, although occasionally the model hallucinates and gives totally wrong answers to questions. For controlling exposed entities it works better than any other component that allows LLMs.
Yes I had time to play with home-llm yesterday. It never failed (to turn devices on and off) in my test. Iām realy impressed. I even got it to stop acting so robotic and have a conversation by slight modification of the prompt. Changing the prompt too much causes weird responses, not that I know what im doing.
The model is small, so it can run on HA hardware, but it would be great if someone could train a 7B model so we can run on GPU. Im using Obbabooga on RTX3060 12G, and HA is on VirtualBox.