# configuration.yaml
environment_variable:
OPENAI_BASE_URL: http://192.168.3.241:9997/v1
Modify URL, restart HA。Key can be freely filled in。
1 Like
Forget the visionary approach and just give me an LLM to write scripts and automations.
Existing LLM’s can already do this.
I have been experimenting with local LLMs controlling HomeAssistant for a while now (I’m the author of the blog post from January), and have noticed that RAG actually makes a massive impact with (relatively) little work. I blogged about it a little while back if you’re interested in the details.
The idea is, it’s unlikely that the user ever wants to take action on their entire house, but rather only some portion of their house (for example a room, a floor, or a specific entity). We can use RAG in its simplest form (retrieve pre-computed embeddings for each area/entity and get the top X results ranked by cosine similarity) and it will drastically cut down on processing times. I have been using this at my house and it made a big impact despite adding even more things to the LLM’s total context, while not causing any quality issues I could perceive.
This performance boost is because prefill cost (the processing required before anything is ever output from the LLM) increases exponentially based on the context, so reducing it has a non-linear effect on performance. This is notably the case without GPUs because as far as my experiments go, prefill is the worst part of CPU inference.
I have also included examples that are dynamically generated for in-context learning (based on the smart home state itself), which I found massively improves function calling accuracy while giving the LLM the ability to adapt to new HomeAssistant APIs that they are not trained for.
With the combination of all of this, I am actually able to use the whole pipeline locally with usable speeds (it’s not fast, but it’s no longer 8 seconds per query, at least if you have GPUs) locally.
5 Likes
Thats interesting! Cheers
I want to integrate a local multimodel LLM into an NVIDIA Jetson AGX to read from video, audio and mics to automatically control HA. I need help getting started with certain things. Anyone here willing to offer me some guidance? I’d very much appreciate it. 
Whatsapp: https://wa.me/34638244348
(Background: I’m a Senior Android Engineer and I’ve been playing a lot with prompting, OpenAI APIs, etc even doing stuff with LLMs for production features at work)
Quite poorly, sure. Chat GPT is usually the most accurate but there are usually way too many issues, kind of defeats the purpose.
Hi there, there was an LLM model comparison of the synthetic home. But I cannot find that repo anymore. Does anybody know about it?
Still don’t quite see what the use case is for AI. 
I can remember some of the early discussions about the value of smart homes - particularly one point: it’s no use being able to turn lights on and off with your mobile phone if it’s easier to flip a wall switch; it’s just showing off.
There’s a case to be made for voice control as part of a multi-pronged approach to making home automation easier to use than wall switches (manual/automated/voice-controlled). I’m just not sure why we need the house to make smart remarks.
Could this be an expectation that has been planted in us by Amazon and Google? I can see that it might be interesting to implement and fun to use, but it’s hardly worth the attention it’s getting. Is it? I would put it roughly on a par with having your own weather sensors in the garden.
1 Like
Agreed, or even better: automated so that a switch is redundant most of the time.
I think it’s more about parsing the input. LLMs are better at “understanding” intent than just the literal meaning and ambiguities or requests.
Are you still working with NVIDIA’s Jetson AI Lab Research Group and are they still helping making progress on all this?
3 Likes
This is my wet dream!
To be able to use the AI of my choice, and to be able to expose my entitites and devices, my files such as configuration, automations, scripts, templates etc, and finally my logs.
Imagine being able having AI suggest improvements based on devices frequently falling out of network, like “include a bridge between these 2 Zigbee devices, since they often loose contact with each other.”
Or “Your Tesla integration have stopped working for the nine thousand time, perhaps get another car?”
Or just to be able to write and troubleshoot automations and scripts based on natural language descriptions. It would be the ultimate hurdle removal for new users.
1 Like
Is someone is working on a “Fabric” AI tooling / AI agent integration for Home Assistant?
fabric (from Daniel Miessler) is an open-source framework for augmenting humans using AI. It provides a modular framework for solving specific problems using a crowdsourced set of AI prompts that can be used anywhere.
Would be awesome to have integration(s) for Fabric’s “Patterns” (currated crownsoured promts) as AI agents / tooling that can use Fabric pattern promts as tools can use from Home Assistant.
fabric intro video:
(Keep in mind that many of these were recorded when Fabric was Python-based, so remember to use the current install instructions)
Also check out this fabric origin story:
lol for this being in this article, it’s making so much sense, and it now being unavailable and taken away with literally no explanation… Great, we got a water cooler!
voice-assistants channel on our Discord server.
It’s still there; the link is just to the wrong place on the server.
A very complex subject, especially with my limited understanding of it. Entirely in layman’s terms my view, or requirement, of HA & AI (& all its component parts) is to be able to machine learn using historic & real-time data from within HA. So my user case is one of combining weather (actual & forecast), 30 minute electricity tariff slots, room temperatures, predicting solar gain (based on combinations of previous sensors), monitoring heat gain & heat loss rates, etc. HA then has an evolving AI agent that can control heat input from our ASHP (not an integrated model as an old ASHP but getting to stage of potentially replacing it).
The reason, manufacturers’ apps for ASHPs, HVACs etc typically don’t go to the next level of predicting heat requirement & calculating the cheapest time to provide that heat. Fairly obvious when to produce hot water based on a simple metric of cost but heating is more nuanced. As I say my poor understanding of how to build this is part of the problem. It feels like a very steep learning curve & tools to help do it, or walk through or even build it, which be of great help & invaluable.
Apologies for my noob request & low level of understanding.