Klar! NLU - Assist alternative with multi-intent, follow-ups, 60+ languages

Share your Projects! Custom Integrations
,
1

Hi everyone,

We built Klar NLU, a local conversation engine for Home Assistant Assist. House commands stay on your machine - no cloud, no model weights for the NLU itself. An optional LLM can chat or polish the spoken reply; it does not turn lights on similiar to Assist. We just were annoyed a little bit / why should an LLM get the full context? why can´t just the NLU turn on two lights etc.

It’s free, open source (MIT), and we don’t sell a subscription.

What it does

  • Turns spoken household sentences into HA intents: lights, climate, covers, locks, fans, media (including Music Assistant), timers, lists, scenes
  • Follow-ups like “turn it off” keep the last target
  • Clarifies when a device is ambiguous
  • Personality in HA (butler, Jarvis, pirate, …) for the voice just for fun :smiley:
  • Optional LLM fallback for chit-chat when it isn’t a home command / less context faster for local AI use

In our Benchmark, Klar hit 100% vs ~31% for built-in Assist templates. More details in our docs

ngl. a lot of AI was used but that should be clear for everybody looking into the NLU topic.

Status

Still a few bugs sometimes

Get it

Expose the devices you want to talk to, try five phrases, and if something misses use Mapping in the UI.

Have fun talking to the house - we’d love to hear how it goes.

2 Likes
2

Hi, can you be clearer in your documentation about what are the roles of the HACS integration and the app/addon? Do we need both? Does one improve the other? I have no idea whether do I install one or the other or both because there is no clear instructions in the installation documentation.

3

Thanks for calling this out.. that gap was on us. Klar is still a very new project, so some of the install docs were thinner than they should have been. We just rewrote that section.

Short version: Klar is two pieces. They do different jobs. Installing both does not make parsing more accurate.

  • HACS integration conversation agent for Assist (syncs rooms/devices, runs intents). You always need this if Assist should use Klar.
  • App (add-on) runs the NLU engine in its own container and serves Mapping / Lab (sidebar Klar NLU).
  • Bundled engine the same integration can download the GitHub binary and start the engine inside Core if you have no App. (example HAOS)

not yet tried to install the Container Image on a separate Host etc. But this should also work you then need to configure some sort of token. Recommended from our Side is the App Addon.

What to install

  • Home Assistant OS: both. HACS for Assist, the App for the engine and Mapping/Lab. At setup pick Use the Klar NLU App or Docker.
  • No Supervisor (Container / Core): HACS only → Start the bundled engine (HACS only). Assist works; Mapping/Lab are not in the sidebar.
  • App without HACS: the UI can open, Assist will not use Klar.

Do not run the App and the bundled engine at the same time.

User docs are something we will keep improving. Feel free to contribute - issues and small docs PRs are very welcome.

4

@lbentivegni I have klar installed and it appears to be working as the conversation agent..it turns things on and off however I can find no way to get it to tell me my upcoming calendar events. Something the LLM CAN do and if klar can’t I would much rather it pass it on rather then telling me to tell it what light to turn off. Can you implement an “Ask the llm” feature so if klar can’t handle it we could tell it “Ask the llm ”.

Every time I ask it anything calendar related like “What’s on my calendar” or “what are my upcoming events” I get “Create Calendar Event is off” as my reply. I added aliases of “my calendar” and “my upcoming events” to the calendar I have exposed. You talk about mapping but I don’t see anything about mapping? Also I don’t speak or read german and that’s all the app will show. I even selected english in the settings and clicked save. I also recall reading it tied in with music assistant but I tried “resume music”, “play depeche mode” and a few other phrases none of which it understood.

5

I installed the app (HAOS) and the integration as well. Now I’m having the following problem: the integration can’t connect to the app because the app is assigned a hostname with a prefix (xyz-klar-nlu). I can change the hostname in the integration, but it’s always overwritten with “klar-nlu”. so no connection possible.

6

@famewolf thanks for the report. All of that should be fixed in the latest release (2026.8.53). Please update both the HACS integration and the App so they match.

Calendar list / create / delete / move is in now, so “What’s on my calendar” and “what are my upcoming events” should work. The “Create Calendar Event is off” reply was a bug.

Just as a note - To send a phrase to the LLM yourself, use a policy in the operator UI:

  1. Settings → Devices & services → Klar NLU → Configure → set Conversation agent for chit-chat to your LLM. Keep the Assist pipeline conversation engine on Klar NLU.
  2. Open the App sidebar Klar NLU (not the Lovelace Klar card) → Rules → Policies.
  3. Click Rule.
  4. When → Phrase: the words you want to say, for example Ask the LLM (4–200 characters).
  5. Then: LLM prompt.
  6. In the box, write the system prompt for the fallback agent (required). Example: Answer this in the user’s language. Do not control devices.
  7. Save.

First matching rule wins. After that, saying “Ask the LLM …” (or whatever phrase you set) goes to that agent with your prompt. You can add more rules the same way, e.g. Phrase what’s on my calendar → LLM prompt. I know i need to document more but there are still bugs which are currently a little bit more important.

Mapping is House → Mapping in that same App. The operator UI follows the Home Assistant language, so English should stick after the update. Music Assistant phrases like “play Depeche Mode” and “resume music” should parse as well.

If anything still misses after the update, send the exact phrase.

7

@mag2000 that hostname overwrite was a bug. The latest release (2026.8.53) keeps Supervisor-prefixed hosts like xyz-klar-nlu instead of rewriting them to klar-nlu.

Update the HACS integration (and the App to the same version). The URL you set should stay.

8

Thanks for the follow up but I’m a little confused because I’m currently on 2026.8.55 which unless you had a typo is newer than that. Additionally, it still shows up all in German, so if you could say specifically how to get it to change the EnglishThat would be very helpful. I’m on HACS and I uninstalled the app when it was all in German, so right now I’m only running the integration. If I add the app back, is it going to mess up anything?

9

@framewolf You’re right.. sorry, I saw that right after I posted. 2026.8.55 is newer. Not a typo. We had another language-config bug after that, and 55 still has it. The fix is 2026.8.61. Update HACS to that and restart.

There are actually three language settings (I had to check them myself):

  1. Integration UI — should now follow your Home Assistant language.
  2. Pipeline language — I made all languages visible to cut config effort. That was a mistake.
  3. Operator UI — under Settings, if you use the App. You don’t have to. It’s for troubleshooting and a bit more customization.

Uninstalling is fine. Almost everything comes from HA. The only thing you might lose is custom policies you set in the Operator UI.

I’m also looking at another small regression (volume vs light scoring very close) and some calendar create bugs. List works.

I’ll switch my own setup to staging once main feels somewhat usable.

Thanks again for testing.

1 Like
10

Got it going and set up the Good Morning and Good Night routines to trigger the appropriate script. Seemed to work well. App shows in english now. Is there something somewhere or can I query a list of comands that Klar will respond to? I’ll admit to being a relative newbie to Home Assistant so just trying to get a feel for all it can do.

11

Klar’s job is: talk to it like a person. You should not have to memorize a command grammar. Throw a normal sentence at it. If it is sure, it runs the house. If it is not sure, or the action is risky, it asks first instead of guessing.

It is a local, rule-based voice engine for Home Assistant, an alternative to the normal Assist behaviour. It is not a cloud assistant and not an LLM with tools. Klar turns speech into device actions. An optional LLM in Home Assistant can chat or polish the spoken reply. It does not get to flip switches.

How you could talk to it

Name a room, a device, or both. Follow-ups work on the same conversation: “turn it off”, “a bit dimmer”, “and the kitchen too”.

I would rate German and English as the strongest packs. They are the ones we live-test hardest, also thanks to your feedbacks. Other Assist locales are compiled too and take the same path. Because Klar is built around a home graph, aliases, multi-intent, and follow-ups (not a giant list of sentence templates), those other languages should still feel a lot better than stock Assist. Home Assistant’s approach is different: custom sentence triggers can get you far there as well, but you have to write and maintain a lot more of that yourself.

Lights

You say What should happen
Turn on the living room lights / Licht im Wohnzimmer an Lights on in that room
Turn off the kitchen and the hallway Two rooms, one sentence
Living room and kitchen lights on / Wohnzimmer und Küche an Same, multi-target
Living room lights at 100% / Wohnzimmerlicht auf 100% Absolute brightness
Set the living room lights to 30 percent Absolute brightness
A bit dimmer / ein bisschen dunkler Relative step down (after a living-room target)
Brighter / Wohnzimmer hell / volle Helligkeit Relative up, or “full” as 100%
Medium / mittelhell Named level (about 50%)
Make it red / wohnzimmer rot Color, if the light can do it
und die Küche auch / and the kitchen too Same action, other room

Climate

You say What should happen
How warm is it in the living room? / Wie warm ist es im Wohnzimmer? Read temperature
Set the living room heat to 21 / Heizung Wohnzimmer auf 21 Setpoint
Turn on the living room heat, then “to 21” Follow-up fills the number

Covers, garage, fans

You say What should happen
Open the living room blinds / Rollo im Wohnzimmer auf Open
Close the living room blinds / Rollo im Wohnzimmer zu Close. Confirm first (risky)
Set the garage door to 40% / Garagentor auf 40 Prozent Position
Set the living room blinds to 75 Position

Locks and other critical actions

You say What should happen
Unlock the front door / Wohnungstür aufschließen Guard: “Really?” Then yes / ja
Lock the front door / Wohnungstür abschließen Same confirm
nein Cancels. Nothing moves

Unlocking, locking, and closing covers are treated as risky. The plan stays in the session until you confirm. A vague “open the door” should not silently fire.

Music and TV

You say What should happen
Play Queen / Spiel Queen Music Assistant search-and-play
Play Queen in the kitchen / Spiel Queen in der Küche That room’s player
Play music in the kitchen / Musik in der Küche Resume / start in that room
Play the album Rumours by Fleetwood Mac in the living room Search with type hints
What’s playing? / Was läuft? Now playing on the last player
Make it quieter / etwas leiser Relative volume
Set the volume in the living room to 30 Absolute volume
Next track / pause / nächster Titel / Pause Transport on the last player
Turn on the living room TV / Wohnzimmer Fernseher an That TV, not the lights

Klar does not invent artists or playlists. Music Assistant has to resolve the search. Alexa as a speaker is a player/integration problem, not a Klar parse miss. If Alexa does not accept Music Assistant playback, Klar can still understand “play Queen” and you can still hear nothing on that box.

Calendar, clock, weather

You say What should happen
What time is it? / Wie spät ist es? Clock
What’s the weather? / Wie wird das Wetter? Weather
What’s on the calendar? / Was steht im Kalender? List events
What do I have tomorrow? / Was steht morgen an? Tomorrow
Add dentist tomorrow at 3 to my calendar / Trage den Termin Zahnarzt morgen um 15 Uhr in den Kalender ein Create event
Delete dentist / move dentist to Friday After a list, follow-ups stay in HA

Known wobble: event titles sometimes pick up leftover words. “Add the event Name tomorrow at 3 to my calendar” is safer than a very loose sentence. Guards / clarify should kick in when the title looks wrong. We are still tightening this.

Timers, lists, scenes, queries

You say What should happen
Start a timer for 10 minutes / Timer 10 Minuten Timer
Cancel the timer / Timer abbrechen Cancel
Add milk to the list / Milch auf die Liste Shopping / todo
Milk done Check it off
Activate All off / Aktiviere Alles aus Scene
Are the living room lights on? / Ist das Licht im Wohnzimmer an? State query

Multi-intent (one breath, several actions)

  • Turn off the living room lights and set heat to 21
    Mach das Licht im Wohnzimmer aus und Heizung auf 21
  • Turn on the living room and kitchen lights
    Wohnzimmer und Küche an
  • Turn off the living room lights and lock the front door
    (lights run; lock still needs confirm)

When it is unsure: guards

Klar is allowed to not act.

  • Clarify: two lamps in one room, or a nickname that matches more than one device. “Which one?”
  • Confirm: locks, unlocks, closing covers, and other risky steps. Answer yes / ja on the same conversation.
  • Reject / hand off: “What’s the capital of France?” is not a home command. “Tell me a joke” / Erzähl einen Witz can go to an optional chat LLM if you configured one. The LLM still does not run devices. (we might add a feature flag for it)

If a device is missing, it is usually not exposed (Settings → Voice assistants → Expose), not a language bug. Nicknames live in Klar NLU → House → Mapping (or HA entity aliases).

What already works well

  • Everyday lighting, heat, covers, fans, scenes, timers, lists
  • Several devices or rooms in one sentence
  • Absolute and relative dim / volume (“100%” vs “quieter” / “less bright”)
  • Follow-ups in one conversation (“and the kitchen too”, “to 21”, “pause”)
  • Calendar list / create / delete / move (titles still sometimes sloppy)
  • Critical actions stop for a spoken confirm
  • Music Assistant search-and-play (“play X”, albums, radio, queue) on a player that can actually search
  • Optional personalities and LLM reply polish. Device control stays with Klar
  • On-device, no cloud in the engine

Limitations (honest)

  • It still mishears or mis-binds sometimes. The design is: ask, don’t guess. If a guard does not fire, that is a bug we want.
  • Speech-to-text is not Klar. If Assist heard the wrong words, the parse never sees what you meant.
  • Calendar create can glue extra words onto the title. Prefer “add the event Name tomorrow at 3”.
  • Music needs a searchable player (Music Assistant). Alexa Echo playback is outside Klar. The sentence can be correct and the speaker still silent.
  • No general knowledge. It will not cook, plan a trip, or invent a playlist that is not in the library.
  • Only exposed entities. Hidden sensors and infra are invisible on purpose.
  • A room with several generic “lights” gets a question, not a random lamp.
  • Generated language packs are thinner than German/English. Same engine, less live-hardened phrasing.
  • There is no per-home learning yet. Your nicknames help (Mapping). Your speaking style is not trained into a private model.

How we improve it today

We do not fine-tune a neural net on your house.

We keep datasets: real household sentences. Next to them sit oracles: the expected detection (intent + slots) and the expected action (which light goes to 30%, which door must confirm). A change is only “good” if those oracles still pass.

That is why a new phrasing (“and the kitchen too”, “a bit dimmer”) lands as a test case first, then as engine behavior. Motivation is the same pattern you want as a user: say it naturally, get a precise, checkable action.

Where this is going

The next step is a more generic learning mode:

  1. Point Klar at a house (your rooms, your names, your habits).
  2. Optionally configure an LLM as a teacher, not as the thing that presses buttons.
  3. Let it propose policies for that home: aliases, “when I say X I mean Y”, safer defaults.
  4. Keep Klar as the executor. The model may suggest a rule. It does not unlock the door on its own.

Until that exists, Mapping + oracles are the way a home gets sharper. Speak normally, let the guards catch the misses, and treat leftover calendar words or a silent Alexa as known edges, not as “you used the wrong command.”

Hope this helps and have fun. And Thanks again for the Feedback

1 Like
12

Thank you SO MUCH for that detailed reply which answered so many of my questions before I even asked them. I also want to take a moment to just say thank you for providing KLAR to us and making it open source. I look forward to putting it through it’s paces. I’m gonna have to be careful though as I have automations and blueprints which handle checking/adding events to calendar, reminders and a few other things. I’m going to have to disable them while I try the new options. For music in music assistant I did try “play infinite library playlist in living room” intending for it to play on the stereo in there. It replied “done” but no music ever started up. I probably messed up somewhere just reporting it for feedback. Also I saw an option named “nlu_rag”. I know with the llm rag is used to reduce the amount of tokens used. Is this somethign we want on?

I have several devices named “Coffee Machine Power”, “Instant Pot Power” etc. If I tell it to “turn on the instant pot” will it catch that or do I need to say power? I plan to test this in the morning (well later morning as it’s 1:30am) and have something cooking overnight in the instant pot and in morning will make coffee so it will get tested then.

Also for adding events to calendar does it prompt for information or do I have to put it all on the line ie “add event to calendar, appointment with doctor tomorrow at 2 pm”. I’ll test that as soon as I get the existing automations, scripts etc disabled. Feel free to ignore all this rambling. Just excited about all the potential and I realize I’m asking way to much.

13

Some things are still in german as you’ll see in the screenshot.

image
image2454×1137 117 KB

14

My understanding is klar handles what it can handle but when it gets a question it doesn’t understand it passes it to the LLM if you’ve configured one. If this is accurate, then it’s not happening. I’ve asked various questions the LLM could answer that klar could not and it just returns various errors. If I’m misunderstanding please let me know.

What’s the weather and What’s coming up are two off top of my head.

15

@famewolf
The UI language issue you reported earlier is fixed.

Your read is close, but Klar does not send every leftover question to the LLM. There are two separate LLM features, and both use the same chit-chat agent.

Set Settings → Devices & services → Klar NLU → Configure → Conversation agent for chit-chat. On that agent (OpenAI, Ollama, Gemma, …), turn Assist tools / Control Home Assistant off. If tools stay on, Klar skips the LLM. That is the safest setup, especially with a small local model.

Fallback is for smalltalk (jokes, stories, greetings) and for speech Klar did not classify as a home command. That is the only path where the model answers on its own.

Refine is off by default. Turn on Let the LLM refine NLU replies. After Klar parses and Home Assistant runs the intent, the model only rewrites the already-spoken NLU line into the selected personality, one or two sentences. It does not take over the command, invent a forecast, or answer a question Klar already handled. If refine fails, the short Klar reply stays.

Weather and calendar are home commands, so they stay with Klar. They do not go to fallback. Refine only restyles the result after Klar has a real answer.

What’s the weather looks for a weather.* entity exposed to Assist (Met.no, OpenWeather, and similar). If that entity is there, Klar reads it. If it is not, English replies “I do not see a weather entity.” The LLM is not asked.

What’s coming up is not in the calendar lexicon, so it is leftover speech and should hit fallback once the agent above is set. For the calendar itself, use “What’s upcoming?”, “What’s on the calendar?”, or “What’s on the calendar tomorrow?” Those need a calendar entity exposed to Assist. Without one you get “No calendar is available.”

So: expose a weather entity for weather, use a calendar phrase plus an exposed calendar for the agenda, use fallback only for things Klar did not treat as a house command, and use refine only to change how a successful Klar reply sounds.

16

@lbentivegni Thank you. Something had reset my connection to home assistant’s conversation agent instead of klar which explains why alot wasn’t working that should have. I do note when I ask “What’s on my calendar tomorrow” I get a weather report and not a calendar report. I tried it in “Lab” under the klar app but not sure this is helpful:

{
  "schema_version": "2.0",
  "text": "What's on my calendar tomorrow?",
  "conversation_id": "432355f7-ded1-4738-bafc-4278f308191b",
  "decision": {
    "type": "execute"
  },
  "speech": "",
  "confidence": 0.9407500000000001,
  "margin": 1,
  "selected_candidate_id": "plan-000",
  "candidates": [
    {
      "id": "plan-000",
      "plan": {
        "confidence": 0.9407500000000001,
        "margin": 1,
        "evidence": [
          {
            "kind": "action",
            "source": "lexicon_exact_question",
            "value": "get_state",
            "score": 0.96,
            "exact": true
          },
          {
            "kind": "binding",
            "source": "calendar",
            "value": "bound",
            "score": 0.85,
            "exact": false
          }
        ],
        "steps": [
          {
            "index": 0,
            "intent": {
              "name": "KlarGetCalendarEvents",
              "slots": [
                {
                  "name": "domain",
                  "value": "calendar"
                },
                {
                  "name": "day",
                  "value": "tomorrow"
                }
              ]
            },
            "confidence": 0.9407500000000001,
            "evidence": [
              {
                "kind": "action",
                "source": "lexicon_exact_question",
                "value": "get_state",
                "score": 0.96,
                "exact": true
              },
              {
                "kind": "binding",
                "source": "calendar",
                "value": "bound",
                "score": 0.85,
                "exact": false
              }
            ]
          }
        ]
      },
      "score": 0.9407500000000001,
      "margin": 1,
      "policy": "calendar",
      "precedence": 1,
      "evidence": [
        {
          "kind": "action",
          "source": "lexicon_exact_question",
          "value": "get_state",
          "score": 0.96,
          "exact": true
        },
        {
          "kind": "binding",
          "source": "calendar",
          "value": "bound",
          "score": 0.85,
          "exact": false
        }
      ]
    }
  ],
  "plan": {
    "confidence": 0.9407500000000001,
    "margin": 1,
    "evidence": [
      {
        "kind": "action",
        "source": "lexicon_exact_question",
        "value": "get_state",
        "score": 0.96,
        "exact": true
      },
      {
        "kind": "binding",
        "source": "calendar",
        "value": "bound",
        "score": 0.85,
        "exact": false
      }
    ],
    "steps": [
      {
        "index": 0,
        "intent": {
          "name": "KlarGetCalendarEvents",
          "slots": [
            {
              "name": "domain",
              "value": "calendar"
            },
            {
              "name": "day",
              "value": "tomorrow"
            }
          ]
        },
        "confidence": 0.9407500000000001,
        "evidence": [
          {
            "kind": "action",
            "source": "lexicon_exact_question",
            "value": "get_state",
            "score": 0.96,
            "exact": true
          },
          {
            "kind": "binding",
            "source": "calendar",
            "value": "bound",
            "score": 0.85,
            "exact": false
          }
        ]
      }
    ]
  },
  "evidence": [
    {
      "kind": "action",
      "source": "lexicon_exact_question",
      "value": "get_state",
      "score": 0.96,
      "exact": true
    },
    {
      "kind": "binding",
      "source": "calendar",
      "value": "bound",
      "score": 0.85,
      "exact": false
    },
    {
      "kind": "preferred_area",
      "source": "satellite",
      "value": "bedroom",
      "score": 1,
      "exact": true
    }
  ],
  "trace": {
    "stages": [
      {
        "stage": "normalize",
        "duration_us": 30,
        "detail": "4 normalized tokens"
      },
      {
        "stage": "features",
        "duration_us": 7,
        "detail": "1 clauses"
      },
      {
        "stage": "action_candidates",
        "duration_us": 2,
        "detail": "1 action hypotheses"
      },
      {
        "stage": "target_resolution",
        "duration_us": 22968,
        "detail": "1 scored targets"
      },
      {
        "stage": "binding",
        "duration_us": 0,
        "detail": "1 complete candidate plans"
      },
      {
        "stage": "ranking",
        "duration_us": 1455,
        "detail": "1 ranked candidates"
      },
      {
        "stage": "safety_decision",
        "duration_us": 3,
        "detail": "execute"
      },
      {
        "stage": "planning",
        "duration_us": 0,
        "detail": "1 executable plan steps"
      }
    ],
    "discarded": [],
    "tokens": [
      "what",
      "on",
      "calendar",
      "tomorrow"
    ],
    "normalized": "what on calendar tomorrow"
  },
  "briefing": false,
  "policy_trace": {
    "compiled_risky": false
  }
}

Additional followup: “What’s the temperature outside?” results in "Thermostat is 76 degrees C. Problem 1. I asked for outside. Problem 2. It’s not 76 degrees C it’s F (for me in USA).