Home Assistant Cloud Assist and Ollama not working smoothly

Configuration Voice Assistant
1

I’ve been using Home Assistant for over a year. Recently, I started using Assist, but it’s not working smoothly.
For example, when I say “stofzuig de keuken” (vacuum the kitchen), I get an answer saying it doesn’t know what to do or that the device doesn’t exist.
Then I tried using Ollama with local AI models (Llama 3.2:3b, Gemma 3:4b, and Qwen 2.5:3b). When I ask to turn on lights in the living room or vacuum a specific room, it occasionally works, but most of the time it gets frustrating because it doesn’t do what I ask.
I currently use “Hey Google” via Nabu Casa (which usually works very well), but I would prefer to switch to Assist for better privacy.
I have created scripts for vacuuming each room and for controlling the lights. Is using scripts the best approach to work with Assist, or am I missing something?

26300
263001344×2992 376 KB
(an example photo of what I get if I ask something)

2

You’re assuming that HA is setup to do all those things natively without your intervention.

What inwas able to pull from your Pic was you were arguing with the machine in Dutch.

Yes ha recently added vaccum intent support… BUT your vac must support the room Ids like the Roborock integration does to use the native builtin stuff Else that doesn’t work for your bot. (there’s tons of minor nuances but in most cases… This is a variant of the issue)

HA voice is early. It’s good. And if you know how to write ha scripts (as you apparently do) you can make it do whatever you want. BUT don’t assume everything is 100% built for you already. It’s more platform than finished product and there are things you can do to make your tools work better.

Now back to your screenshot.
LLMs will happily tell you whatever to finish the statement. If you’ve done nothing to tell the system about your tools and it doesn’t understand it will use whatever is available to answer. And.. Wel you see the result.

Id bet this is the result of poor grounding combined with your model choice. Based on what I think it responded it doesn’t understand how to use your tool. (description of tool isn’t sufficient for the llm to know it’s thee right one.) use the debug tool in settings Voice Assistants to see what it thought you said and how it worked out a response.

Im mid write on a series of articles about what it takes to make Ai work in HA. (since you’re using ollama you’re trying to do local inference…)

Maybe some of this can help. I should get to the context chapter today. This is also the ‘grandma’s cardboard box’ problem I list in Fridays Party. But the articles above should be easier to read.

3

It is also with the lighting and the ac, tv the same. If i use the Assist from home assistant it self, it is even worse than with an local ai. But after tomorrow, my Vacation wil start and than i have plenty time to check everything.

4

That means its almost 100% a context problem

Speech in general ND LLM in HA in particular is not plug and play.

5

With the help from Google gemini it know works with automatisations instead of scripts. He made this: alias: “Spraak: Stofzuiger Ruimtes”
description: “Start de Dreame voor specifieke ruimtes via Assist sentence triggers”
trigger:

Hal

  • trigger: conversation
    command:
    • “stofzuig de hal”
    • “maak de hal schoon”
    • “hal stofzuigen”
    • “reinig de hal”
      id: hal

Keuken

  • trigger: conversation
    command:
    • “stofzuig de keuken”
    • “maak de keuken schoon”
    • “keuken stofzuigen”
    • “reinig de keuken”
      id: keuken

Slaapkamer

  • trigger: conversation
    command:
    • “stofzuig de slaapkamer”
    • “maak de slaapkamer schoon”
    • “slaapkamer stofzuigen”
    • “reinig de slaapkamer”
      id: slaapkamer

Woonkamer

  • trigger: conversation
    command:
    • “stofzuig de woonkamer”
    • “maak de woonkamer schoon”
    • “woonkamer stofzuigen”
    • “reinig de woonkamer”
      id: woonkamer

Hobbykamer

  • trigger: conversation
    command:
    • “stofzuig de hobbykamer”
    • “maak de hobbykamer schoon”
    • “hobbykamer stofzuigen”
    • “reinig de hobbykamer”
      id: hobbykamer

condition:
action:

  • choose:

    • conditions:

      • condition: trigger
        id: hal
        sequence:
      • action: dreame_vacuum.vacuum_clean_segment
        target:
        entity_id: vacuum.hobbykamer_anja_anja
        data:
        suction_level: 2
        segments: 4

    • conditions:

      • condition: trigger
        id: keuken
        sequence:
      • action: dreame_vacuum.vacuum_clean_segment
        target:
        entity_id: vacuum.hobbykamer_anja_anja
        data:
        suction_level: 2
        segments: 2

    • conditions:

      • condition: trigger
        id: slaapkamer
        sequence:
      • action: dreame_vacuum.vacuum_clean_segment
        target:
        entity_id: vacuum.hobbykamer_anja_anja
        data:
        suction_level: 2
        segments: 5

    • conditions:

      • condition: trigger
        id: woonkamer
        sequence:
      • action: dreame_vacuum.vacuum_clean_segment
        target:
        entity_id: vacuum.hobbykamer_anja_anja
        data:
        suction_level: 2
        segments: 6

    • conditions:

      • condition: trigger
        id: hobbykamer
        sequence:
      • action: dreame_vacuum.vacuum_clean_segment
        target:
        entity_id: vacuum.hobbykamer_anja_anja
        data:
        suction_level: 2
        segments: 1
        mode: single

And know i use Home Assistant Assist and the Google stt/tts