Introducing HA Configuration Agent - AI-powered Home Assistant configuration assistant with approval workflow

Share your Projects!
, , , ,
41

Just for fun, an example of the perils of using an LLM to write an automation:

Me: write an automation to turn on the bathroom light when the bathroom motion sensor detects movement

trigger:
  - platform: state
    entity_id: binary_sensor.bathroom_motion_sensor
    to: 'on'
actions:
  - action: light.turn_on
    target:
      entity_id: light.bathroom

Me: modify it so that the light is turned off when no movement has been detected for 10 minutes

trigger:
  - platform: state
    entity_id: binary_sensor.bathroom_motion_sensor
    to: 'on'
  - platform: state
    entity_id: binary_sensor.bathroom_motion_sensor
    to: 'off'
    for:
      minutes: 10
actions:
  - action: light.turn_on
    target:
      entity_id: light.bathroom
  - action: light.turn_off
    target:
      entity_id: light.bathroom

Me: that will not work. the light will be turned on and off every time the automation triggers

trigger:
  - platform: state
    entity_id: binary_sensor.bathroom_motion_sensor
    to: 'on'
  - platform: state
    entity_id: binary_sensor.bathroom_motion_sensor
    to: 'off'
    for:
      minutes: 10
condition:
  - condition: state
    entity_id: binary_sensor.bathroom_motion_sensor
    state: 'on'
actions:
  - action: light.turn_on
    target:
      entity_id: light.bathroom
  - action: light.turn_off
    target:
      entity_id: light.bathroom

Me: that will not work. the light will be turned on and off every time motion is detected. when motion has not been detected for 10 minutes nothing will happen

trigger:
  - platform: state
    entity_id: binary_sensor.bathroom_motion_sensor
    to: 'on'
  - platform: state
    entity_id: binary_sensor.bathroom_motion_sensor
    to: 'off'
    for:
      minutes: 10
actions:
  - choose:
      - conditions:
          - condition: state
            entity_id: binary_sensor.bathroom_motion_sensor
            state: 'on'
        sequence:
          - action: light.turn_on
            target:
              entity_id: light.bathroom
      - conditions:
          - condition: state
            entity_id: binary_sensor.bathroom_motion_sensor
            state: 'off'
        sequence:
          - action: light.turn_off
            target:
              entity_id: light.bathroom

Me: add an override to optionally prevent the motion sensor turning the light on and off

trigger:
  - platform: state
    entity_id: binary_sensor.bathroom_motion_sensor
    to: 'on'
  - platform: state
    entity_id: binary_sensor.bathroom_motion_sensor
    to: 'off'
    for:
      minutes: 10
condition:
  - condition: state
    entity_id: input_boolean.motion_sensor_override
    state: 'off'
actions:
  - choose:
      - conditions:
          - condition: state
            entity_id: binary_sensor.bathroom_motion_sensor
            state: 'on'
        sequence:
          - action: light.turn_on
            target:
              entity_id: light.bathroom
      - conditions:
          - condition: state
            entity_id: binary_sensor.bathroom_motion_sensor
            state: 'off'
        sequence:
          - action: light.turn_off
            target:
              entity_id: light.bathroom

I’m impressed that it got the idea of an override. :grin:

Every time I try this experiment (with the same wording) it invents a different cockeyed approach. Back to the prompt.

42

That is due to the set ‘temperature*’ setting of the llm. This adds some randomness to the answers. If it is set to 1 then you get the same answer each time.

  • this setting name may differ with other llm’s or providers.
43

That’s a great idea! I’m going to add temperature as a configuration variable. Thanks! You have to be careful though, some LLMs will error if you pass temperature so I have to make it by default unspecified.

44

Upgrade to 0.1.9 to have Temperature configuration. A temperature of 0 would mean no variation in the output and would produce nearly identical results on multiple requests. Feel free to put 2.0 and get very varied results. I could also put Top K but I think that’s probably overkill :wink:

45

With 0.1.9 I get “AI system not ready. Please configure OPENAI_API_KEY.” In the log “ERROR - Failed to initialize agent system: could not convert string to float: ‘null’”

Restored 0.1.8 and it’s working again.

1 Like
46

Apologies. HA plugin configs doesn’t like optional floats apparently. I just pushed a small config change to fix it without changing the version number. If you install 0.1.9 again you should be fine. Your version of 0.1.9 also would have worked if you specified the temperature.

2 Likes
47

Continuing my experiments… (with gpt-4o-mini)

The aim is to use the Config Agent as an analysis and documentation tool - it is not allowed to change anything.

In the prompt I explained the role of labels and gave it a list of the labels currently in use. I am now able to type “Give me a list of all the automations labelled ‘sms’.” and I get a list of automations, with entity IDs and the first line of the description.

Severe weather alert

    Entity ID: automation.severe_weather_alert
    Description: Sends SMS message when severe weather alert is issued.

Fridge too warm

    Entity ID: automation.fridge_too_warm
    Description: Sends SMS when fridge has been above 5C for the past hour.

Computer running too hot

    Entity ID: automation.computer_running_too_hot
    Description: Sends a text if computer processor temperature is above threshold.

This is excellent - just what I was looking for.

Accuracy is another matter. I have the temperature set to 0, but so far it has always either missed an automation, added an automation, or (in one memorable test) invented a fictitious automation of its own. Name, ID and description were all plausible too - I’m secretly impressed.

On a different note, longer conversations, where I am trying to refine the Config Agent’s response, are quite expensive. Using the API, when you add an extra query to clarify something the whole conversation is re-sent. Not only do you run out of tokens quite quickly, but you are charged multiple times to reach one accurate conclusion. Ideally I would like to see something like you have in automation/script traces, where you can step backwards and forwards through earlier runs and use them to frame a new, better question.

Edit: Tried this with gpt-5-mini. Much more accurate results, but it throws an error if temperature is set to 0.

1 Like
48

Yeah the whole conversation being re-sent is exactly how the whole thing works. Unless we put some limit on the number of messages (and I don’t love that) you gotta resend the whole thing. That’s why Caching is great for Gemini and Claude models because they charge you much less for the resubmitted functionality. OpenAI doesn’t do that.

I’m going to add a context counter to your UI so you can see how much input and output tokens you’re sending. I’ll try to get cached ones differently but I don’t know if that’s even possible.

49

0.1.10 release with token usage tracking (input, output and cached).

50

Error on update:

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/commands.py:317
integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 10:50:03 (2 occurrences)
Last logged: 10:50:26

[140471107282848] Error during service call to update.install: Error updating AI Configuration Agent: Docker build failed for 9b396a0a/amd64-addon-ai-config-agent: (exit code 1). Build output: ERROR: failed to build: invalid tag "9b396a0a/amd64-addon-ai-config-agent:": invalid reference format
1 Like
51

Yes, it’s the way APIs work. The trouble is, iteration is a key part of using an LLM, especially if you’re going to let it create automations and such. You’re never going to get it right first go.

As a reporting tool I think the Config Agent could be really useful, because you can refine the prompt to get the level of analysis you want. As a creative tool it may only be suitable for users with their own local LLM.

52

Can the Config Agent read files in .storage? It claims to be able to, but when i tell it to read a specific file it claims it doesn’t exist:

I looked for the entity notes file (for example /homeassistant/.storage/entity_notes.notes) but I couldn't find it.

What I searched:

    I searched the configuration tree for "entity_notes" and "entity_notes.notes".
    That returned some entity files referring to the Entity Notes integration (entities/update.entity_notes_update.json and entities/switch.entity_notes_pre_release.json), but not the expected storage file containing the notes.

I can see and read it myself.

53

It’s probably because the grep pattern of ‘**’ doesn’t match dotfiles. Let me look in to making it support that. I’ll get back to you.

1 Like
54

Config Agent agrees! :grin:

.storage/entity_notes.notes exists. why would it not be visible to you

Thanks — that helps. The reason I couldn't see .storage/entity_notes.notes is a tooling limitation: the configuration-file search I can run does not return items from Home Assistant's .storage directory (hidden runtime storage). I can read the YAML and entities files you keep in /homeassistant, but not the .storage files via the automatic search I used.
55

Try 0.1.11 and see if it works now if you name the file exactly.
I’ve made it so that any search that starts with “/” is considered a glob pattern and searches on the filenames instead.

1 Like
56

Sorry.

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
[00:35:14] INFO: Starting AI Configuration Agent...
[00:35:14] INFO: OpenAI API: https://api.openai.com/v1
[00:35:14] INFO: Model: gpt-5-mini
[00:35:14] INFO: HA Config: /homeassistant
/app/src/main.py:33: UserWarning: Field name "validate" in "RestoreBackupRequest" shadows an attribute in parent "BaseModel"
  class RestoreBackupRequest(BaseModel):
/app/src/main.py:42: UserWarning: Field name "validate" in "ApprovalRequest" shadows an attribute in parent "BaseModel"
  class ApprovalRequest(BaseModel):
Traceback (most recent call last):
  File "/usr/local/bin/uvicorn", line 7, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1462, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1383, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1246, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 814, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 423, in main
    run(
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 593, in run
    server.run()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 67, in run
    return asyncio_run(self.serve(sockets=sockets), loop_factory=self.config.get_loop_factory())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/_compat.py", line 30, in asyncio_run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 71, in serve
    await self._serve(sockets)
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 78, in _serve
    config.load()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/config.py", line 439, in load
    self.loaded_app = import_from_string(self.app)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 19, in import_from_string
    module = importlib.import_module(module_str)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/app/src/main.py", line 117, in <module>
    app = FastAPI(
          ^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 907, in __init__
    assert self.version, "A version must be provided for OpenAPI, e.g.: '2.1.0'"
           ^^^^^^^^^^^^
AssertionError: A version must be provided for OpenAPI, e.g.: '2.1.0'
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
57

On version 0.2.0 now, with gtp-5-mini and… well, this is impressive:

Me: tell me about the input_boolean “probably in kitchen”

I can’t find an input_boolean named “probably in kitchen” — there is no input_boolean with that name. You probably mean the Bayesian binary sensor called “Probably in kitchen”. Details below.

Entity summary

Entity ID: binary_sensor.probably_in_kitchen
Platform: bayesian (Bayesian binary_sensor)
Original (display) name: Probably in kitchen
Unique ID: bayesian-d92dbc6e-3c04-4ea9-bd48-c58ae1593bb9
Labels: [“probably_in”]
Area / device: none assigned (area_id: null, device_id: null)

How it works (configuration highlights)

Prior probability: 0.13 (initial estimate before observations)
Probability threshold: 0.90 — the binary_sensor state becomes “on” when the calculated probability ≥ 0.90
The sensor exposes an attribute named probability (a float 0…1). The automations/templates in this config also expose a percent sensor:
sensor.kitchen_probability — uses state_attr(“binary_sensor.probably_in_kitchen”,“probability”) * 100

Observations used (each updates the Bayesian probability)

sensor.last_movement → to_state: “Kitchen”
prob_given_true: 0.9
prob_given_false: 0.02
sensor.martin_tag_area → to_state: “Kitchen”
prob_given_true: 0.8
prob_given_false: 0.01
sensor.wallet_tag_area → to_state: “Kitchen”
prob_given_true: 0.8
prob_given_false: 0.01
sensor.fairphone_3_area → to_state: “Kitchen”
prob_given_true: 0.33
prob_given_false: 0.02
sensor.watch_area → to_state: “Kitchen”
prob_given_true: 0.8
prob_given_false: 0.01
input_boolean.away_mode → to_state: “on”
prob_given_true: 0.01
prob_given_false: 0.09
sensor.keys_tag_area → to_state: “Kitchen”
prob_given_true: 0.67
prob_given_false: 0.90

Notes and observations

The binary_sensor computes the probability from these observations and switches to on when probability ≥ 0.90.
The templates.yaml file defines sensor.kitchen_probability (percent) and is the usual way other automations/UI code reads the room probability.
The keys_tag_area observation has prob_given_false: 0.90 — that means the configuration treats the keys being in the kitchen as fairly likely even when the kitchen is not occupied. If that looks wrong I can explain how to adjust those weights.

Would you like any of the following?

A copyable YAML snippet to create an input_boolean named probably_in_kitchen (if you really meant an input_boolean)
Help editing the bayesian observation weights or the threshold
A short automation example that acts when the kitchen probability goes above X%

Edit: The keys observation is correct, by the way. I normally drop them on the kitchen counter when I come in, and they’re there most of the time.

1 Like
58

So I wanted to make sure v 0.2.0 is working ok for you now?
That version is eventually going to become a custom component instead of an add-on (I was able to make it work in both cases).

59

Working well, thank you.

Do you intend to continue maintaining the add-on? If not, will the custom integration differ at all (apart from installation)?

60

Hi there @yinzara

Thank you for creating this tool. I have been excited for specifically this feature since we started to see more AI control in home assistant. Letting an AI create and manage my automations based on best practices is suuuuuuper useful for me.

As my number of automations and scripts grow, I get more and more headaches trying to manage them and create new ones that are consistent with my previous conventions. My knowlege has also grown over time and many of my earliest scripts and automations are a bit naive and it shows.

As I demoed this tool, I ran into a bug where the workflow failed, but I didn’t get an error message. I believe this us due to a hidden timeout waiting for the agent to respond. I created a github issue describing what I experienced. My automations.yaml file is in the thousands of lines, and the agent has to verbatim report the entire automations.yaml file back in it’s tool call. This tool call response can take many minutes depending on the speed of the ai being used. (for me I’m using an agent that returns text at ~10-20 tokens per second).

I forked the code and tried toying around with the propose_config_changes interface and I see some of the difficulties in making this interface friendlier to large file sizes. There’s so many different types of edits that the agent can make by allowing it to just dump the entire file. Once you say, “Don’t dump the entire file, just a subset”, trying to manage individual edits could easily balloon to supporting add/remove operations for each individual type of config change – which is much more complex.

Do you have any thoughts or experiences using this with large automations/scripts files and how it could be improved?

Another observation I made when demoing this tool was the context size of the conversation with the agent. The conversation had the large automations.yaml and scripts.yaml files injected into it, but the agent used the tool asking for the files to be injected into the conversation again. Eventually I ran out of conversation context space (192k tokens in the case of my specific usage).

I think it would be useful to intelligently withhold files which were previously injected into the conversation because the agent already has that file in it’s context.

Anyway, again, thank you for building this! This is amazing, super awesome and I’ve been excited for someone to build this. I’ve even taken a couple stabs at this before (but not as a hacs tool).