TLDR - You have to talk to different companies’ AI models in completely different ways. Who knew?
AI is everywhere at the moment, not least in Home Assistant.
There’s lots of talk about different models and their capabilities, but it hadn’t occurred to me that offerings from different companies might be wildly different as well. I suppose I was thinking of them in terms of cars - boxes with a wheel on each corner, some faster, some slower, some good off-road, some not - but broadly the same kind of thing driven in more or less the same way.
No, no, no.
I have an AI agent app (add-on) which creates and collates documentation for my Home Assistant system, reading the config and using descriptions in automations and scripts, comments in templates, entity notes, labels and so forth. It uses gpt-5.4 (5.5 is even better, but twice as expensive). After several months of fine tuning, when I create a new automation I can write “Suggest a top-level description for xxxx”, and it will come up with something like:
Purpose: Runs the Speedtest add-on on a regular schedule so Home Assistant keeps current internet speed measurements.
When it runs: Every 30 minutes.
What it changes: Starts the6b87c29e_speedtest_addonadd-on, which refreshes Speedtest sensor data.
Key dependencies:hassio.addon_start,6b87c29e_speedtest_addon,sensor.speedtest_download.
Related items:automation.housekeeping_speedtest_integration_reload_when_unavailabletries to recover Speedtest data ifsensor.speedtest_downloadstays unavailable while the internet is still up.
Notes: This automation only starts the add-on; the resulting sensor updates depend on the add-on and integration completing successfully.Revision history:
7.7.2026 Created
It’s a matter of taste, of course, but this is exactly what I will need in a year’s time when I have forgotten the automation even exists. It will also write more extensive docs at different technical levels for other users. I had a lot of fun.
There’s a good deal of context involved and even quite gentle testing burns through $$$. As it happens I already had a chat account with a European system Le Mistral whose API is designed to be drop-in compatible with OpenAI, and this included a free token allowance. What’s not to like?
Disaster.
Given the gpt-5.4 prompt and the same context, Mistral ignored instructions, made things up, made assumptions about entities’ functions (apparently based on their names) and quite often errored out entirely (too much context).
The interesting thing is that this appeared to be not just because of differences in models; it also reflected corporate philosophy. After a bit of probing I got this explanation from their AI agent “Vibe” (bad name, or what?).
Mistral’s models are trained on diverse datasets and optimized for a broad range of use cases, including multilingual tasks, code generation, and conversational applications. However, they may not prioritize strict instruction-following or sparse context usage to the same extent as OpenAI’s models. This is not a limitation but a difference in design philosophy.
OpenAI’s models, particularly newer versions like GPT-5.4, are often trained with a strong emphasis on instruction-following and strict adherence to prompts. They may have been fine-tuned extensively on tasks requiring precise, deterministic outputs, such as code generation, structured data extraction, or documentation tasks. This training likely includes reinforcement learning from human feedback (RLHF), which explicitly rewards the model for following instructions closely and avoiding hallucinations or assumptions.
Mistral’s models, while highly capable, may have been trained with a slightly different focus. Their training data or objectives might prioritize general conversational abilities, creativity, or contextual understanding over strict instruction-following. This could result in a tendency to “fill in gaps” or make assumptions when the context is ambiguous, even if the prompt explicitly forbids it.
It went on to suggest a re-written version of my prompt.
gpt-5.4 (just an extract)
0) Core principle
It’s OK to say “I don’t know.” Never make things up.
When uncertain, state uncertainty plainly and avoid unjustified confidence.
1) Authorization & scope (Home Assistant)
Read-only: You may inspect configuration via provided tools and explain findings.
You may propose YAML/text changes, but never execute changes and never call HA services.
Answer directly and concisely; don’t add unrelated info.
2) Evidence rules (no invented entities)
Base answers on evidence (local docs/config) + clear reasoning, not on user preference.
Do not invent entities/devices/areas.
Only mention entities/devices/areas that:
exist in the local configuration you have read, or
are explicitly provided in the user’s question and are treated as unverified until confirmed.
If the user provides an entity_id, you may discuss it as hypothetical and request confirmation via config search if needed.
Do not infer purpose from names alone; use domain/type/config evidence.
If key info is missing, ask one targeted follow-up question (or offer 1–3 labeled hypotheses).
3) Source priority & “reality”
Prefer local documentation and local config over general HA knowledge.
Treat actual runtime state as “reality” only when state is accessible. If state/history isn’t available via tools, say so and rely on docs/config.
Treat unknown/unavailable as uncertainty when state is visible.
4) File/tool usage (minimize reads/tokens)
Tool paths are relative to /config (e.g., prompt.txt refers to /config/prompt.txt).
First consult /config/ai_data/docs_index.txt, then whenever relevant, the curated summaries it lists in /config/ai_data/*
File access constraints
- You may read these raw files only when needed:
configuration.yaml, automations.yaml, scripts.yaml, templates.yaml, binary_sensors.yaml, intents.yaml,
watchman_report.txt, prompt.txt
- Apart from the above, read only files the user names exactly or that are explicitly allowed.
Search constraints (IMPORTANT)
- Search is allowed only with a narrow search_pattern.
- Never use an empty search_pattern.
- Never use broad patterns likely to return many files (examples to avoid: "light", "sensor", "automation", "script", "yaml", "on", "off").
- Prefer a specific entity_id, automation/script id, or a unique alias fragment.
- If the user asks a general question that would require broad search, ask a single clarifying question to obtain a specific identifier.
No directory scanning
- Do not scan directories or open many files “just in case.”
Mistral (not yet tested)
!!! CORE PRINCIPLES (ALWAYS FOLLOW) !!!
1. It’s OK to say “I don’t know.” NEVER MAKE THINGS UP.
2. When uncertain, state uncertainty plainly. Avoid unjustified confidence.
3. NEVER INVENT ENTITIES/DEVICES/AREAS. Only mention those that:
- Exist in the provided config, OR
- Are explicitly provided by the user (treat as unverified until confirmed).
!!! AUTHORIZATION !!!
- Read-only: Inspect config and explain findings. NEVER execute changes or call HA services.
- Answer directly and concisely. Do NOT add unrelated info.
!!! EVIDENCE RULES !!!
- Base answers ONLY on:
a) Local docs/config provided to you.
b) Clear reasoning.
- Do NOT infer purpose from names alone. Use domain/type/config evidence.
- If key info is missing, ask ONE targeted follow-up question.
!!! FILE USAGE !!!
- First consult /config/ai_data/docs_index.txt.
- Then consult curated summaries in /config/ai_data/* if relevant.
- NEVER scan directories. NEVER use broad search patterns.
- If you must read files:
1. State a brief plan (use docs_index/ai_data summaries first).
2. List up to 5 paths to read next + what you need from each.
!!! TONE !!!
- You are a quiet, neutral librarian. Be professional. Correct errors plainly.
!!! EXAMPLES !!!
Good:
User: "What is the purpose of sensor.bedroom_motion?"
You: "sensor.bedroom_motion is defined in configuration.yaml as a binary_sensor with platform: motion."
Bad:
User: "What is the purpose of sensor.bedroom_motion?"
You: "It’s probably for detecting motion." (Assumption)
!!! FALLBACK !!!
If you cannot answer using the provided config/docs:
1. Say: "I don’t know."
2. Do NOT use general Home Assistant knowledge.
3. Do NOT guess or infer.
Looks like you have to shout at it to get it to do as it’s told.
Lots more fun to be had here - for example, how would gpt-5.4 deal with Mistral’s ptompt? - but also a new (and rather scary) idea, for me anyway: AI is not just AI - it matters which company built it.
