Building an 'Abilities Concierge' in Home Assistant

Share your Projects!
1

Designing a Room‑Aware, Intent‑Driven Smart Home in Home Assistant

An architectural walkthrough of the Abilities Concierge

Context
This post documents the architecture behind a Home Assistant system designed to reduce friction for both owners and guests.
A separate post covers the hospitality and UX motivations; Homes That Behave Well: Teaching the House to Speak Like a Guest this one focuses on how the system is built—the objects, patterns, and data flow that make it work.

The User Experience I Set Out to Create (Briefly)

The target experience was intentionally simple:

  • A person should be able to say “lamps on”, not “turn on the table lamp and floor lamp in the den”.
  • A guest should be able to say “open the shades”, without knowing whether they’re in the Den, Bedroom, or Dining Room.
  • If someone is confused, “help” should be a valid command.
  • The system should explain why something didn’t happen when asked.
  • Nothing should learn silently or unpredictably.
  • Every room has an associated Sonos Speaker setup - all (most) Voice Assistant responses are routed through the Sonos Speakers for that room.

From a technical perspective, that implies:

  • No device names in voice commands
  • No static per‑room configuration lists
  • No schedule‑only automation
  • Explicit separation between discovery, action, and learning

Everything below follows from that.

My Home Assistant Ecosystem

I should start with all of the elements of my Home Assistant Ecosystem that Im’ using for this Ability Concierge.

Voice Assistant(s)

  • Home Assistant Voice
  • Future Proof Homes Satellite
  • Amazon Echo devices (although the integration isn’t great)
  • Almost all rooms have a Voice Assistant so we don’t have to shout across the house
  • Each Voice Assistant is tied to the room it services

Full Home Sound

  • Sonos Amps (for the in-ceiling speaker rooms)
  • Sonos S1’s for the remaining rooms
  • Every room in the house has a set of Sonos Speakers tied to that room
  • I use the standard Sonos integration

Lighting

  • Wired in lighting are controlled by Insteon Dimmers & Switches
  • Most lamps are plugged into wired in Insteon Smart plugs
  • If there isn’t a wired smart plug nearby, then I use Insteon lamp modules
  • Not every room has both Lights and Lamps - so this Concierge figures it out
  • I use the standard Insteon integration

Shades/Blinds

  • Hunter Douglas PowerView shades
  • Not every room has shades - so this Concierge figures it out
  • I use the standard Hunter Douglas Integration

Music Assistant

  • This is used for the ‘Announce’ ability, but if it becomes unavailable, the Concierge has a fallback TTS function it uses instead

My Home Logical Setup

My Home Assistant installation is very logically structured.

Floors, Rooms and Areas

The house is 2 floors with an outside space

  • Each room has a name: Den, Tom’s Office, Kitchen, Living Room, Primary Bedroom, etc. It is this taxonomy that the Concierge uses.

Device Naming Conventions & Tags

Every device has a unique name to specifically identify it in Home Assistant. In most cases it is a combination of Room, Location and Device

  • Den Corner Lamp
  • Den Sofa Lamp
  • Living Room Table Lamp
  • Guest Bedroom Corner Lamp
  • Primary Bedroom Shade
  • Den Shade - in the case where there are multiple in a room, I number them - Den Shade 1, Den Shade 2, etc. Usually working my way around a room - then I have a group that use called ‘Den Shades’ that groups them giving more flexibility

I use the tags to identify the type of device - in very broad categories

  • Lamps (these are all of the plug-in lighting devices)
  • Lights (these are all of the overhead recessed lights, art lights, etc.)
  • TV (I used these on my AppleTV devices to tell them apart from audio media players)
  • Shades
  • Voice Assistants

All of the elements of the Abilities Concierge

This is everything I created to make up the Abilities Concierge (I have created a tag to identify all of these elements so I know that they belong to a much larger purpose)

Helpers

  • Learned Brightness Helpers - There is one helper for each lamp and light the naming convention is {Lamp Name} - Learned Brightness
  • Each Brightness Helper is a number 1 - 100 and stores the last used brightness of the lamp
  • This is used to set default levels based on the last used levels when someone just says ‘lamps on’.
  • Speaker Profile - There is one helper for each room - the naming convention is the Room Name (as it appears in the URL for each room) and hold JSON 3 values (as text) Chat Volume, Music Volume, duck Volume
  • The JSON holds the last values used - and if there are none, there is a default set that is used the first time
  • These are used to set the proper volumes for the Sonos speaker and the Announcements/TTS - and if the speaker was being used to play music, it ducks the music and returns it back to the original level once complete
  • There are two additional date/time helpers for Brightness Last Update and Speaker Profile Last Update - so that I can monitor that the silent processes are running and the last time the values were updated - I’m less interested in knowing each helper update - just that the helpers are being updated.

Scripts

These scripts are intentionally small, single‑purpose building blocks. None of them are voice‑triggered directly. They are designed to be reusable by multiple automations now and in the future.

Learn Lighting – Usual (Room)

Teaches the system how a room is normally lit.

  • Watches for manual lamp brightness changes that remain stable for ~30 seconds
  • Converts the lamp’s brightness (0–255) into a human‑meaningful percentage
  • Stores that value in a per‑lamp helper (input_number.<lamp>_learned_brightness)
  • Runs silently — no announcements, no notifications
  • This happens naturally as people live in the house; no explicit “training mode” is required
  • Forms the basis for later commands like “lamps on” restoring the room to its usual lighting

Learn Music Volume – Room

Establishes what “normal” music volume means for a room.

  • Identifies the active Sonos speaker in the room
  • Reads the current volume level
  • Updates only the music value inside the room’s speaker profile helper
  • Preserves existing chat (voice) and duck (temporary reduction) values
  • Uses safe parsing to avoid corrupting stored JSON
  • Silent and conservative — it never guesses or overwrites unrelated settings

Resolve Speaker Profile

Ensures every room has a valid, usable audio profile.

  • Checks whether input_text.<room>_speaker_profile exists and is well‑formed
  • Seeds defaults if the helper is missing, empty, or incomplete
  • Clamps values to safe ranges
  • This script allows the rest of the system to assume audio intent always exists without defensive checks everywhere

Room & Audio Resolver (Keystone)

The glue that connects room context to audio behavior.

  • Discovers Sonos speakers in the resolved room
  • Selects the best speaker (prefer one already playing)
  • Ensures a valid speaker profile exists (via Resolve Speaker Profile)
  • Returns a consistent set of values used by voice output scripts
  • Almost every voice‑related behavior depends on this script indirectly

Room Abilities – Speak (Sonos)

Explains what the room can do — dynamically and honestly.

  • Discovers, at runtime:
    • lamps vs ceiling lights
    • shade availability (and whether scenes exist)
    • music capability (Sonos)
    • TV capability (Apple TV only)
  • Generates room‑specific spoken explanations
  • Produces examples that only reference things that actually exist in that room
  • Avoids repetition and avoids mentioning unavailable features
  • Ends with a soft prompt like “You can say…” to guide the next action

Room Capabilities – Snapshot (Debug)

A diagnostic tool for understanding what the system sees.

  • Captures a structured snapshot of:
    • discovered devices
    • inferred capabilities
    • helper states
  • Used for debugging and validation during development
  • Never exposed to voice or guests
  • Helps confirm that dynamic discovery is behaving as expected

Sonos Speak with Ducking (Room)

Makes the room speak in a calm, predictable way.

  • Discovers Sonos speakers in the room
  • Temporarily ducks active music to the configured duck level
  • Speaks the message at the room’s chat volume
  • Restores music volume and resumes playback if needed
  • Fails silently if no Sonos exists
  • This ensures one voice per room, independent of the assistant hardware

Turn On Lamps – Usual

Restores a room’s lighting posture.

  • Dynamically discovers lamps using the learned‑brightness helper rule
  • Turns each lamp on to its learned brightness level
  • Applies a safe fallback if a lamp has no learned value yet
  • Never speaks
  • This is what allows a guest to say “lamps on” without knowing anything else

Automations

These are the voice‑exposed entry points. They interpret intent, then delegate the actual work to scripts.

Concierge – Voice Entry (HA Voice)

The “help desk” for the room.

  • Triggered by phrases like:
    • “help”
    • “what can I do here”
    • “what voice commands can I use”
  • Resolves the room from the speaking device
  • Calls Room Abilities – Speak (Sonos)
  • This is how a guest learns the house’s language without a manual

Concierge – Learn on Use

Background learning without ceremony.

  • Listens for stable lamp brightness changes
  • Triggers Learn Lighting – Usual (Room)
  • Updates learned brightness helpers automatically
  • Completely silent — users are never interrupted
  • This is how the house gradually adapts without explicit training sessions

Concierge – Follow‑Up Commands (HA Voice)

The main execution engine for room‑aware commands.

  • Triggered by verb + noun phrases like:
    • “lamps on”
    • “open shades”
    • “music pause”
  • Resolves room context
  • Parses intent from trigger.sentence
  • Speaks a short confirmation first (on the Sonos speakers)
  • Executes the appropriate script or scene in parallel
  • Handles lamps, lights, music, TV, and shades consistently
  • This automation does not “know” devices — it orchestrates scripts that discover them

Concierge – Intentional Learning (HA Voice)

Explicit, opt‑in learning.

  • Triggered by phrases like:
    • “this is good”
    • “that’s perfect”
    • “remember this”
  • Examines current context:
    • Are lamps on?
    • Is music playing?
  • Decides what learning is appropriate
  • Confirms learning via Sonos
  • Calls the relevant learning script
  • Ensures guests never accidentally retrain the system

Why This Breakdown Matters

By keeping scripts small and automations declarative, the system gains:

  • composability
  • testability
  • explainability
  • and long‑term maintainability

Most importantly, this structure allows the user experience to stay simple while the architecture remains robust.

Why This Architecture Works

This system avoids common HA failure modes:

  • :x: No device‑name coupling
  • :x: No hidden learning
  • :x: No schedule wars

Instead, it relies on:

  • room context
  • live discovery
  • explicit intent
  • scenes for actuation
  • conservative learning
  • silence by default

Reusable Takeaways for Other HA Builds

If you’re designing complex automations:

  1. Make room context your root input
  2. Discover capabilities at runtime
  3. Separate discovery, action, and learning
  4. Treat learning as permissioned
  5. Use scenes where grouping matters
  6. Let the system explain itself
  7. Becomes a foundation for future automations (being room aware, know the Sonos speaker in the room and direct conversations there, etc.)

Final Thought on why this was important to me

This wasn’t about adding features—it was about simplifying the language of the house.

Once the system understands where it is and what exists, the YAML gets simpler and the experience gets calmer.

That’s the difference between a smart home that obeys rules and one that understands intent.

Let me know what you think…

2 Likes
2

Can you tell a bit more about the voice assistant(s) you use for this.

3

I updated the original post to include my Home Assistant ecosystem and other important setup items and boiled down each of the components that I used. I hope this is helpful.