Looking for help on understanding configuration logic

Hello everyone,
I am having some trouble understanding some of the configuration.yaml (and underlying) logic. I am using it, but I would like to understand it.

Right now (or at least previously), if you wanted to configure an integration via yaml you had to put it under “packages”. If you wanted to configure a sensor, you had to put it under “sensors”.
And the same applies to automations etc. etc.

But I don’t understand this. Why is a sensor treated differently and cannot be with packages? Especially since sometimes the “package” and the “sensor” are both creating entities for the same piece of hardware (e.g. mqtt connection to my TRVs).
And why can I not put two external configurations in the same folder even though within those yamls the “kind” is clearly specified?

So, if I put the yaml of a sensor and the yaml of a package (which to me are not really two different things if they are part of the same hardware***see note at end) into the same folder, the system will throw an error.

Could someone maybe explain to me why there is the need for these distinctions even if the yaml code itself already contains all information for the system to be able to tell what it is?

Why define in the configuration.yaml that the file describes a sensor even though the file itself contains the information that it is defining a sensor?
I would understand if the yaml did not contain this info, but it does. So what is the idea behind it?

Thank you for your help
Alex

*** I am aware that a sensor is not the same as a switch. But it a piece of hardware contains a switch and a sensor, it would make more sense to be able to configure the piece of hardware in just one location.
To put it more extreme: if the distinction was not just sensor and package but e.g. temperature-sensor vs. humidity-sensor, we would have a million entries in the configuration.yaml and a million files for a very limited amount of devices.

P.S.: I just noticed that MQTT now should allow sensors to be place in packages folder (maybe also vice versa for mqtt climate?). But that is only due to the new mqtt syntax, if I am not mistaken). Must try it now.

I’m not sure if I can but I’ll try to help you understand…

the configuration.yaml file is the main configuration element in HA. it is the “baseline” file.

all of the yaml configuration has to be put in the configuration.yaml file.

All of the code in that file is configured by specifying “key:value” dictionaries or lists.

Each section is defining a top-level domain (or it’s also called an integration) and everything under that top-level domain configures the device/entity that will be a part of that domain/integration.

here is a basic example of a straight configuration.yaml:

homeassistant:
  name: Home Assistant
  latitude: !secret lat
  longitude: !secret long
  unit_system: imperial

logger:
  default: warning #The default level & above to be logged
  logs:
    custom_components.nws_alerts: debug
frontend:
  themes: 
    some_theme:
       .
       .
    some_other_theme:
      .
      .

lovelace:
  mode: yaml
  resources:
    - url: /hacsfiles/cover-control-button-row/cover-control-button-row.js
      type: module
      
system_health:

config:

http:
  server_port: 8124
  use_x_forwarded_for: True
  trusted_proxies:
    - 172.17.0.0/16
  ip_ban_enabled: True
  login_attempts_threshold: 5

history:

logbook:

map:

sun:

mqtt:
  broker: 192.168.1.11
  port: 1883
  client_id: home-assistant-4
  username: !secret MQTT_user  
  password: !secret MQTT_password
  
group: 
  some_group:
  some_other_group

automation: 
  - alias: automation 1
    .
  - alias: automation 2
    .

script: 
  script_1:
    .
  script_2:
    .

input_boolean: 
  some_input_boolean:
  some_other_boolean:

light: 
  - platform: x
    .
  - patform: y
    .

You can completely write your entire config inside the configuration.yaml file and it will work exactly as you want.

But it’s going to get massive and hard to read. So most people break up the configuration.yaml file domains by using !include directives.

the !include directive tells HA that the info contained in the !included file should be treated as if it was appended to the section in the configuration.yaml file where the !include was specified.

And if you use the automation or script UI editor you will be required to use an !include for your auitomations.yaml and scripts.yaml since that is where the UI editors store their respective code.

so modifying the above using !includes:

homeassistant:
  name: Home Assistant
  latitude: !secret lat
  longitude: !secret long
  unit_system: imperial

logger:
  default: warning #The default level & above to be logged
  logs:
    custom_components.nws_alerts: debug

frontend:
  themes: !include themes.yaml 

lovelace:
  mode: yaml
  resources:
    - url: /hacsfiles/cover-control-button-row/cover-control-button-row.js
      type: module
      
system_health:

config:

http:
  server_port: 8124
  use_x_forwarded_for: True
  trusted_proxies:
    - 172.17.0.0/16
  ip_ban_enabled: True
  login_attempts_threshold: 5

history:

logbook:

map:

sun:

mqtt:
  broker: 192.168.1.11
  port: 1883
  client_id: home-assistant-4
  username: !secret MQTT_user  
  password: !secret MQTT_password
  
group: !include groups.yaml

automation: !include automnations.yaml

script: !include scripts.yaml

input_boolean: !include input_booleans.yaml

light: !include lights.yaml

then you will need to create all the yaml files specified using !includes above containing what would otherwise be under those domain keys in the main file.

You can also split up your config into sub-directories with yaml files in those too.

Lastly I think you have a misunderstanding of what packages are for.

Packages have nothing to do with integrations configuration. Every yaml configured integration can be configured directly in the configuration.yaml itself. No packages are required at all.

Packages are to allow you to configure things that wouldn’t normally be allowed to be configured together under a single yaml file.

Example.

under the sensors.yaml file (provided you created an !include directive in configuration.yaml for that file under the “sensor:” key) you can ONLY have sensors configured there. It’s the same for every other domain/integration as well. lights.yaml can only have lights, switches.yaml can only have switches, etc.

Packages allow you to group any integration at all under one yaml file and HA treats it as if the contents of that file was contained directly in configuration.yaml under the respective top-level integrations.

you need to !include packages under the homeassistant key:

homeassistant:
  packages: !include_dir_named packages

and here is an example package yaml file containing various integrations all in the same file:

homeassistant:
  customize:
    cover.both_garage_doors:
      device_class: garage

alert:
  north_gd_open:
    name: The North Garage Door is Still open
    entity_id: sensor.n_gd_pos_template
    state: 'Open'
    repeat: 30
    can_acknowledge: true
    skip_first: true
    message: "The North Garage Door Is Still Open"
    done_message: The North Garage Door is Now Closed
    notifiers:
      - pushbullet_notify

cover:
  - platform: template
    covers:
      north_garage_door:
        friendly_name: 'North Garage Door'
        value_template: "{{ is_state('binary_sensor.garage_door_north_position_sensor', 'on') }}"
        open_cover:
          service: script.turn_on
          entity_id: script.open_gdn
        close_cover:
          service: script.turn_on
          entity_id: script.close_gdn
        stop_cover:
          service: switch.turn_on
          entity_id: switch.garage_door_north_operator_switch
        icon_template: "{% if not is_state('binary_sensor.garage_door_north_position_sensor', 'off') %}mdi:garage-open{% else %}mdi:garage{% endif %}"
                
  - platform: group
    name: Both Garage Doors
    entities:
      - cover.north_garage_door
      - cover.south_garage_door

input_boolean:
  show_garage_power_view:
    name: Show Garage Power Strip View
  north_gd_announce_enabled:
    name: Allow North Garage Door Announcement
    initial: 'on'

input_datetime:
  charger_1_off_time:
    name: Charger 1 Turned Off At
    has_date: true
    has_time: true
  charger_2_off_time:
    name: Charger 2 Turned Off At
    has_date: true
    has_time: true

mqtt:
  sensor:
    - name: 'South Garage Door Position Sensor'
      state_topic: 'USensor_South_GD_Pos/distance_CM'
  
sensor:
  - platform: average
    name: 'Drill Charger Plug Power Average'
    duration:
      minutes: 5
    entities:
      - sensor.drill_charger_plug_power
    
  - platform: template
    sensors:
      n_gd_pos_template:
        friendly_name: "North Garage Door Position"
        value_template: >
          {% if is_state('binary_sensor.garage_door_north_position_sensor', 'off') %}
            Closed
          {% elif is_state('binary_sensor.garage_door_north_position_sensor', 'on') %}
            Open
          {% endif %}
        entity_picture_template: >
          {% if is_state('binary_sensor.garage_door_north_position_sensor', 'off') %}
            /local/my-garage-closed4.png
          {% elif is_state('binary_sensor.garage_door_north_position_sensor', 'on') %}
            /local/my-garage-open4.png
          {% endif %}
automation:

  - alias: GD Announce North Garage Door is Open After Nights
    id: gd_announce_north_garage_door_is_open_after_nights
    trigger:
      - platform: time
        at:
          - '08:40:00'
          - '08:55:00'
    condition:
      - condition: state
        entity_id: cover.north_garage_door
        state: 'open'
      - condition: state
        entity_id: input_boolean.north_gd_announce_enabled
        state: 'on'
    action:
      - service: notify.alexa_media
        data:
          target: 
            - media_player.computer_room_dot
            - media_player.kitchen_dot
          data:
            type: tts 
          message: "The north Garage door is still open"

script:
  close_gdn:
    alias: Close the North Garage Door
    sequence:
      - condition: state
        entity_id: binary_sensor.garage_door_north_position_sensor
        state: 'on'
      - service: switch.turn_on
        entity_id: switch.garage_door_north_operator_switch
  
  open_gdn:
    alias: Open the North Garage Door
    sequence:
      - condition: state
        entity_id: binary_sensor.garage_door_north_position_sensor
        state: 'off'
      - service: switch.turn_on
        entity_id: switch.garage_door_north_operator_switch

as you can see packages are basically just miniature versions of the configuration.yaml file.

I usually use packages to group all associated “things” that are logically related. in the above example everything there is for my garage stuff.

And just to be clear none of the above accounts for ANYTHING configured via the UI. This is just for anything yaml configured.

2 Likes

That is super helpful @finity, thank you :slight_smile:

So, am I correct in understanding that packages is essentially a wild card key and can be used for anything?

So I could delete sensors:, automations: and all the keys from the configuration.yaml and only use packages? And I could then move the yamls from the automations, scripts etc. folders to the packages folder and HA will read them just as before?

If so, packages would be the ultimate config joker :smiley:

I think that you might be able to eliminate almost everything except the homeassistant: key (because you need that to define the packages: !include entry).

You might need to keep the automation: and script: keys and you definitely need to keep the automations.yaml & scripts.yaml files since you need those to use the UI editor.

Unless you don’t use the UI editor then you don’t need any of that except homeassistant: → packages:

However that is all theoretical since I’ve never tried it.

I don’t mind keeping the keys, but it would be pretty nice if I could just use one folder with subfolders and simply have packages point there.
Because then I could have device/group folders and hierarchy like

packages
- shelly
-- 1pm
--- living room
---- automations
---- scripts
-- 2.5

- tasmota
-- esp32
--- living room
-- gosund
--- living room

or a flatter hierrchy like

packages
- tasmota (with scripts, integrations, automations, templates etc.)
- shelly  (with scripts, integrations, automations, templates etc.)
- home connect

Or something similar like manufacturer instead of room etc. . Not sure yet because I never knew I could clean it up this nicely.

I will give it a try and see if it works (and which structure is best).

Not a hundred percent correct, but very close. :slight_smile: You won’t need the automations and script keys in configuration.yaml. The UI does not offer the possibility to change things, if you have them configured in yaml files. The automations.yaml will be created by the UI, if you created an automation in the UI, but it is not necessary at first. The system will create it, if needed. :slight_smile:

@AleXSR7001
You might want to take a different look at these files, more technically. I’ll try my version of explanation, maybe this offers some different insight. :rofl:

The main part of HA are “domains”. These are the upper most configuration levels and need to be defined in configuration.yaml, if you want to use an “!include” approach. These are eg. light, script, automation and so on. This is not about integrations, as these as well use the same domains as we did ourselves in configuration.yaml. Integrations don’t invent new domains (normally), they use the ones from HA. Eg. the HUE integration uses the light domain for the bulbs it controlls, or the switch domain for switches and so on. Or you take a weather integration, it uses the weather domain, and maybe some sensors under it.

As stated before, this can get very un-readable, if you have a bigger installation or better if you have a lot of different devices, sensors and… To avoid this, you can place complete domains in other files and link these files in configuration.yaml. Eg. light: !include my_lights.yaml.

Technically this is nothing fancy. The file configuration.yaml gets parsed, and if it finds some excluded files, it just takes these files and parses them exactly where they are noted. In the end, for the system to work with, it is one big configuration.yaml that is presented to the system.

This is the way you would wanna go, if you have your things sorted by domains. This is useful for some installations, especially if they are not too big.

The other way is the approach by packages. Packages offer the possibilty to group things, eg. for a room or something that is almost always used together. Eg. I have a weather package, where I have everything weather related configured. My weather provider (accu_weather) as well as all automations regarding weather (close the windows notification, if an alert comes in / close the covers if it rains), the used helpers (input_xxx), some template sensors and so on. If I need to take a look for something weather related, I know where to find it.

ha_configuartion_vs_packages

You get the idea :wink: What the system does in the background is different to an !include in configuration.yaml. It takes the package file apart, and sorts it into the specific domains that are used.

To sum that up: packages are one of two ways to include and group/order things. One way is the domain approach by including domains via a file. The other way is to break things up and put them in to content related packages. It is just a different approach to include something. And the “!include” files is something that can get very un-readable as well. Assume you have a hundred lights over different rooms, you’ll never find the specific light you’re searching for.

But let me give you a recommendation, as I’m mostly using packages. :slight_smile: Don’t clutter it to much! It is not useful, to divide things to their deepest level. Use something, that is easy to remember for you. :slight_smile: Put all things together, that fit together - nothing more, nothing less. But don’t use it in a “company” approach, use it more like a group approach. In my presence detection package is everything that has to do with it, even the frontdoor sensor, the light to turn on when coming home and so on. “Thematical diversion” if you will :rofl: :rofl:

###############
#
# base package 
# 
###############
homeassistant:
  customize:
    person.steffi:
      picture: /local/pictures/steffi.jpg
      entity_picture: /local/pictures/steffi.jpg
    person.patrick:
      picture: /local/pictures/pat.jpg
      entity_picture: /local/pictures/pat.jpg
    climate.livingroom_ac:
      hvac_modes:
        - 'off'
        - 'cool'
        - 'dry'
        - 'fan_only'
      swing_modes:
        - 'off'
        - 'vertical'
    sensor.livingroom_average_temperature:
      icon: 'mdi:thermometer'

sensor:
  - platform: systemmonitor
    resources:
      - type: processor_temperature
      - type: processor_use
      - type: memory_use
      - type: disk_use_percent
  - platform: uptime
  - platform: time_date
    display_options:
      - 'time'
      - 'date'
      - 'date_time'
  - platform: sun2
    entity_namespace: sun2
    monitored_conditions:
      - solar_midnight
      #- astronomical_dawn
      #- nautical_dawn
      - dawn
      - sunrise
      - solar_noon
      - sunset
      - dusk
      #- nautical_dusk
      #- astronomical_dusk
      - daylight
      - civil_daylight
      #- nautical_daylight
      #- astronomical_daylight
      - night
      - civil_night
      #- nautical_night
      #- astronomical_night
      - elevation
      - min_elevation
      - max_elevation
  - platform: template
    sensors:
      friendly_day_of_week:
        value_template: >
          {% set dayofweek = (as_timestamp(states('sensor.date')) | timestamp_custom('%a', true)) %}
          {% if dayofweek == 'Mon' %}
          Montag
          {% elif dayofweek == 'Tue' %}
          Dienstag
          {% elif dayofweek == 'Wed' %}
          Mittwoch
          {% elif dayofweek == 'Thu' %}
          Donnerstag
          {% elif dayofweek == 'Fri' %}
          Freitag
          {% elif dayofweek == 'Sat' %}
          Samstag
          {% else %}
          Sonntag
          {% endif %}
      friendly_date:
        value_template: >
          {{ (as_timestamp(states('sensor.date')) | timestamp_custom('%d.%m.%Y', true)) }}
      friendly_day_and_date:
        value_template: >
          {{ states('sensor.friendly_day_of_week') }}, {{ states('sensor.friendly_date') }}

binary_sensor:
  - platform: workday
    country: DE
    province: BY
    workdays: [mon, tue, wed, thu, fri]
    excludes: [sat, sun, holiday]
  - platform: sun2
    entity_namespace: sun2
    monitored_conditions:
      - elevation
      - elevation: 3
      - elevation:
          above: -6
          name: Above Civil Dawn

script:
  nothing:
    sequence:
      - delay: '00:00:01'
  nothing_light:
    sequence:
      - delay: '00:00:01'

automation:
  - id: Create All Lights Group on Startup
    alias: Create All Lights Group on Startup
    trigger:
      platform: homeassistant
      event: start
    action:
      - service: python_script.create_group_all
        data:
          domain: light
          group: all_lights

group:
  all_windows:    
    name: Alle Fenster
    entities:
      - binary_sensor.bathroom_window_contact_on_off
      - binary_sensor.bedroom_window_contact_on_off
      - binary_sensor.guesttoilet_window_contact_on_off
      - binary_sensor.kitchen_window_contact_on_off
      - binary_sensor.diningroom_window_contact_on_off
  all_doors:
    name: Alle Türen
    entities:
      - binary_sensor.livingroom_balconydoor_contact_on_off
      - binary_sensor.garage_garagedoor_contact_on_off
      - binary_sensor.hallway_frontdoor_contact_on_off 
      - binary_sensor.garage_backdoor_contact_on_off

And to make things a little unclear again: if your installation grows, you’ll find yourself including domains in configuration.yaml and using packages together. :rofl: :rofl: :rofl:

Hope this helps as an addition to what @finity already explained very well! :slight_smile:

Thank you, that is a very good second explanation.

I think I now understand it. And I think I will vhose the packages approach. For me it is easier than the domain approach because I often have devices with multiple domains and I look at my home as a set of devices, not domains.

I will try to keep it simple. And the current domains approavh I had (because I thought this was mandatory) was making it complex.

I will try something like
packages

  • shelly (including associated automations, scripts etc.)
  • mqtt (including climate, sensors, automations, scripts etc.)
  • tasmota (including associated automations, scripts etc.)

And in the base folder packages also all integrations and devices that do not fit a different grouping (e.g. home connect).

Thanks for the compliment! :wink: :slight_smile:

I’d take more of an “waht-am-I-searching-for” approach. :rofl:

Think of it the other way around. Let’s assume you have all parted into different package files. What if you’re looking for something, is it all in one file? Than it seems to be a good choice. If on the other hand you still need to open two or three files, than your packages approach is not the “right” way.

Eg. I’d put the Shellys to the automations they belong to. If you have one Shelly in your kitchen, put it in kitchen_packages. But the other one is located or better used in combination with something in the living room, put it there.

But in the end it’s about your comfort. You need to work with that system, if you’re comfortable with all Shellys in one file, do it! The good thing with packages is, you can always move them around or change them back and forth.

And just to note: the reload functions of specific domains work in packages as well. When you reload your automations (the reload under developer tools), they will be reloaded, regardless where they are, in a package or in an included file. :wink:

Let us know what you came up with, others might be interested to see how you do it. Everybody drives a different approach here, and it is always nice to see, how others handle that. :slight_smile:

1 Like

My approach will probably change over time, just like the old domain approach did.

My current syntax is keyword_domain_name/description.yaml
The keyword is either the manufacturer or a related name that I associate with that group. So e.g. my shellys are all power controls so 1PM, 2.5 etc. No TRVs or light bulbs or anything. Should the latter be added at some point, I will rename.
All my external power sockets (so plug-in sockets) are from Gosund, so they are a sub-category of Tasmota (the firmware).
So the keyword is simply my own reference. And then followed by the domain (so I know the syntax I need) and then the name or description (so I know what the yaml does).

For now I am starting with (root folder is packages):

./eq3:
eq3_automation_battery_dead_notification.yaml
eq3_automation_battery_dying_notification.yaml
eq3_automation_battery_low_notification.yaml
eq3_automation_climate_away.yaml
eq3_customize_automations.yaml
eq3_mqtt_climate.yaml
eq3_mqtt_sensor_battery.yaml
eq3_mqtt_sensor_temperature.yaml
eq3_mqtt_sensor_valve.yaml
eq3_script_away.yaml
eq3_script_homeoffice.yaml
eq3_script_sleep.yaml
eq3_script_sleep_low.yaml
eq3_script_weekend.yaml
eq3_script_weekend_low.yaml

./shelly:
shelly_automation_garage_door_notification.yaml
shelly_automation_overpowering_overheating_notification.yaml
shelly_customize_garage_door.yaml

./tasmota:
esp32_template_uptime_in_s.yaml
gosund

./tasmota/gosund:
gosund_automation_naim_audio_nsub_on_off.yaml
gosund_customize.yaml

./xiaomi_hygrotemp:
xiaomi_mqtt_sensor_battery.yaml
xiaomi_mqtt_sensor_dew_point.yaml
xiaomi_mqtt_sensor_humidity.yaml
xiaomi_mqtt_sensor_temperature.yaml

.:
app_tiles_script.yaml
days_back_to_show_input_select.yaml
db_file_size_sensor.yaml
home_connect_integration.yaml
ista_ecotrend_integration.yaml
ista_ecotrend_template.yaml
logger_automation_set_default.yaml
logger_integration.yaml
recorder_integration.yaml
telegram_bot_integration.yaml
telegram_notify.yaml
xiaomi_hygrotemp
1 Like

Quick update:
I also played around with @paddy0174 `s idea of grouping by room. This is a very logical approach. Unfortunately, for me personally, I have gotten used to looking at manufacturer first, purpose second and location third. I would call it “syntax” approach.

My “syntax” approach is based on two things

  1. I started off with Shellys, then came Gosund, then came eq-3…
    So I started “coding” per manufacturer.

  2. The device/entity integration is usually consistent for all devices of one manufacturer. So e.g. all Shellys are mqtt and since there is a Shelly integration, any code will be the same basis.
    All Gosund use Tasmota, so again, all need the same code.
    All eq-3 are routed via a Tasmota based BLE hub, so they all use the same code.

And the most common need for me to touch my yaml files is to fix Breaking Changes or to adjust for new HA syntax. So this will always affect a group of devices, usually per manufacturer and never per location.

I think the location approach is better when you have many integrations and you tend to modify code per room (e.g. you bundle light behaviour or climate behaviour independent of device type/code needs).

If you primarily want to maintan the code according to HA syntax/breaking changes or to add more devices of the same kind, then I prefer my “syntax” approach.

It is also quite possible that one of the reasons this works better for me is, that I do not use a lot of automations “per room”. If I make adjustments, e.g. climate, they are always for all rooms (night, away, comfort etc.).