Using duplicate keys in package yamls

Timmiej93 · September 9, 2023, 1:12pm

I started using package yamls to keep my things neatly organized, and to prevent having to switch between multiple files when working on a project. However, when I also tried organizing the contents of the file, I ran into the issue of not being able to use duplicate keys (being sensor, template, etc.).

For example, I can’t do this, even though it would make much more sense for me:

# ==============================
# HELPER ENTITIES
# ==============================

template:
  - sensor:
    - ...

binary_sensor:
  - platform: trend
    ...

# ==============================
# MAIN SECTION
# ==============================

template:
  - sensor:
    - ...

Proposal
Allow the usage of duplicate keys in package yaml files. Especially when including the package file with include_dir_(merge_)named, there’s already some processing going on that allows the same keys from multiple files, so it would be awesome if that could be extended to allowing the same key in one file too.

Discussion
I started a discussion (that was supposed to be a feature request) here: Allow duplicate top level keys in a package YAML

tom_l · September 9, 2023, 4:20pm

Edit: never mind I see this was suggested in the topic you linked to.

You should be able to use duplicate labelled integration keys.

See this example:

So something like this:

# ==============================
# HELPER ENTITIES
# ==============================

template helper:
  - sensor:
    - ...

binary_sensor helper:
  - platform: trend
    ...

# ==============================
# MAIN SECTION
# ==============================

template main:
  - sensor:
    - ...

Timmiej93 · September 9, 2023, 4:36pm

Thanks for the suggestion. It was indeed suggested in the other topic, but somehow it won’t work. Any idea as to why it doesn’t work?

tom_l · September 9, 2023, 4:39pm

Did you also have those keys in your configuration.yaml file?

That may be where the duplicates are coming from if you labelled all the ones in your packages.

paddy0174 · September 9, 2023, 4:53pm

So is this the topic where we should answer or the other one? @tom_l could you please close the other topic with a link to this one here?

The labelled keys should work as suggested, I was using this exactly like described from @tom_l until a few months ago. In my case it turned out to be a nightmare. I was always looking for this and that, and I needed to search through the whole file, instead of only one block.

I tried another variant with packages, I sorted them in sub folders. What was my vacuum_package.yaml before is now a folder vacuum and inside i have zone_cleaning.yaml and spot_cleaning.yaml and so on. For my workflow this is much better, but everybody should find its own comfort zone.

Btw. @Timmiej93 you should vote yourself for your FR.

tom_l · September 9, 2023, 4:56pm

Closed the other topic and redirected here as requested.

Timmiej93 · September 10, 2023, 10:47am

No, this is the first time I’ve used labels, so they only exist in the package yaml.

Either or is fine with me, I kept both since I figured one would be directed more toward discussing workarounds, and the other would focus on if this would be a good feature to add. I agree that closing one is probably better.

I kinda expected it to do that automatically, thanks.

Here’s my (simplified) configuration.yaml, and the package yaml, zigbee_rain_meter.yaml.

# configuration.yaml
# Configure a default setup of Home Assistant (frontend, api, etc)
default_config:

# Text to speech
tts:
  - platform: google_translate

group: !include configurationYamls/group.yaml
automation: !include configurationYamls/automation.yaml
script: !include configurationYamls/script.yaml
scene: !include configurationYamls/scene.yaml
template: !include configurationYamls/template.yaml
sensor: !include configurationYamls/sensor.yaml
binary_sensor: !include configurationYamls/binary_sensor.yaml

homeassistant:
  packages: !include_dir_named configurationYamls/projectYamls
# I also tried packages without homeassistant, but then I get "while parsing a 
#   block mapping in "/config/configuration.yaml", line 2, column 1 expected <block end>,
#   but found '<block mapping start>' in "/config/configuration.yaml", line 17, column 3".
#   Line 2 is `default_config:`, line 17 is the line that has `packages: ...`.

http:
  use_x_forwarded_for: true
  trusted_proxies:
    - 172.18.0.0/16

system_log:
  max_entries: 200

api:

frontend:
  extra_module_url:
    - /hacsfiles/custom-sidebar-v2/custom-sidebar-v2.js
    - /local/custom_sidebar_order.js
  themes: !include_dir_merge_named themes/

mqtt_statestream:
  base_topic: homeassistant
  publish_attributes: false
  publish_timestamps: false

influxdb:
  ...

zha:
  enable_quirks: True
  custom_quirks_path: custom_zha_quirks

panel_custom:
  ...

# Package: zigbee_rain_meter.yaml
#
# RAIN METER BASED USING DOOR AND WINDOW SENSOR
# Standard tipping style rain meter, using the Aqara Door and Window sensor
# 
# Model:      Aqara MCCGQ11LM
#   Battery:  CR1632
#
# Entities used and not defined here:

# ==============================
# HELPER ENTITIES
# ==============================

# Calculate amount of rain in liter or mm per m² based on funnel size and volume required to tip the bucket
# Measurements: 1 liter tips bucket 152, 174 and 168 times. Assuming 170 is correct.
template helper:
  - sensor:
    - name: "Aqara Rain Meter - Rainfall per tick"
      unique_id: '0a7476cc-d6c1-40ba-8351-606518c3497sdfs'
      state_class: total
      unit_of_measurement: l/m²
      state: >
        {# Fill funnel radius and ticksPerLiter #}
        {% set funnelRadius = 0.055 %}
        {% set ticksPerLiter = 170 %}

        {# Don't change anything below #}
        {% set litersPerTick = 1 / 170 %}
        {{ (1 / (pi * ( funnelRadius**2 ))) * literPerTick }}

  - sensor:
    - name: "TESTTESTTEST"
      unique_id: '0a7476csdfc-d6c1-40ba-8351-606518c3497sdfs'
      state_class: total
      unit_of_measurement: l/m²
      state: 1

# --------------------------------------------------------------
# Convert rain meter ticks into rainfall intensity in mm/h / l/h
# --------------------------------------------------------------

binary_sensor:
  - platform: trend
    sensors:
      aqara_rain_sensor_rainfall_trend:
        friendly_name: "Rainfall trend"
        entity_id: sensor.aqara_rain_sensor_rainfall_today
        max_samples: 10
        sample_duration: 300

template helper2:
  - sensor:
    - name: "Aqara rain intensity"
      state_class: measurement
      device_class: precipitation
      unit_of_measurement: 'mm/h'
      unique_id: aqara_rain_sensor_rainfall_per_hour
      state: >-
        {% set rain_hour = ((state_attr('binary_sensor.aqara_rain_sensor_rainfall_trend', 'gradient') | float(0)) * 3600) | round(1, 'floor') %}
        {% if rain_hour >= 0 %}
          {{ rain_hour }}
        {% else %}
          {{ 0 }}
        {% endif %}

# ==============================
# SENSORS & BINARY SENSORS
# ==============================

sensor:
    # Flips ON | TODAY
    # Determines how many times the sensor was in the ON state today
  - platform: history_stats
    unique_id: "aqara_rain_sensor_flips_on_today"
    name: "Aqara rain sensor flips/on (Today)"
    entity_id: binary_sensor.aqara_dw_2_open_closed
    state: "on"
    type: count
    start: "{{ now().replace(hour=0, minute=0, second=0, microsecond=0) }}"
    duration:
      hours: 24
    # Flips OFF | TODAY
    # Determines how many times the sensor was in the OFF state today
  - platform: history_stats
    unique_id: "aqara_rain_sensor_flips_off_today"
    name: "Aqara rain sensor flips/off (Today)"
    entity_id: binary_sensor.aqara_dw_2_open_closed
    state: "off"
    type: count
    start: "{{ now().replace(hour=0, minute=0, second=0, microsecond=0) }}"
    duration:
      hours: 24
      
    # Flips ON | YESTERDAY
    # Determines how many times the sensor was in the ON state yesterday
  - platform: history_stats
    unique_id: "aqara_rain_sensor_flips_on_yesterday"
    name: "Aqara rain sensor flips/on (Yesterday)"
    entity_id: binary_sensor.aqara_dw_2_open_closed
    state: "on"
    type: count
    start: "{{ now().replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(hours=24) }}"
    duration:
      hours: 24

    # Flips OFF | YESTERDAY
    # Determines how many times the sensor was in the OFF state yesterday
  - platform: history_stats
    unique_id: "aqara_rain_sensor_flips_off_yesterday"
    name: "Aqara rain sensor flips/off (Yesterday)"
    entity_id: binary_sensor.aqara_dw_2_open_closed
    state: "off"
    type: count
    start: "{{ now().replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(hours=24) }}"
    duration:
      hours: 24

    # TEST TEST TEST
    # Flips TOTAL | TODAY
    # Determines how many times the sensor toggled today
  - platform: history_stats
    unique_id: "aqara_rain_sensor_flips_total_today"
    name: "Aqara rain sensor flips/total (Today)"
    entity_id: binary_sensor.aqara_dw_2_open_closed
    state:
      - "off"
      - "on"
    type: count
    start: "{{ now().replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(hours=24) }}"
    duration:
      hours: 24

template main:
  - sensor:
      # Calculate rain volume for the current day based on flips on and off for today
    - name: "Aqara Rainfall (Today)"
      state_class: total_increasing
      device_class: precipitation
      unit_of_measurement: mm
      unique_id: aqara_rain_sensor_rainfall_today
      state: >-
        {% set count = (states('sensor.aqara_rain_sensor_flips_on_today') | int(0)) + (states('sensor.aqara_rain_sensor_flips_off_today') | int(0)) %}
        {% set mm = count * float(states('sensor.aqara_rain_meter_rainfall_per_tick')) %}
        {% if count >= 0 %}
          {{ mm | round (1) }}
        {% endif %}

      # Calculate rain volume for yesterday based on flips on and off for yesterday
    - name: "Aqara Rainfall (Yesterday)"
      state_class: total_increasing
      device_class: precipitation
      unit_of_measurement: mm
      unique_id: aqara_rain_sensor_rainfall_yesterday
      state: >-
        {% set count = (states('sensor.aqara_rain_sensor_flips_on_yesterday') | int(0)) + (states('sensor.aqara_rain_sensor_flips_off_yesterday') | int(0)) %}
        {% set mm = count * float(states('sensor.aqara_rain_meter_rainfall_per_tick')) %}
        {% if count >= 0 %}
          {{ mm | round (1) }}
        {% endif %}

Like mentioned in the other topic, if I put the templates with labels like this, and use

homeassistant:
  packages: ...

then the file loads, but none of the template sensors load:

If I put all templates under one template: key, without labels, they load just fine.

An important note: I think we can forget about the duplicates error. While testing this, to make sure I was writing down the correct error messages here, I noticed that HA just repeats the last error message when I use labeled templates. Now if I try loading the packages without having homeassistant: above, it gives me these errors. This makes me think HA isn’t providing proper error messages for this issue. Does anyone see an issue why the labels wouldn’t work?

2023-09-10 10:19:28.276 ERROR (SyncWorker_12) [homeassistant.util.yaml.loader] while parsing a block mapping
in "/config/configuration.yaml", line 2, column 1
expected <block end>, but found '<block mapping start>'
in "/config/configuration.yaml", line 17, column 3
2023-09-10 10:19:28.277 ERROR (MainThread) [homeassistant.components.homeassistant] The system cannot reload because the configuration is not valid: Error loading /config/configuration.yaml: while parsing a block mapping
in "/config/configuration.yaml", line 2, column 1
expected <block end>, but found '<block mapping start>'
in "/config/configuration.yaml", line 17, column 3
2023-09-10 10:19:28.278 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140178582534416] Cannot quick reload all YAML configurations because the configuration is not valid: Error loading /config/configuration.yaml: while parsing a block mapping
in "/config/configuration.yaml", line 2, column 1
expected <block end>, but found '<block mapping start>'
in "/config/configuration.yaml", line 17, column 3
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 205, in handle_call_service
await hass.services.async_call(
File "/usr/src/homeassistant/homeassistant/core.py", line 1910, in async_call
task.result()
File "/usr/src/homeassistant/homeassistant/core.py", line 1950, in _execute_service
await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 887, in admin_handler
await result
File "/usr/src/homeassistant/homeassistant/components/homeassistant/__init__.py", line 316, in async_handle_reload_all
raise HomeAssistantError(
homeassistant.exceptions.HomeAssistantError: Cannot quick reload all YAML configurations because the configuration is not valid: Error loading /config/configuration.yaml: while parsing a block mapping
in "/config/configuration.yaml", line 2, column 1
expected <block end>, but found '<block mapping start>'
in "/config/configuration.yaml", line 17, column 3

tom_l · September 10, 2023, 10:51am

I meant these, in your configuration.yanl file.

Timmiej93 · September 10, 2023, 11:12am

Ah right, I misread. I did have those entities in my template.yaml file before, but they’re commented out, just like the binary_sensor and sensor entities.

Since the template entities work when they’re under one template: key, I’m pretty convinced it has to be something to do with the labels, I just can’t figure out what.

I just updated HA (realized I was on 2023.6), and tried using the labeled template keys again, but the template entities then simply show up as unavailable, while the rest of the package yaml’s contents work fine.

Update: It really seems like my HA install just doesn’t like labels. I tried adding a simple test template in my template.yaml file:

- sensor:
  - name: "test sensor"
    state: >
      {{ "test sensor is working" }}

After a refresh, that showed up immediately, working just fine. So I removed the entry from template.yaml, and added it to the configuration.yaml file:

template: !include configurationYamls/template.yaml

...

template:
  - sensor:
    - name: "test sensor 2"
      state: >
        {{ "test sensor is working again" }}

To my surprise, this worked! So I tried having both, to confirm it’s really working. Test sensor 1 in template.yaml, test sensor 2 in configuration.yaml. Now only test sensor 2, from the configuration.yaml file shows up. Odd. So, I added a label to the template key in configuration.yaml:

template label1:
  - sensor:
    - name: "test sensor 2"
      state: >
        {{ "test sensor is working again" }}

And of course, now only test sensor 1 is showing, the template.yaml file one… So as a last ditch effort, I also added a label to the main template key in configuartion.yaml: template main: !include configurationYamls/template.yaml, but this throws an error and doesn’t even bother reloading.

This is starting to get very strange, random and frustrating. Any thoughts why this could be happening?