MQTT breaking changes RC 2022.6

Configuration
15

Except it was always mqtt integration and always said MQTT. I’m not really understanding, the config went from this:

sensor:
  - platform: mqtt
    name: My MQTT sensor
    ...

cover:
  - platform: mqtt
    name: My MQTT cover
    ...

To this:

mqtt:
  sensor:
    - name: My MQTT sensor
      ...

  cover:
    - name: My MQTT cover
      ...

If anything it says MQTT less times now since you don’t have to put platform: mqtt in everything.

But more importantly here is this was never the cover integration, it was always the mqtt integration. Why does that matter? In the first part what wasn’t obvious to people who aren’t deeply familiar with the config is that your MQTT sensors, covers, selects, etc. all shared the same MQTT connection to the broker. Because they were all part of the MQTT integration despite being spread out all over your config. A user would likely be quite confused why in the UI all these things appeared under MQTT instead of under cover, sensor, etc. in the settings menus.

Now everything is grouped right under MQTT showing that they are clearly all part of that integration. All these entities share the same connection details and are based on topics on the same broker.

16

I do not think one way is better than another. You can see it both ways.

It is just another breaking change that 10000s of users need to deal with - every month.

It is too much now!

5 Likes
17

While I tend to agree with you, the new yaml format for MQTT looks cleaner and more logical to me. The repeated platform key appearing for every sensor was weird. I have a lot of MQTT stuff, so I really notice the difference.

That said, I also think that making this a breaking change was unnecessary. Both formats could have been kept at only minimal maintenance overhead. They made it a breaking change simply because it would make the config handling code cleaner.

2 Likes
18

The curious part is that this is the opposite of how entities in Home Assistant have traditionally been configured/organized.

For countless versions, entities have been organized according to their domain such as light, switch, sensor, binary_sensor, lock, etc and not the underlying integration that supports the entity. It’s been the responsibility of the platform option to indicate which integration is used for a given entity.

It was the recently revised Template integration that (I believe) was first to diverge from this long-standing organizational model. Now the MQTT integration has followed suit.

Would you happen to know the rationale for this change of organizational structure?

FWIW, I am unaffected by this change because I use MQTT Discovery (via scripts) to define all MQTT entities in my system (sensors, locks, binary_sensors, lights, switches, etc). According to the documentation, the new method of defining MQTT entities still doesn’t allow you to define devices (whereas you can with MQTT Discovery).

2 Likes
19

I’m not 100% on all the reasons. I looked up the first or in the series of them that rearranged Mqtt config to see if the author said anything:

Sounds like config entry set up prefers it. I cant say requires because Mqtt was using config entries before this but perhaps it was given an exception.

I think it may also have performance benefits. In many of the integrations that have done this I’ve seen centralized coordinators appear in the code to manage communication with the external service and update many entities with one response.

I think this was possible before but I’m not sure it was possible to have a centralized coordinator that updated entities of many different types. Because I believe in the new model the integration is handed the entire config to set itself up with all at once. Whereas before it was asked to set up sensors, covers, selects, etc all independently.

I’m less sure about this though since I don’t have a clear source to point to. Just a pattern I noticed in more recent code.

20

FWIW, someone asked the same question elsewhere (Reddit) and frenck replied that’s it’s a long overdue effort to bring MQTT-based entities into compliance with this ADR:

3 Likes
21

This ADR is interesting…

This is something that really confused me when I built my little integration: it’s a media player (i.e. configured under the media_player section in YAML), but it uses MQTT. So, the integration in this case is the media player, and the platform is MQTT, but originally it wasn’t clear to me which way around this had to be. Now I’m wondering whether I will need to update this at some point to sit under an mqtt: section. I think, the class/type/function of something should be treated differently to how it connects. I’m possibly missing something subtle here, but it seems like the concepts are getting mixed. Personally, I like to think of the domain as the top level concept, with the mechanism (templates, MQTT, REST) as secondary to that. I’ll deal with it either way, but thought I’d bring this up.

1 Like
22

HA is moving away from YAML to UI configuration to, I assume, become more accessible to new users. It would be madness to set up lights under “Lights”, unless they were MQTT lights, in which case you had to set them up under “MQTT” (or unless they were “Templates”, etc etc). Once it’s all done, you can’t imagine a new screen where you manage all of your lights in one place.

23

No, its actually neither of those. Your integration is called shairpoint_sync therefore if you are trying to follow that ADR then all config for it should be under the key sharepoint_sync in configuration.yaml. So something like this:

shairpoint_sync:
  name: Shairport Sync Player
  topic: your/mqtt/topic

If your integration supported multiple platforms and each had their own options then it might be more like this:

shairpoint_sync:
  media_player:
    name: Shairport Sync Player
    topic: your/mqtt/topic
  sensor:
    name: Shairport Sync Player sensor
    topic: your/mqtt/topic

The fact that your integration uses mqtt under the hood isn’t relevant. How your integration actually communicates with its external devices/services is an implementation detail. Ideally you hide those kinds of details from the user and simply expose options that make sense without requiring the user to actually know how it works under the hood.

MQTT makes this confusing since its a communication protocol rather then a specific service. But the thing to remember is that the only config that can be under the mqtt key in configuration.yaml is stuff actually handed to the mqtt integration. You need your config to be handed to the shairpoint_sync integration not the mqtt integration so it can’t be under mqtt. It either has to be under a shairpoint_sync key or using the old style with a platform key under media_player.

Also keep in mind that ADRs apply to core integrations. ADRs are enforced by code reviews and no one is reviewing custom integrations but the author(s). If you were thinking of contributing that back to core then yea you’d need to follow the ADRs. But if you just want to leave it in HACS then you can continue doing whichever approach you prefer here.

25

Yeah, I buggered up my own example…

It’s actually this:

So TIL that it’s the old way – I didn’t know that.

I agree 100% with that statement, but it seems to contradict the main point, which is to put things under a mqtt: key/section. It’s just a comms layer – like HTTP, etc. which I think is your point too:

And the above is what I’m kind of questioning or trying to understand better. I know I’ll probably not get the true answer here unless one of the devs chime in, but it’s good to hear all the explanations regardless.

True, but I personally try to follow best practices and keep the options open. It wasn’t really to talk about my contribution, but it’s an example I understand well, which was the reason I used it.

26

It’s besides the point. I know this, and I’m fine with that and yes that’s the reason it’s being done. I need to add the config flow to my stuff – but that’s another topic.

I don’t understand why this seems illogical – or mad.

27

Having a nightmare with this
My configuration.yaml file:

mqtt: !include_dir_merge_named mqtt/
sensor: !include_dir_merge_list sensors

because I have sensors that are not MQTT
I have 2 files in mqtt:
acurite-humidity.yaml

sensor:
- name: "Indoor Humidity"
#'acurite/humidity'
  state_topic: "home/rtl_433"
  unit_of_measurement: '%'
  value_template: >
    {% if value_json is defined and value_json.id == 11794 %}
      {{ value_json.humidity }}
    {% else %}
      {{ states('sensor.indoor_humidity') }}
    {% endif %}

and acurite-temperature.yaml

sensor:
- name: "Indoor Temp"
#'acurite/temp'
  state_topic: "home/rtl_433"
  unit_of_measurement: '°C'
  value_template: >
    {% if value_json is defined and value_json.id == 11794 %}
      {{ value_json.temperature_C }}
    {% else %}
      {{ states('sensor.indoor_temp') }}
    {% endif %}

HA only shows the first sensor.
You can’t put more than 1 sensor in the same file - get an error.
(this was all working before the mqtt change!)

28

Unfortunately, YAML - Home Assistant documents the old syntax to explain nesting a collection of mappings in a mapping.

sensor:
  - platform: mqtt
    state_topic: "sensor/topic"
  - platform: mqtt
    state_topic: "sensor2/topic"

It will be very confusing for some time to those of us struggling to understand YAML syntax.

1 Like
29

Because you’re using merge dir named. If you want to use it that way, you have to make separate folders for each domain…

mqtt:
  sensor: !include_dir_merge_list mqtt/sensor/

And then remove sensor: from your files.

Or if you want to use merge_named, then you can only have 1 file per domain.

1 Like
30

can’t use the keyword ‘sensor’ more than once in configuration.yaml
and I have other non mqtt sensors

31

got it!
configuration.yaml:

mqtt:
  sensor: !include_dir_merge_list mqtt/
sensor: !include_dir_merge_list sensors

mqtt/humidity.yaml

- name: "Indoor Humidity"
#'acurite/humidity'
  state_topic: "home/rtl_433"
  unit_of_measurement: '%'
  ....
32

I have a number of MQTT devices and would like to get ahead of 2022.9 and not repeat my debacle of waiting too long to change my Z-Wave stuff.
So, what’s wrong here?

OLD:

switch:
  - platform: mqtt
    name: "steve_Heat"
    command_topic: "cmnd/steve_heat/power"
    state_topic: "stat/steve_heat/POWER"
    qos: 1
    payload_on: "ON"
    payload_off: "OFF"
    retain: false

NEW:

mqtt:
  switch:
    -name: "steve_Heat"
     command_topic: "cmnd/steve_heat/power"
     state_topic: "stat/steve_heat/POWER"
     qos: 1
     payload_on: "ON"
     payload_off: "OFF"
     retain: false

Is there a dictionary or Rosetta stone that helps interpret the error messages?
What is a <block end> and a '<block mapping start>' ?
[Edit to fix forum software munging my question].

33 
mqtt:
  - switch:
      - name: "steve_Heat"
        command_topic: "cmnd/steve_heat/power"
        state_topic: "stat/steve_heat/POWER"
        qos: 1
        payload_on: "ON"
        payload_off: "OFF"
        retain: false
34

I didn’t notice that my earlier question was munged by the forum software, so I’ll ask again in code tags.

Also your suggested fix produced this error:

The system cannot restart because the configuration is not valid: Error loading /config/configuration.yaml: while parsing a block mapping in "/config/configuration.yaml", line 262, column 7 expected <block end>, but found '<block mapping start>' in "/config/configuration.yaml", line 263, column 8

Is there a codex or universal translator to help understand, just what is a <block end> and a '<block mapping start>' ?

35

Try this way. I have a similar switch setup in a package.

mqtt:
  switch:
    - name: "steve_Heat"
      command_topic: "cmnd/steve_heat/power"
      state_topic: "stat/steve_heat/POWER"
      qos: 1
      payload_on: "ON"
      payload_off: "OFF"
      retain: false