MQTT problem after Core update to 2023.6

[schraepfert]

Hello Community,
after the core update to 2023.6 I already noticed problems with MQTT. Now I found that after the update to 2023.7 the problems are the same. At the moment I am back on 2023.4 thanks to a full backup.

My setup:

• RaspberryMatic CCU3 with CCU-Jack as MQTT broker (over 200 MQTT entities).
• Home Assistant fetches the data via MQTT integration unencrypted over port 1883 with username and password
• The MQTT integration is configured in the UI
• The entities are configured manually via mqtt.yaml text file

The problems are as follows:
After the update, I can switch an MQTT switch exactly once. The UI shows the change, but switching back is not possible. As soon as I reload the MQTT integration, I can switch once again, because the entities and their states are read in again.
In the MQTT explorer it can be seen that whenever I press the switch again, a command is sent, but this just always confirms the current state and does not switch. It’s as if the MQTT state is not being transmitted properly, but the MQTT Set command is.

What I have already done:

• Reboot the system after the update.
• Deleted MQTT integration after update and set it up again
• Deleted MQTT integration, deleted mqtt.yaml, rebooted system and reinstalled everything
• checked core logs → no peculiarities

Has anyone here perhaps had the same problem and can help?
I am unfortunately just trapped on 2023.4

Thank you very much for your help.

Greetings
Thomas

[tom_l]

Did you check the release notes (for each update between the version you are on and the latest version) for breaking changes to mqtt?

Category: Release-Notes

What errors related to this are visible in Settings - > System → Logs?

[schraepfert]

Hi tom_I,

thank you for your reply.

The only thing I could find in the release notes is the following section:

The way how retained messages are processed has changed:
The way MQTT messages with a set retain flag are handled has changed such that existing
subscribers are no longer passed retained messages re-sent by the broker as a result of new subscribers subscribing to the same topic. Instead, retained messages re-sent by the broker are only passed to the new subscriber.

At least that would be the closest, since I set the retain flag on all entities. But still, as I understand it, this text passage should only apply when a new subscriber is added.
So I wouldn’t call this a change that affects my operation.

In the logs, as I said, I can’t find anything that indicates any errors in MQTT.

It just seems to me that it is not reading the state-topic correctly. So it doesn’t get that the state of the device has changed. But in the definition of the state-topic nothing has changed to my knowledge.

Greetings
Thomas

[schraepfert]

Does no one have another idea?

[tom_l]

@schraepfert

The entities are configured manually via mqtt.yaml text file

Can you share the configuration of an entity that is not updating correctly and a screen shot of the topic in MQTT Explorer.

[schraepfert]

Here is my config inside the mqtt.yaml:

switch:
  - name: "Lampe TV"
    unique_id: lampe_tv
    icon: hass:lightbulb
    state_topic: "device/status/XXXXXXXXXX3CC4/14/STATE" 
    command_topic: "device/set/XXXXXXXXXX3CC4/14/STATE"
    qos: 1
    payload_on: '{"v":true}'
    payload_off: '{"v":false}' 
    value_template: "{{value_json.v}}"
    state_on: true
    state_off: false
    optimistic: false
    retain: true

and the screenshot of my MQTT-Explorer:

[francisp (Francis)]

@schraepfert

retain: true

You realize that this retains the command, and not the state ?

[schraepfert]

The manual says:

In an ideal scenario, the MQTT device will have a state_topic to publish state changes. If these messages are published with a RETAIN flag, the MQTT switch will receive an instant state update after subscription, and will start with the correct state. Otherwise, the initial state of the switch will be unknown . A MQTT device can reset the current state to unknown using a None payload.

I think that’s what I want to achieve.
Should that have something to do with the problem? Since it seems to be a general problem in MQTT and also affects sensors that don’t have a retain flag to configure, I don’t think it has anything to do with it.

Or am I thinking wrong?

[francisp (Francis)]

It is your device that must set the retain flag on the MQTT state message.

For devices that don’t support that (and don’t update often), you can republish the MQTT message with the retained flag set in an automation.

[schraepfert]

Ah ok! That’s new to me, but makes sense.
So I don’t need a retain flag at all, since my device sets it itself?

But can this have something to do with the problem?

[francisp (Francis)]

To debug, open 2 browser windows

in one, go to settings, devices & services → mqtt, configure

under listen to a topic, enter

device/+/XXXXXXXXXX3CC4/14/STATE

Click on start listening

In the other window, toggle your lampe_tv

In the first window, you should see what HA receives from mqtt

[schraepfert]

Hi Francis,

sorry for the late reply. I had to take the system out of service again to do the update again. It does better in the evening.

Here is the output before the update:

And here after the update:
See my next post as I am only allowed to upload one image per post

After the update no status update comes back as described, which is why all further attempts want to turn off the switch, although it is already off.

All entities behave in the same way. Also my shellies (which are also sent to the Raspberrymatic CCU-Jack) do not update the sensors in the Home Assistant anymore.

Have a nice weekend!

[schraepfert]

And here the Image after the update:

[schraepfert]

I have now solved the problem by installing the Mosquitto addon in HomeAssistant and using the bridge function to synchronize the MQTT values of the CCU.
The HomeAssistant itself then no longer accesses the CCU to transmit the values, but the Mosquitto Broker Addon.

It seems that the problem is indirectly related to the CCU-Jack. When updating the HomeAssistant Core, perhaps a small thing in the MQTT protocol was changed, so that this can no longer interact with the CCU-Jack.

In any case, everything works now and I can update HomeAssistant again.

Thanks a lot for your help!

[mdzio]

Hello Home Assistant Community,

There was an error in the CCU jack when handling the retain flag. This has been fixed in version 2.9.1. A pre-release version can be downloaded here. Please test it!

Greetings
Mathias

Developer of CCU-Jack