[MQTT] Send binary payload after 2024.8 update

[yellow_panther (Yanick)]

Hi all,

Since update 2024.8 I have the message that the payload_template parameter of MQTT publish will be deprecated in 2025.2.0.

In most cases it worked to replace payload_template by payload . However, there is one place where it changes the data and breaks the functionality, and that’s when I need to send a binary value to an MQTT to RS485 gateway that will poll a solar controller.

In this case, the data is sent as a string of bytes mixed with characters and hexadecimal sequences.

Here is the script used to reproduce the problem :

data_template:
  retain: true
  topic: test/sendbin
  payload: >
    {{ pack(0x0104, ">H") }}
    {{ pack(0x3100, ">H") }}
    {{ pack(0x0014, ">H") }}
    {{ pack(0xFEF9, ">H") }}
action: mqtt.publish

And here are the results in “MQTT Explorer”.

In the first case (payload) we receive a string of bytes (b’\x01\x041\x00\x14\xfe\xf9’), in the second case (payload_template) we receive a string of 8 characters (few of which are printable), which is what I want.

Anyone have any ideas on how I can fix this? Or do I need to open a bug report?
Maybe I’ve done it all wrong and there’s a better way?

Thanks.

[jchh ((not John))]

The payload one is right though, isn’t it?

[yellow_panther (Yanick)]

The payload I got with payload_template, yes, but not the big string of mixed chars and hex values.

I can live with payload_template for now, but given the deprecation I’ll have to change.

[jchh ((not John))]

@yellow_panther

The payload I got with payload_template

I honestly don’t know what you mean by this but, I had (I think) a similar issue whereby I was calling a script (that had mqtt.publish in it and I used the variable {{ payload }} for `payload, so it looked like this:

- action: mqtt.publish
  data:
    topic "..."
    payload_template "{{ payload }}"

This worked with payload_template but not with payload and some kind soul (thanks, @123) suggested changing the variable to something other than payload …and it worked!

so it now looks like:

- action: mqtt.publish
  data:
    topic "..."
    payload "{{ message }}"

Is this applicable to you?

[yellow_panther (Yanick)]

Thanks for your hint. Unfortunately I don’t use a variable in the payload, so the case is not quite the same.

I’ll try to be clearer.

In both cases the payload is the same:

    {{ pack(0x0104, ">H") }}
    {{ pack(0x3100, ">H") }}
    {{ pack(0x0014, ">H") }}
    {{ pack(0xFEF9, ">H") }}

If I use payload_template:, I get this MQTT message:

And if I use payload:, I get this:

In the first case, I have my 8 characters (4x2), but in the second I have 32 characters, which no longer corresponds to the payload at all.

[jchh ((not John))]

what should the result look like? (is the payload_template (below) correct?

I’m wondering whether you need a filter. Appreciating that you are not using a variable, one of the suggestions that @123 gave me was to use:
{{ desc[should] | to_json }}

would that correspond to:

    {{ pack(0x0104, ">H") | to_json }}
    {{ pack(0x3100, ">H") | to_json }}
    {{ pack(0x0014, ">H") | to_json }}
    {{ pack(0xFEF9, ">H") | to_json }}

I’m guessing that | to_json is incorrect for your use case, but perhaps there is another filter for you? Is the payload in a particular format?

…perhaps:

    {{ pack(0x0104, ">H") | to_json(ensure_ascii=False) }}
    {{ pack(0x3100, ">H") | to_json(ensure_ascii=False) }}
    {{ pack(0x0014, ">H") | to_json(ensure_ascii=False) }}
    {{ pack(0xFEF9, ">H") | to_json(ensure_ascii=False) }}

[yellow_panther (Yanick)]

The result should be the value 0x010431000014FEF9, which is represented in MQTT explorer by:

Unfortunately, JSON format is not an option, as the controller expects an 8-byte hex command. And I can’t modify it.

Thanks for the advice, I’ve tried it, including other solutions, but nothing conclusive so far.

I’ve opened a bug report:

Payload inconsistency in MQTT publish (payload_template deprecation)

We’ll see what happens.

[jchh ((not John))]

Someone way more capable than me may be able to chime in. Either way, best of luck!

[yellow_panther (Yanick)]

Well it looks like it was an issue, jbouwh made a PR to fix it.

Publish raw rendered mqtt payload as raw for mqtt publish action

I’ll wait until I can test it and then I’ll close this topic.

[yellow_panther (Yanick)]

Tested just now with version 2024.9.0, everything works fine