Mqtt.py log warning "No matching payload found for entity"

kslb · March 26, 2019, 2:01pm

My working and tested solution (for alarming purposes)…

# ALARM - vklopi alarm DSC spodaj (RF ključ C)
- id: '30006'
  alias: 'ALARM - vklopi alarm DSC spodaj (RF ključ C)'
  hide_entity: true
  trigger:
    - platform: mqtt
      topic: "tele/Sonoff-RF-Bridge/RESULT"
    - platform: mqtt
      topic: "tele/Sonoff-RF-Bridge2/RESULT"
  condition:
    condition: and
    conditions:
    - condition: template
      value_template: "{{ trigger.payload_json.RfReceived.Data == '!RF-SECRET-RECEIVED-CODE-KEY-C' }}"
    - condition: state
      entity_id: alarm_control_panel.DSC5020
      state: disarmed
  action:
    service: alarm_control_panel.alarm_arm_away
    data:
      entity_id: alarm_control_panel.DSC5020
      code: !SECRET-DSC-ALARM-CODE

# ALARM - izklopi alarm DSC spodaj (RF ključ D)
- id: '30007'
  alias: 'ALARM - izklopi alarm DSC spodaj (RF ključ D)'
  hide_entity: true
  trigger:
    - platform: mqtt
      topic: "tele/Sonoff-RF-Bridge/RESULT"
    - platform: mqtt
      topic: "tele/Sonoff-RF-Bridge2/RESULT"
  condition:
    condition: or
    conditions:
    - condition: template
      value_template: "{{ trigger.payload_json.RfReceived.Data == '!RF-SECRET-RECEIVED-CODE-KEY-D' }}"
    - condition: state
      entity_id: alarm_control_panel.DSC5020
      state: armed_away
    - condition: state
      entity_id: alarm_control_panel.DSC5020
      state: armed_home
  action:
    service: alarm_control_panel.alarm_disarm
    data:
      entity_id: alarm_control_panel.DSC5020
      code: !SECRET-DSC-ALARM-CODE

kslb · March 26, 2019, 2:04pm

Before I had binary_sensor which was switched ON/OFF via payload ON/OFF…

123 · March 26, 2019, 2:43pm

I assume you are aware that using a 433MHz RF button to arm/disarm an alarm panel is not very secure. The RF codes are easily detectable and reproducible. You would need to (minimally) use a rolling-code system and, as far as I know, RF bridges based on stock and Tasmota firmware don’t support rolling-code.

If you are aware of the vulnerability and feel the risk is acceptable then all’s well. However, be advised that for a professionally-installed residential, especially commercial, alarm system, it would be unacceptable to use an unencrypted and unchanging RF signal to control it.

AhmadK · March 26, 2019, 11:22pm

Thanks for your solution.
I spent last 2 days playing with various ways of handling my PIRs, smoke detectors, door contacts, RF wall switches and remotes.

First I enabled adding code to each mqtt message and changed all my config to work with it.
Have to say it’s great for devices that only send one code (PIRs, smoke detectors) - you just need to define corresponding mqtt binary sensors.
For devices with two or more codes things get complicated as apart from creating mqtt switch/sensors you need automations to forward every command to a corresponding switch/sensor topic.

It worked, but I didn’t like it because of extra automations and need for plenty of !secret constants.

Then I tried your solution. Have to say in that form it’s ok for demonstration, but not suitable if you have a real setup even with a modest number of devices.
First of all, I believe it’s hard to maintain 3 separate but interlinked dicts if you have plenty of devices, many possibilities to make a mistake.
Second, in my setup (Hass.io @ RPi3 with Mosquitto add-on) mqtt messages take a while to appear and therefore reposting mqtt messages makes it even slower (not dramatically, but still).

I went for a proper dict that contains records with code, command and retain flag:

          'sensor/smoke_detector/1st_floor/landing': {'codes': {5997800: {'state': 'detected', 'command': 'ON'}}},
          'wall_switch/1st_floor/master_bedroom': {
            'codes': {
              10952785: {'state': 'on', 'command': 'ON'},
              10952784: {'state': 'off', 'command': 'OFF'}
            }
          }

That means I have to use nested for loops instead of simple lookup:

        {% set devices = {...}
        {% set code = trigger.payload_json.value | int %}
        {% set ns = namespace(topic='unknown') %}
        {% for device, definition in devices.items() %}
          {% for state_code, details in definition.codes.items() if state_code | int == code %}
            {% if loop.first %}
              {% set ns.topic = device %}
            {% endif %}
          {% endfor %}
        {% endfor %}
        home/{{ ns.topic }}

It might add up to slow response as well.
And my major complain is the size of dict AND the fact that I need to replicate it to each template (command, topic and retain). It doesn’t look great and I don’t like to copy it every time I add/change something in devices’ description.

Any thoughts on how it can be improved, or maybe a better solution?

Petrica · March 27, 2019, 6:19am

Migrate to NUC

MicroSD card will fail exactly when you needed it the most, anyway.

123 · March 27, 2019, 1:03pm

@AhmadK

When I started designing the ‘deluxe’ demultiplexer automation, I thought of using an ‘all-in-one’ dictionary where each value is an array containing topic, payload, and retain.

Example: x

{% set x = { 'A1A':['sensor1','ON','true'], 'A2A':['sensor1','OFF','true'], 'B1B':['sensor2','ON','false'], 'C1C':['sensor3','ON','false'] } %}

The problem is the scope of x is limited to within the option where it’s defined. That means, as you have also discovered, you have to define the dictionary three times: one for topic:, another for payload: and yet another for retain:.

Although Jinja2 supports macros, their scope is also limited to where they are defined. See this thread for more information: Jijnja2 macro in template

Given that I had to define the dictionary three times, I chose to make it compact. Each one of the three dictionaries only contains what’s needed for its associated option:

topic   { 'A1A':'sensor1', 'A2A':'sensor1', 'B1B':'sensor2', 'C1C':'sensor3' }
payload { 'A1A':'ON',      'A2A':'OFF',     'B1B':'ON',      'C1C':'ON' }
retain  { 'A1A':'true',    'A2A':'true',    'B1B':'false',   'C1C':'false' }

Yes, this makes it more challenging to construct because each command’s data is spread over three dictionaries. However, this is a ‘one-time’ editing exercise and the dictionary’s structure is simple (basic key:value pairs).

I agree interlinked dictionaries aren’t optimal but, seeing that we’re obliged to have three of them (due to scope limitations) it’s a way of keeping them compact. The alternative is to use the ‘all-in-one’ style of dictionary (shown above) and use three full copies (for topic, payload, and retain). Effectively, this is what you’ve done but with a more complex dictionary design.

Second, in my setup (Hass.io @ RPi3 with Mosquitto add-on) mqtt messages take a while to appear and therefore reposting mqtt messages makes it even slower (not dramatically, but still).

I have not noticed a delay on my system. I’m running Home Assistant on an old laptop and the Mosquitto broker runs on a Raspberry Pi 3B. They’re separate hosts yet MQTT communications are virtually instantaneous.

Any thoughts on how it can be improved, or maybe a better solution?

If you don’t like the demultiplexer solution, there’s always the other solution I offered: create a value_template for each sensor that always returns a valid state.

klogg · March 27, 2019, 4:17pm

Can I just add a further complication? (For fun?)
I have more than one RF Bridge* so my automations have to have different dictionaries which made it a lot harder to construct without errors. I’m not complaining I like the result but I leave it as an exercise for you if you’d like to consider a ‘super-deluxe’ method.

*I have three actually, which I now think is overkill but I was having some problems with dead spots causing some sensors to be unreliable no matter where I placed the Bridges. I still occasionally get sensors stuck in the ON position so I wonder if it is just cheap sensors not performing properly or something else going on^. I am however not sure that the third Bridge was necessary. But even so, because of the layout of my house I do need two.

^Comments welcome if this could be something I am overlooking in the processing. Thinking, as I write this, I am wondering if two sensors firing simultaneously could cause one to not be registered by HA in time?

123 · March 27, 2019, 4:30pm

I’d just keep things simple and use three independent demultiplexer automations (i.e. one per RF bridge).

Don’t forget that the root cause of this ‘fun’ is the fact the RF bridge publishes everything to one topic. There aren’t too many ways to handle this arrangement. Either you have all sensors subscribed to the RF bridge’s single topic (or in your case to one of three different RF bridges) or one automation that creates multiple topics, one per sensor. Both solutions have pros and cons and it’s up to the user to ‘pick their poison’.

There’s no escaping the ‘lots of editing’ task. It’s either going to be in creating big dictionaries for the automation or in creating templates for every sensor. As the saying goes: Six of one, half a dozen of the other.

Petrica · March 27, 2019, 6:24pm

Can you mention what RF bridge do you use?

If you’re not already using it, you might want to have a look at OpenMQTT Gateway as, for multiple RF bridges connected to same topic, the de-duplication is done at bridge level (so no need for multiple topics to be dealt with by HA); RF signal is picked up by the closest bridge and case the other bridges pick it up too, then it won’t be published if it occurs in a 2 to 3 seconds interval after initial publish. But again, this requires bridges to subscribe to same MQTT topic but to have different board names.

sentur · March 27, 2019, 7:53pm

On this topic. Is that the main difference / benefit of using Tasmota flashed Sonoffs RF Bridge Vs a OpenMQTTGateway flashed Sonoffs RF bridge?

AhmadK · March 27, 2019, 8:24pm

Do you know if it’s possible to create a python_script which does the job of preparing all three values AND then publishes a message?

123 · March 27, 2019, 8:35pm

Have I done it? No.

But it might be fun to try. (… and I leave it as an exercise for someone else)

Based on the example shown in the documentation, I expect it would provide more flexibility.

klogg · March 27, 2019, 9:33pm

Which is what I’ve done, and as I said I am happy with the results and I don’t expect to have to edit the three automations again (ha ha!!!) so now it is done I can forget it…

(I was just fishing in case there was a super-deluxe option. I didn’t expect it though.)

They are Sonoff RF Bridges flashed with Tasmota. I was, once upon a time bridging with Open MQTT Gateway to use Owntracks but I have long since closed that open port and moved away from having any open. (I can use VPN now if I need remote access).

123 · March 28, 2019, 1:39am

I couldn’t resist experimenting with python_script.

It offers several advantages:

It’s far simpler than using templates.
There’s only one instance of the ‘command dictionary’.
The dictionary is easier to maintain.
Because it’s a python_script, you can add items to the dictionary without having to restart Home Assistant.

To make this work, you have to configure Home Assistant to use the python_script component.

Add python_script: to your configuration file.
Create the following sub-directory: config/python_scripts
Restart Home Assistant.

Create a new file in config/python_scripts called rfbridge_demux.py with the following contents:

d = { 'A1A':['sensor1','ON','true'],
      'A2A':['sensor1','OFF','true'],
      'B1B':['sensor2','ON','false'],
      'C1C':['sensor3','ON','false']
    }

p = data.get('payload')

if p is not None:
  if p in d.keys():
    service_data = {'topic':'home/{}'.format(d[p][0]), 'payload':'{}'.format(d[p][1]), 'qos':0, 'retain':'{}'.format(d[p][2])}
  else:
    service_data = {'topic':'home/unknown', 'payload':'{}'.format(p), 'qos':0, 'retain':'false'}
    logger.warning('<rfbridge_demux> Received unknown RF command: {}'.format(p))
  hass.services.call('mqtt', 'publish', service_data, False)

Obviously you will have to modify the dictionary so it contains your RF bridge’s commands. The format of each item in the dictionary is straightforward:
'payload':['topic','state','retain']

Here’s the required automation. It’s much simpler than the previous version using templates.

- alias: 'rfbridge_demultiplexer'
  trigger:
  - platform: mqtt
    topic: tele/RF_Bridge/RESULT
  action:
  - service: python_script.rfbridge_demux
    data_template:
      payload: '{{trigger.payload_json.RfReceived.Data}}'

I tested it and can confirm it works. Feel free to modify rfbridge_demux.py to suit your specific needs.

AhmadK · March 28, 2019, 5:31am

And you made my day! Thanks man. Great stuff, will think about it when I wake up properly
Yesterday I spent re-thinking my configuration and came up with another solution.
In my case (PIRs/smoke detectors, 3-state door contacts) it is not that reasonable to have separate MQTT topics as only multi-code devices struggle from that “No matching payload” warning (yeah, we are still on this topic) and it can be easily avoided by defining a proper value_template/payload_xxx combination.
So I took a combined approach based on some demuxing - currently I have 1 per code type per device type (meaning one for PIRs, one for smoke detectors and 3 for 3-state door contacts). It’s compact and relatively easy to maintain when you get an idea…

I think we can stop spamming this topic with it and I’m going to create a separate one in Share your project section to describe my approach so others can find it easier and discuss/improve etc.

Anyway, you did a fantastic job here and I think the topic problem is solved now with plenty of options available.

123 · March 28, 2019, 10:45am

Tag the post as ‘solution’ so others can find it easily in this long thread.

AhmadK · March 28, 2019, 11:57am

I’d love to, but I can see 2 problems here

You gave us MORE than one solution and and it’s up to a reader to decide which one works best for them.
I can see no “Solution” checkbox - think you have one if you are a topic starter but I’m not

123 · March 28, 2019, 12:21pm

Now that you mention it, neither do I. You’re right, only the threadstarter can do that. Oh well, people will just have to wade through the thread.

Mosl · March 29, 2019, 2:25am

I’ve removed the code, but HA will still issue the same warning

Thomas_Brady · April 7, 2019, 9:57am

This post reminds me this is something I need to fix I have 34 of these and growing so much log spam!