MQTT Discovery pending

mhermann · April 22, 2023, 8:14am

Hi,

I’m having trouble with Home Assistant not auto discovering my MQTT devices. I set up an external broker and the HA MQTT integration about a year ago and got the auto discovery to work correctly. I recently noticed, that the discovery doesn’t work properly anymore but can’t figure out why. No device or entity gets created when HA receives a discovery message.

Here’s my setup:

My Home Assistant runs on a Raspberry Pi 4, inside a docker container. On the same machine I run an MQTT broker (Mosquitto) inside another container. My device I’m trying to discover is an Arduino, which runs an MQTT client.

The Arduino publishes a discovery topic which shows up in MQTT Explorer as
homeassistant/light/ledStrip_C72044AA50573738352E3120FF111B36/config

{
  "~": "homeassistant/light/ledStrip_C72044AA50573738352E3120FF111B36",
  "name": "ledStrip_C72044AA50573738352E3120FF111B36",
  "uniq_id": "ledStrip_C72044AA50573738352E3120FF111B36",
  "cmd_t": "~/set",
  "stat_t": "~/state",
  "schema": "json",
  "brightness": true,
  "retain": true,
  "color_mode": true,
  "supported_color_modes": [
    "hs"
  ],
  "effect": "SlowBreathingWave",
  "fx_list": [
    "SlowBreathingWave",
    "ChristmasSparkle"
  ]
}

In Home Assistant I’ve added and configured the MQTT extension to connect to my broker.

As far as I can tell, Home assistant connects to the MQTT broker correctly. When I listen to the “homeassistant/light/ledStrip_C72044AA50573738352E3120FF111B36/config” topic in Home Assistant, I can see the messages.

This is what my configuration.yaml looks like:

# Configure a default setup of Home Assistant (frontend, api, etc)
default_config:

# Text to speech
tts:
  - platform: google_translate

group: !include groups.yaml
automation: !include automations.yaml
script: !include scripts.yaml
scene: !include scenes.yaml

logger:
  default: critical
  logs:
    homeassistant.components.mqtt: debug

#mqtt:
#  discovery: true
#  discovery_prefix: homeassistant

Note the MQTT section, which was previously required. I received deprecation warnings from HA, so I commented the lines out. Apparently those settings are now handled by the MQTT integration UI.

My Home Assistant logs show the following messages when I restart my Arduino to resend the discovery topic:

2023-04-22 09:30:41.339 DEBUG (MainThread) [homeassistant.components.mqtt.client] Received retained message on homeassistant/light/ledStrip_C72044AA50573738352E3120FF111B36/config (qos=0): b'{"~":"homeassistant/light/ledStrip_C72044AA50573738352E3120FF111B36","name":"ledStrip_C72044AA50573738352E3120FF111B36","uniq_id":"ledStrip_C72044AA50573738352E3120FF111B36","cmd_t":"~/set","stat_t":"~/state","schema":"json","brightness":true,"retain":true,"color_mode":true,"supported_color_modes":["hs"],"effect":"SlowBreathingWave","fx_list":["SlowBreathingWave","ChristmasSparkle"]}'
2023-04-22 09:30:41.340 DEBUG (MainThread) [homeassistant.components.mqtt.discovery] Process discovery payload {'name': 'ledStrip_C72044AA50573738352E3120FF111B36', 'unique_id': 'ledStrip_C72044AA50573738352E3120FF111B36', 'command_topic': 'homeassistant/light/ledStrip_C72044AA50573738352E3120FF111B36/set', 'state_topic': 'homeassistant/light/ledStrip_C72044AA50573738352E3120FF111B36/state', 'schema': 'json', 'brightness': True, 'retain': True, 'color_mode': True, 'supported_color_modes': ['hs'], 'effect': 'SlowBreathingWave', 'effect_list': ['SlowBreathingWave', 'ChristmasSparkle'], 'platform': 'mqtt'}
2023-04-22 09:30:41.340 INFO (MainThread) [homeassistant.components.mqtt.discovery] Found new component: light ledStrip_C72044AA50573738352E3120FF111B36
2023-04-22 09:30:41.358 DEBUG (MainThread) [homeassistant.components.mqtt.discovery] Pending discovery for ('light', 'ledStrip_C72044AA50573738352E3120FF111B36'): deque([])

So, it seems to me that Home Assistant sees the discovery topic, but ultimately never executes the discovery to create an entity.

I’ve updated Home Assistant (currently running 2023.4.6 Frontend 20230411.1 - latest), removed and reinstalled the MQTT integration and restarted HA several times.

Have I misconfigured something? Any ideas why the discovery doesn’t work? Feedback is much appreciated.

Thanks!

FloatingBoater · April 22, 2023, 3:48pm

Hi,

After writing a Python device using HASS Discovery, I found HASS changes configuration state almost immediately after I published the discovery topics to the broker. This made manual testing of the structure straightforward as the debug cycle was basically:

Delete the previous retained topics using MQTT Explorer
Publish new discovery data with mosquitto_pub on the command line
See what happened in HASS immediately
Cross-check what the broker actually published using https://mqtt-explorer.com/ and not your code!

I’ve not touched the code for a year, but managed to get units, icons, etc all working by trial and error with no YAML config at all - all sensors and actuators created via HASS Discovery.

Unlike Tasmota Discovery, HASS Discovery is baked-into the MQTT add-on, so there’s nothing to configure.

My best guess is therefore something has changed in the interpretation of your HASS Discovery topics.
My code was based on (pre-Tasmota Discovery) Tasmota HASS discovery using FULL topics, without ~, e.g.:

ChannelX mqtt_topic_state: `tele/grow-1/SENSOR` / `tele/grow-1/LWT`

homeassistant/sensor/grow-1_chan1/config = {
	"name": "glz-rpi-grow Channel 1", 
	"stat_t": "tele/glz-rpi-grow/SENSOR", 
	"val_tpl": "{{value_json.Channel1}}", 
	"avty_t": "tele/glz-rpi-grow/LWT", 
	"pl_avail": "Online", 
	"pl_not_avail": "Offline", 
	"uniq_id": "glz-rpi-grow_chan1", 
	"dev": {
		"ids": ["glz-rpi-grow"], 
		"name": "glz-rpi-grow", 
		"mf": "Pimoroni", 
		"mdl": "Grow", 
		"sw": "0.2", 
		"sa": "Conservatory"
	}, 
	"icon": "mdi:water-percent", 
	"dev_cla": "humidity", 
	"unit_of_meas": "%", 
	"expire_after": "1200", 
	"frc_upd": "true"
}

Tested MQTT command:

mosquitto_pub -d -h BROKER --username 'NAME' --pw "PW" -t 'homeassistant/sensor/grow-1_chan1/config' --retain -m '{"name":"Grow-1 Channel 1", "stat_t":"tele/grow-1/SENSOR", "val_tpl":"{{value_json.Channel1}}", "avty_t":"tele/grow-1/LWT", "pl_avail":"Online", "pl_not_avail":"Offline", "uniq_id":"grow-1_chan1", "dev": {"ids":["grow-1"], "name":"Grow 1", "mf":"Pimoroni", "mdl":"Grow", "sw":"0.2", "sa":"Conservatory"}, "icon":"mdi:sprout", "dev_cla":"humidity", "unit_of_meas":"%", "expire_after":"1200", "frc_upd": true}'

Example switch (driving an irrigation pump)

homeassistant/switch/grow-1_pump/config {
  "name":"glz-rpi-grow Water 1",
  "stat_t":"tele/glz-rpi-grow/PUMP/1",
  "avty_t":"tele/glz-rpi-grow/LWT",
  "pl_avail":"Online",
  "pl_not_avail":"Offline",
  "cmd_t":"cmnd/glz-rpi-grow/PUMP/1",
  "pl_off":"OFF",
  "pl_on":"ON",
  "val_tpl":"{{value_json.POWER}}",
  "uniq_id":"glz-rpi-grow_pump1",
  "dev": {
    "ids": ["glz-rpi-grow"], 
    "name": "glz-rpi-grow", 
    "mf": "Pimoroni", 
    "mdl": "Grow", 
    "sw": "0.2", 
    "sa": "Conservatory"
  }, 
  "icon": "mdi:watering-can", 
  "expire_after": "1200", 
  "frc_upd": "true"
}

I’d suggest a stare at the accurate, but not useful docs using the above examples!

If this helps, this post!

mhermann · April 22, 2023, 5:16pm

Your switch example gets discovered correctly. This just made me check my discovery topic. It seems like "schema": "json" is the issue in my topic. Removing that key makes the device/entity show up in HA. Unfortunately, the entity won’t work as intended without json support.

I’ve done some digging. It seems like the MQTT integration used to be able to handle json responses when the schema key was set accordingly. Now there seem to be a bunch of MQTT integrations, including MQTT_JSON.

However, that integration needs manual setup in the configuration.yaml, including a list of devices, which defeats the auto discovery.

So I don’t know how to proceed at this point. Can auto discovery be used with MQTT supporting json schema? I’d prefer if I didn’t have to revert to an old HA version to be able to use the discovery feature.

mhermann · April 22, 2023, 6:07pm

Hmm, I don’t know, this seems like a bug.

I can use the default MQTT integration and send a discovery topic with "schema": "basic", but the other two schemas (json and template) don’t seem to work.

I’m digging in the HA core source code and I’m seeing a lot of code concerning the json schema for MQTT lights and MQTT discovery. No idea why it doesn’t work.

FloatingBoater · April 22, 2023, 6:41pm

It’s been over a year since my head was in the code, and at the risk of sounding dumb (without seeing your topic payloads) , can you not use "val_tpl":"{{value_json.POWER}}" to set a template to parse and extract JSON values?

(Top Top - prototype in the Dev Tools template editor).

That there’s new options does suggest a different way… likely best to stare at the MQTT Python source in GitHub as the MQTT docs are more reference than how-to.

r_s_bell · April 22, 2023, 7:21pm

I am having alot of issues with my mqtt devices not showing in the auto discovery integration. They just disappeared last night for no apparent reason.

mhermann · April 22, 2023, 9:09pm

Ok, I think I’ve got it.

I enabled debug logs for everything via the configuration.yaml

logger:
  default: debug

Then I got error messages after the pending discovery. Apparently my discovery payload was faulty. I was using "effect": "SlowBreathingWave", assuming that’s the setting for the default effect. In actuality, the code is looking for a boolean, to check if effects should be enabled or disabled.

That’s not very intuitive, considering that "effect" is also used in the command topic as string when switching between effects. Though, I guess it is consistent with "brightness" and "color_mode", which are also booleans during the discovery.

So, the issue was simply wrong values in the payload. At this point I don’t know what the separate MQTT JSON integration is for. Overall, the whole MQTT discovery setup is not very intuitive or user friendly. I’m not sure though what the best way is, to improve the workflow. Maybe an MQTT payload verification tool in the “Developer Tools” section?

Anyway, this is my current discovery payload, which works again. Home Assistant creates a device and an entity.

{
  "~": "homeassistant/light/ledStrip_C72044AA50573738352E3120FF111B36",
  "uniq_id": "ledStrip_C72044AA50573738352E3120FF111B36",
  "name": "ledStrip_C72044AA50573738352E3120FF111B36",
  "cmd_t": "~/set",
  "stat_t": "~/state",
  "ret": true,
  "schema": "json",
  "dev": {
    "ids": [
      "C72044AA50573738352E3120FF111B36"
    ],
    "name": "Arduino NANO 33 IoT",
    "mdl": "Arduino NANO 33 IoT",
    "mf": "Arduino"
  },
  "brightness": true,
  "color_mode": true,
  "supported_color_modes": [
    "hs"
  ],
  "effect": true,
  "fx_list": [
    "SlowBreathingWave",
    "ChristmasSparkle"
  ]
}

Thank you for the help, James!

MQTT Discovery pending

ChannelX mqtt_topic_state: tele/grow-1/SENSOR / tele/grow-1/LWT

Example switch (driving an irrigation pump)

ChannelX mqtt_topic_state: `tele/grow-1/SENSOR` / `tele/grow-1/LWT`