MQTT discovery works but leaves entities deactivated

[ulihuber]

After some research, try and error, I managed to program a ESP32 based sensor that appears over MQTT as device with multiple entities.
The JSON discovery data (i.e. for one entity) is as follows:

{
  "name": "Batterie",
  "uniq_id": "Klima_Ulrich_Batterie",
  "state_topic": "Klima/Ulrich/state",
  "unit_of_measurement": "V",
  "device_class": "voltage",
  "force_update": true,
  "enabled_by_default": true,
  "value_template": "{{ value_json.batt}}",
  "device": {
    "manufacturer": "UHU",
    "model": "Klima32_5_C6.ino Feb  9 2025  17:36:49",
    "name": "AHT_Ulrich",
    "identifiers": [
      "AHT_Ulrich"
    ]
  }
}

The sensor data is

{
  "batt": "2.9947",
  "temp": "0.00",
  "humid": "0.00",
  "quality": "2",
  "IP": "192.168.11.124",
  "SSID": "Huber1",
  "RSSI": "-61",
  "wakeups": "1",
  "lowbat": "false",
  "errCode": "Kein Fehler"
}

This works fine on my testsystem where I get my entities activated straight away.
On my older and much more populated ‘production’ systems I get my entities as well, but they are all deactivated. I can, however, activate them manually and they seem to work fine if I do so.

I have no idea what blocked the discovery process from activating the entities.
ChatGPT suggested adding “enabled_by_default”: true, but this did not change anything, nor did it explain why it works on my test system without this line.

Same with suggested adding following lines to mqtt.yaml.

discovery: true
discovery_prefix: homeassistant

First, this is not present in my test system, Second this did not work out as I get

Invalid config for 'mqtt' at configuration.yam, Line 38: extra keys not allowed @ data['mqtt'][0]['discovery'].

Any sugestions are highly appreciated

Uli

[nickh66 (Nick Harrison)]

Hi,
I can’t say i have seen this before but i had a thought that you could test. Firstly, to clarify, you have to HA systems talking to the same MQTT broker - is that correct?
My thought is this. What happens if your main system (the one not behaving the way you want) is the first one to complete the discovery process. In that scenario does the test system that goes second in this test still behave normally?
Cheers
Nick

[ulihuber]

@nickh66

HA systems talking to the same MQTT broker - is that correct?

No. these are completely different, closed systems, running on separate hardware. Production on VirtualBox on my server, Test under Proxmox on an Intel NUC. They do not share more than the local network.

And I think, there should be no problem with the MQTT broker, as I get these entities perfectly generated in HA. They are just not activated.

[ShadowFist]

@ulihuber

They are just not activated

Not sure if it will fix your issue, but uniq_id should definitely be unique_id

[francisp (Francis)]

@ulihuber

discovery: true
discovery_prefix: homeassistant

Where did you find this in the documentation ? Discovery is set in the mqtt integration settings

[ulihuber]

@francisp

Discovery is set in the mqtt integration settings

You are right. Sorry. I did not find these lines in the documentation but somewhere in an googled thread. Because I had already set the switch in MQTT integration, I assumed there could be a second place where this has to be activated. So, as the swich is set, there has to be another reason.

[Sir_Goodenough ((SG) WhatAreWeFixing.Today)]

there are shortcuts for everything in MQTT Discovery, that one is correct… @ShadowFist

MQTT

Also there is another thread today about the same subject…

Entities Not Being Enabled After MQTT Discovery

Hi everyone, I’m using MQTT discovery, and while it detects the devices, many of the entities remain deactivated or don’t show up in the UI. It seems like they’re recognized but not fully integrated. Has anyone dealt with this before? Any tips on getting them activated properly? Thanks!

Perhaps comparing Notes are in order… @ulihuber

Also perhaps setting

'ent_cat':             'entity_category',

might help. Perhaps it thinks it’s diagnostic and not enabling it.

[ulihuber]

I enabled log level ‘debug’ for MQTT and found the following odds:

Test system:

2025-02-11 16:49:30.035 DEBUG (MainThread) [homeassistant.components.mqtt.client] Received message on homeassistant/sensor/Klima_Test123/Batterie/config (qos=0): b'{"name":"Batterie","unique_id":"Klima_Test123_Batterie","state_topic":"Klima/Test123/state","unit_of_measurement":"V","device_class":"voltage","force_update":true,"enabled_by_default":true,"value_template":"{{ value_json.batt}}","device":{"manufacturer":"UHU","model":"Klima32_5_C6.ino Feb 11 2025  16:48:44","name":"AHT_Test123","identifiers":["AHT_Test123"]}}'
2025-02-11 16:49:30.038 DEBUG (MainThread) [homeassistant.components.mqtt.discovery] Process component discovery payload {'unique_id': 'Klima_Test123_Batterie', 'state_topic': 'Klima/Test123/state', 'unit_of_measurement': 'V', 'device_class': 'voltage', 'force_update': True, 'enabled_by_default': True, 'value_template': '{{ value_json.batt}}', 'device': {'manufacturer': 'UHU', 'model': 'Klima32_5_C6.ino Feb 11 2025  16:48:44', 'identifiers': ['AHT_Test123'], 'name': 'AHT_Test123'}, 'name': 'Batterie'}
2025-02-11 16:49:30.060 INFO (MainThread) [homeassistant.components.sensor] Setting up mqtt.sensor
2025-02-11 16:49:30.061 INFO (MainThread) [homeassistant.components.mqtt.discovery] Found new component: sensor Klima_Test123 Batterie
2025-02-11 16:49:30.071 INFO (MainThread) [homeassistant.helpers.entity_registry] Registered new sensor.mqtt entity: sensor.aht_test123_batterie
2025-02-11 16:49:30.075 DEBUG (MainThread) [homeassistant.components.mqtt.discovery] Pending discovery for ('sensor', 'Klima_Test123 Batterie'): deque([])
2025-02-11 16:49:30.175 DEBUG (MainThread) [homeassistant.components.mqtt.client] 192.168.11.200: register write 38
2025-02-11 16:49:30.176 DEBUG (MainThread) [homeassistant.components.mqtt.client] Subscribing with mid: 67 to topics with qos: [('Klima/Test123/state', 0)]
2025-02-11 16:49:30.181 DEBUG (MainThread) [homeassistant.components.mqtt.client] 192.168.11.200: unregister write 38

Production system:

2025-02-11 16:21:10.982 DEBUG (MainThread) [homeassistant.components.mqtt.client] Received retained message on homeassistant/sensor/Klima_Test123/Batterie/config (qos=0): b'{"name":"Batterie","unique_id":"Klima_Test123_Batterie","state_topic":"Klima/Test123/state","unit_of_measurement":"V","device_class":"voltage","force_update":true,"enabled_by_default":true,"value_template":"{{ value_json.batt}}","device":{"manufacturer":"UHU","model":"Klima32_5_C6.ino Feb 11 2025  16:09:04","name":"AHT_Test123","identifiers":["AHT_Test123"]}}'
2025-02-11 16:21:10.983 DEBUG (MainThread) [homeassistant.components.mqtt.discovery] Process component discovery payload {'unique_id': 'Klima_Test123_Batterie', 'state_topic': 'Klima/Test123/state', 'unit_of_measurement': 'V', 'device_class': 'voltage', 'force_update': True, 'enabled_by_default': True, 'value_template': '{{ value_json.batt}}', 'device': {'manufacturer': 'UHU', 'model': 'Klima32_5_C6.ino Feb 11 2025  16:09:04', 'identifiers': ['AHT_Test123'], 'name': 'AHT_Test123'}, 'name': 'Batterie'}
2025-02-11 16:21:10.983 INFO (MainThread) [homeassistant.components.mqtt.discovery] Found new component: sensor Klima_Test123 Batterie
2025-02-11 16:21:10.985 DEBUG (MainThread) [homeassistant.components.mqtt.discovery] Pending discovery for ('sensor', 'Klima_Test123 Batterie'): deque([])

The msg “Setting up mqtt.sensor” is missing and the discovery stalls. Honestly I expected more information from a level “debug”. But that’s what I got.

Now I removed for a test all ‘manual’ MQTT definitions from mqtt.yaml, but the problem remains.
But I found something interresting: On my test system, the device is being removed from HA when I delete the discovery packet from the MQTT broker. This does not happen on the produktion system.

[ulihuber]

More debug information:

2025-02-11 21:49:44.804 DEBUG (MainThread) [paho.mqtt.client] Received PUBLISH (d0, q0, r0, m0), 'homeassistant/sensor/Klima_Test123/Batterie/config', ...  (361 bytes)
2025-02-11 21:49:44.804 DEBUG (MainThread) [homeassistant.components.mqtt.client] Received message on homeassistant/sensor/Klima_Test123/Batterie/config (qos=0): b'{"name":"Batterie","unique_id":"Klima_Test123_Batterie","state_topic":"Klima/Test123/state","unit_of_measurement":"V","device_class":"voltage","force_update":true,"enabled_by_default":true,"value_template":"{{ value_json.batt}}","device":{"manufacturer":"UHU","model":"Klima32_5_C6.ino Feb 11 2025  21:49:03","name":"AHT_Test123","identifiers":["AHT_Test123"]}}'
2025-02-11 21:49:44.804 DEBUG (MainThread) [homeassistant.components.mqtt.discovery] Process component discovery payload {'unique_id': 'Klima_Test123_Batterie', 'state_topic': 'Klima/Test123/state', 'unit_of_measurement': 'V', 'device_class': 'voltage', 'force_update': True, 'enabled_by_default': True, 'value_template': '{{ value_json.batt}}', 'device': {'manufacturer': 'UHU', 'model': 'Klima32_5_C6.ino Feb 11 2025  21:49:03', 'identifiers': ['AHT_Test123'], 'name': 'AHT_Test123'}, 'name': 'Batterie'}
2025-02-11 21:49:44.804 INFO (MainThread) [homeassistant.components.mqtt.discovery] Found new component: sensor Klima_Test123 Batterie
2025-02-11 21:49:44.806 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event device_registry_updated[L]: action=create, device_id=ceb3ac230aae3a1d0cbbce5b0c5a3376>
2025-02-11 21:49:44.806 INFO (MainThread) [homeassistant.helpers.entity_registry] Registered new sensor.mqtt entity: sensor.aht_test123_batterie
2025-02-11 21:49:44.806 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event entity_registry_updated[L]: action=create, entity_id=sensor.aht_test123_batterie>
2025-02-11 21:49:44.806 DEBUG (MainThread) [homeassistant.components.sensor] Not adding entity Batterie because it's disabled
2025-02-11 21:49:44.806 DEBUG (MainThread) [homeassistant.components.mqtt.discovery] Pending discovery for ('sensor', 'Klima_Test123 Batterie'): deque([])

So all comes down to “Not adding entity Batterie because it’s disabled”.
But where? There is no other entity ith this name. And same happens to all 10 entities from this device.

[ulihuber]

I asked ChatGPT and the advice was to look into /config/.storage/core.entity_registry to find lines with the entity-id and disabled_by: user or disabled_by: integration. Deleting these lines or editing the attribute would help.
Just for others having the same problem: It didn’t…
Nor did deleting the corresponding lines in /config/.storage/core.device_registry

Another test was invoking a 2 second delay after each discovery packet. No change, still deactivated after discovery.

Uli

[ulihuber]

Hi,
I don’t know if anybody is still following this journey.
Another idea was to check disabled entities :
grep “disabled_by” /config/.storage/core.entity_registry
This leds to :

{"aliases":[],"area_id":null,"categories":{},"capabilities":null,"config_entry_id":"3cfbe28cab7b7d66dc88da1efd513856","created_at":"2025-02-10T22:40:58.545486+00:00","device_class":null,"device_id":"ba02ba3d9e29fd0494749256936ce17c","disabled_by":"integration","entity_category":null,"entity_id":"sensor.aht_test123_batterie","hidden_by":null,"icon":null,"id":"32608015ca61087338d1fdafcd2cebc4","has_entity_name":true,"labels":[],"modified_at":"2025-02-12T15:28:23.816800+00:00","name":null,"options":{},"original_device_class":"voltage","original_icon":null,"original_name":"Batterie","platform":"mqtt","supported_features":0,"translation_key":null,"unique_id":"Klima_Test123_Batterie","previous_unique_id":null,"unit_of_measurement":"V"},

The question is now, what reason could result in “disabled_by”:“integration”.

[ulihuber]

HEUREKA - solved :-))

/config/.storage/core.config_entries had the folowing line:

{"created_at":"1970-01-01T00:00:00+00:00","data":{"broker":"192.168.11.105","password":"uhu123","port":1883,"username":"uhu"},"disabled_by":null,"discovery_keys":{},"domain":"mqtt","entry_id":"3cfbe28cab7b7d66dc88da1efd513856","minor_version":2,"modified_at":"2025-02-06T21:55:18.657238+00:00","options":{"birth_message":{"payload":"online","qos":0,"retain":false,"topic":"homeassistant/status"},"discovery":true,"discovery_prefix":"homeassistant","will_message":{"payload":"offline","qos":0,"retain":false,"topic":"homeassistant/status"}},**"pref_disable_new_entities":true**,"pref_disable_polling":false,"source":"hassio","title":"192.168.11.105","unique_id":null,"version":1},

Changed to “pref_disable_new_entities”:true and discovery works fine with enabled entities.

Has anyone knowledge of where this could be activated without digging deep into the configuration file?
Uli

[123 (Taras)]

Is this the option you’re looking for?

Settings
→ Devices & Services
→ Integrations
→ MQTT
→ Configure
→ Configure MQTT Options
→ Enable discovery

[ulihuber]

Is this the option you’re looking for?

No. Enable discovery was always set.
This is another switch and I wonder if it is handled somewhere in the configuration frontend.

[123 (Taras)]

@ulihuber

wonder if it is handled somewhere in the configuration frontend

If you examine the menu I suggested above, you’ll see there’s no such option.

Whatever enabled that internal option was probably done programmatically.

[ulihuber]

Whatever enabled that internal option was probably done programmatically.

The code on Github is huge and not easy to dig in. What I found is that assignment of “true” to “pref_disable_new_entities” is mostly (only) in context of “test”. Maybe it’s something experimental and got activated by some add-on.