ESPhome Sensors only work with API not MQTT

[Rubber9122]

Hi All,
I have a weird situation. I need to make a remote ESPhome device that will be able to communicate only through MQTT but when I only use MQTT most sensors are missing. However all sensors are working fine over API.
Discovery seems to work for only a few diagnostic sensors, but not for custom ones.
This is only with MQTT:

And if I add the device with ESPhome integration:
Some sensors are doubled that are coming from both sources.
And the MQTT side looks fine with all sensor/switch having a config entry for under homeassistant topic:

Here is my ESPhome config, hope that helps:

substitutions:
  name: "esptest1"
  friendly_name: esptest1

esphome:
  name: ${name}
  friendly_name: ${friendly_name}
  min_version: 2024.6.0
  name_add_mac_suffix: false
  project:
    name: esphome.web
    version: '1.0'

esp32:
  board: esp32dev
  framework:
    type: arduino

logger:
  level: VERBOSE

api:
  encryption:
    key: "yeqmICIZpD9Zwt5uhXMim/gt5j2bkoIGEis4oFdh6gA="

ota:
  - platform: esphome
    password: "572acc0dd73a83de15a7cf0502413d5c"

wifi:
  networks:
  - ssid: "ssid0"
    password: !secret wifi_pwd
  - ssid: "ssid1"
    password: !secret wifi_pwd

  ap:
    ssid: "ESPH-test"
    password: "test"

captive_portal:

mqtt:
  broker: IP
  port: 1883
  username: mqtt_esph_user
  password: !secret mqtt_esph_pwd

web_server:

http_request:
  verify_ssl: false

uart:
  rx_pin:
    number: GPIO16
  baud_rate: 115200
  rx_buffer_size: 3000

dsmr:
  crc_check: false
  max_telegram_length: 3000

sensor:
  - platform: dsmr
    energy_delivered_lux:
      name: Energy Consumed
  - platform: wifi_signal # Reports the WiFi signal strength/RSSI in dB
    name: "WiFi Signal dB"
    id: wifi_signal_db
    update_interval: 60s
    entity_category: "diagnostic"
  - platform: copy # Reports the WiFi signal strength in %
    source_id: wifi_signal_db
    name: "WiFi Signal Percent"
    filters:
      - lambda: return min(max(2 * (x + 100.0), 0.0), 100.0);
    unit_of_measurement: "Signal %"
    entity_category: "diagnostic"
    device_class: ""
  - platform: template
    name: "Module Version"
    id: module_version
    entity_category: "diagnostic"
    lambda: |-
      return 0;
  - platform: uptime
    type: seconds
    update_interval: 60s
    name: Uptime
    unit_of_measurement: "s"
    device_class: "duration"
    entity_category: "diagnostic"

text_sensor:
  - platform: dsmr
    identification:
      name: "DSMR Identification"
    p1_version:
      name: "DSMR Version"
    electricity_tariff:
      name: "Electric Tariff"
  - platform: version
    name: "ESPHome Version"
    hide_timestamp: true

switch:
  - platform: restart
    name: ESP restart

I have already tried multiple versions of ESPhome and HA

Thank for the help in advance!

[Rubber9122]

[Rubber9122]

Anyone have any ideas?

[Spiro]

Hmmm!

discovery: true

Try adding this in under MQTT section.
After you flash with mqtt version restart the esp device
Check with mqttexplorer if entities topics are showing up there in mqtt
Try reducing the yaml to minimum for mqtt then reintroduce items to see if its stopping the rest of the entities showing

[Rubber9122]

In the MQTT explorer the topics and entries seem fine, everything is there. Adding

discovery: true

did not change anything visibly on either side.

[Spiro]

If it won’t discover them automatically then you are proabably going to to have to create them in configuration.yaml I’ve done that with alot of mqtt sensors. It’s a bit more work

# Example configuration.yaml entry
mqtt:
  sensor:
    - name: "Bedroom Temperature"
      state_topic: "home/bedroom/temperature"

[Rubber9122]

Thank you for the suggestion, that got me on the right track.
While looking at creating the MQTT sensors I was looking at the unique_id argument for setting them up. I had a look at the MQTT side to see what are the uniq_ids in there and it hit me.
The unique id for the sensors I have checked were ESPsensorsolar_energy_day, ESPsensorpower_exported etc. Once checking the other MQTT ESP device I am running without issues I saw the same IDs.
So I have added:

  discovery_unique_id_generator: mac
  discovery_object_id_generator: device_name

to the MQTT section and the sensors immediately populated after a flashing.

I hope this helps someone in the future trying to deploy multiple ESP devices with same/similar config.