MQTT sensor returns "Unknown" despite topic and json data present in the broker - was working before the data source was rebooted

Hi folks,

Still a bit of a newbie to Home Assistant and MQTT, but I had a number of sensors defined that were working OK up until I had to reboot the Pi that was feeding them in.

After the reboot of the Pi, all those sensors are stuck at “Unknown” despite the MQTT topic being published OK and I can see the data in MQTT Explorer, and the python script on the Pi that generates the data says it’s connected OK and is publishing.

I’ve obfuscated the MQTT user name and password in the sections below.

I’m running supervised HA on a Pi4, with the Mosquitto adding and integration both loaded and running.

Mosquitto broker is up and running and shows the Pi connecting OK

1611238861: New client connected from 192.168.1.100 as raspi-00000000b1b6694d (p2, c1, k60, u'mqttusernameremoved').

MQTT DATA showing in MQTT Explorer connected to HA

washingmachine = Idle
EnviroMonitor = {"Gas_Calibrated": false, "Bar": [987.2, "0"], "Hum": [34.9, "1"], "P2.5": 1, "P10": 1, "P1": 0, "Temp": 24.5, "Min Temp": 24.2, "Max Temp": 24.8, "Red": 0.09, "Oxi": 0.13, "NH3": 0.01, "Lux": 12.2}

Extract from SENSOR.YAML

# Enviromnent monitoring sensors from indoor monitoring station. 
# These were working, but I had to reboot the Pi, and now all
# are showing "Unknown" in HA, despite the MQTT topic being present
# when I check in MQTT Explorer
#
  - platform: mqtt
    state_topic: "EnviroMonitor"
    value_template: "{{ value_json.P1 }}"
    name: "enviro_pm1"
    unit_of_measurement: 'µg/m3'
    icon: "mdi:thought-bubble-outline"
  - platform: mqtt
    state_topic: "EnviroMonitor"
    value_template: "{{ value_json.P10 }}"
    name: "enviro_pm10"
    unit_of_measurement: 'µg/m3'
    icon: "mdi:thought-bubble-outline"
  - platform: mqtt
    state_topic: "EnviroMonitor"
    value_template: "{{ value_json['P2.5'] }}"
    name: "enviro_pm2"
    unit_of_measurement: 'µg/m3'
    icon: "mdi:thought-bubble"
  - platform: mqtt
    state_topic: "EnviroMonitor"
    value_template: "{{ value_json['Hum'][0] }}"
    name: "enviro_humidity"
    unit_of_measurement: '%'
    icon: "mdi:water-percent"
  - platform: mqtt
    state_topic: "EnviroMonitor"
    value_template: "{{ value_json['Bar'][0] }}"
    name: "enviro_pressure"
    unit_of_measurement: 'Bar'
    icon: "mdi:arrow-down-bold"
  - platform: mqtt
    state_topic: "EnviroMonitor"
    value_template: "{{ value_json.Temp }}"
    name: "enviro_temperature"
    unit_of_measurement: 'C'
    icon: "mdi:thermometer"
  - platform: mqtt
    state_topic: "EnviroMonitor"
    value_template: "{{ value_json.Lux }}"
    name: "enviro_lux"
    unit_of_measurement: 'lx'
    icon: "mdi:weather-sunny"
  - platform: mqtt
    state_topic: "EnviroMonitor"
    value_template: "{{ value_json.NH3 }}"
    name: "enviro_nh3"
    unit_of_measurement: 'ppm'
    icon: "mdi:thought-bubble"
  - platform: mqtt
    state_topic: "EnviroMonitor"
    value_template: "{{ value_json.Red }}"
    name: "enviro_reduced"
    unit_of_measurement: 'ppm'
    icon: "mdi:thought-bubble"
  - platform: mqtt
    state_topic: "EnviroMonitor"
    value_template: "{{ value_json.Oxi }}"
    name: "enviro_oxidised"
    unit_of_measurement: 'ppm'
    icon: "mdi:thought-bubble"

# Washing Machine status (calculated in Node-RED and thrown out to MQTT), this
# is just the msg.payload from the calculations based on Amps used by washer. 
# Works fine, sensor is populated with the text.
#
  - platform: mqtt
    state_topic: "washingmachine"
    name: "Washing Machine Status"
    icon: "mdi:washing-machine"

The MQTT Integration is loaded in HA and shows that it’s connected to “Mosquitto Core” as well as the MQTT Broker is showing in supervisor. Mosquitto Broker version is 5.1

Here’s my Mosquitto config from the add-on in Supervisor. (password and username removed)

logins:
  - username: mqttusernameremoved
    password: mqttpasswordremoved
anonymous: false
customize:
  active: true
  folder: mosquitto
certfile: fullchain.pem
keyfile: privkey.pem
require_certificate: false

And “Discovery” is enabled in the MQTT integration.

Frankly, I’m utterly stumped. I’ve tried rebooting the Pi, rebooting HA and the sensors in HA stay resolutely at “Unknown”. Can’t see anything in the Supervisor log or Core log (under system in HA) that says there’s an MQTT error; and most weirdly the washing machine MQTT sensors is fine; which implies that HA is able to pickup data out of MQTT.

Also, just to check that something running on the HA Pi can see the relevant MQTT Topic, if I use an MQTT-in node in Node-Red and dump it to the Debug window, it shows the MQTT data is present as well in the relevant topic.

I’ve had a look through several of the MQTT topics here, but they don’t seem to match up with this issue where MQTT sensors were working, but now are stuck “Unknown” after a reboot of the data source.

(Also weirdly, I’m also running OpenZwave on HA as I have some Zwave devices, and I used to see a topic “OpenZWave” in MQTT Explorer, but I don’t see that anymore. Zwave devices are working fine.)

Anyone got any advice?

Many thanks in advance!

Dave

If you close the MQTT Explorer session and then reconnect it are the payloads now gone ? If so they are not ‘retained’ on the broker

Thanks xAPPO

When I close MQTT explorer add re-open it, the payloads for “EnviroMonitor” and “washingmachine” are gone, but they come back in again quickly when they get re-published from their sources.

The “washing machine” sensor for example still shows the data from the MQTT payload and doesn’t show “Unknown”. It’s only the environment ones that are stuck at “Unknown”

They weren’t set previously to be “retained” and the data was being populated into all the MQTT sensors OK.

The fact they are missing shows they weren’t retained but as you say if they do update frequently then that should update HA fine when they do.

… and your clients are all using different clientID’s and QOS1 (or 0)?

Yep, the clients have different client IDs.

I shutdown HA from Supervisor > System so I could grab the startup details from the Supervisor log in case that adds anything at all?

21-01-21 16:16:07 INFO (SyncWorker_1) [supervisor.docker.addon] Starting Docker add-on homeassistant/aarch64-addon-mosquitto with version 5.1
21-01-21 16:16:09 INFO (MainThread) [supervisor.services.modules.mqtt] Set core_mosquitto as service provider for mqtt

and here’s the MQTT log from the Mosquitto add-on (again, MQTT username has been obfuscated), you can see everything has it’s own separate Client IDs

[16:16:09] INFO: Setup mosquitto configuration
[16:16:09] WARNING: SSL not enabled - No valid certs found!
[16:16:09] INFO: Found local users inside config
[16:16:09] INFO: Initialize Hass.io Add-on services
[16:16:09] INFO: Initialize Home Assistant discovery
[16:16:09] INFO: Start Mosquitto daemon
1611245769: Loading config file /share/mosquitto/acl.conf
1611245769: mosquitto version 1.6.3 starting
1611245769: Config loaded from /etc/mosquitto.conf.
1611245769: Loading plugin: /usr/share/mosquitto/auth-plug.so
1611245769:  ├── Username/password checking enabled.
1611245769:  ├── TLS-PSK checking enabled.
1611245769:  └── Extended authentication not enabled.
1611245769: |-- *** auth-plug: startup
1611245769: Opening ipv4 listen socket on port 1883.
1611245769: Opening ipv6 listen socket on port 1883.
1611245769: Opening websockets listen socket on port 1884.
1611245769: Warning: Mosquitto should not be run as root/administrator.
1611245972: New connection from 172.30.33.3 on port 1883.
[INFO] found addons on local database
1611245973: New client connected from 172.30.33.3 as mosq-Fg2zHw5AT2LpwjiIC1 (p2, c1, k60, u'addons').
1611245975: Client mosq-Fg2zHw5AT2LpwjiIC1 disconnected.
1611245977: New connection from 172.30.33.3 on port 1883.
1611245977: New bridge connected from 172.30.33.3 as zwave (p2, c1, k60, u'addons').
1611245981: New connection from 192.168.1.100 on port 1883.
[INFO] found mqttusernameremoved on local database
1611245983: New client connected from 192.168.1.100 as raspi-00000000b1b6694d (p2, c1, k60, u'mqttuser').
1611245997: New connection from 172.30.32.1 on port 1883.
[INFO] found homeassistant on local database
1611245997: New client connected from 172.30.32.1 as 4Vg3hKsqp1E0VBcfu0H4w1 (p2, c1, k60, u'homeassistant').
1611246095: New connection from 172.30.32.1 on port 1883.
1611246095: New client connected from 172.30.32.1 as mqtt_24cfedc.818ed12 (p2, c1, k60, u'mqttusernameremoved').
1611246189: New connection from 192.168.1.238 on port 1883.
1611246189: New client connected from 192.168.1.238 as mqtt-explorer-2080f3b7 (p2, c1, k60, u'mqttusernameremoved').

And now to make it even weirder, the Washing Machine MQTT sensor is stuck as “Unknown” as well, despite MQTT Explorer showing the payloads for both “washingmachine” and “EnviroMonitor” being present; and theres’ still no Zwave payloads which used to be there too. :open_mouth:

I see a bridge connection - from a Docker container. That’s always been there has it … and worked ? I use external mosquito broker so a little unfamiliar with the Docker container approach and why it’s bridged (between brokers I assume) or is that ZWave/Zigbee > 2mqtt

Can you think of anything that changed recently ?
I’m sure Taras will be along with some advice soon

I have no idea where that Bridge is from. I’m guessing it’s OpenZwave; but yes, it was working before as far as I can tell anyway.

Just tested, and it looks like the problem is the MQTT Integration has “gone deaf”

  • If I listen to # in Node-Red on an MQTT-In node, and dump it to debug, I see the EnviroMonitor and washingmachine topics/payloads
  • If I listen to # from the MQTT Integration in HA, I get no results.

I’m wondering if the MQTT HA integration is broken? (Configuration > Integrations)

Those last two comments above do point to HA being deaf to the discovery MQTT subscribes it makes

That’s what I was wondering. I’ve only just tried that, hence not mentioning it previously.

Also the only recent change relating to MQTT is that the OpenZwave supervised add-on got updated from 0.7.1 to 0.8.0, so I’m wondering if that’s why the OpenZwave MQTT topic/payload has vanished? Zwave devices are still all working fine though.

Thanks for all your help so far xAPPO

I think correctly or incorrectly you may now have two MQTT brokers following the openZWave integration update and maybe your device publishes are to a different broker than HA is now discovering from… HA is working with ZWave you say which it couldn’t without a ZWave topic. The brokers do seem bridged though.

I don’t use openZWave or the HA MQTT provided brokers so I’m probably no further use I’m afraid. I’ll watch out for the solution with interest though.

Good luck

Thanks!

I can only see that I using one broker from the logs though?

  • the Node-Red MQTT-In node is connecting to the Mosquitto broker,
  • the RasPi running the enviro stuff is showing as connecting to Mosquitto broker.
  • MQTT Explorer is also connecting to Mosquitto broker.

There’s also that zwave “bridge” connecting to Mosquitto broker, but that seems to be to allow MQTT control of Zwave devices when using OpenZWave from what I’ve found on a quick google search.

Do you think deleting the MQTT Integration from HA (leaving the broker in place), rebooting HA and then re-adding the integration, followed by another reboot might be worth a try?

I have no experience of openZWave so I can’t advise and I would hate for you to lose ZWave info as I’m aware it’s a pita to setup.

I can understand that the bridge is passing the ZWave command payloads from the visible mosquito broker onto another broker, and that would work…but is HA discovering devices from both brokers or only one ?

It appears if ZWave status updates OK that is coming from the ‘other’ broker as no openZWave topic exists on the one you are monitoring. I didn’t know HA could discover from multiple brokers but it’s a nice feature and I suppose required if you are using an externally populated ZWave MQTT implementation. … same with Zigbee

1 Like

It could of course be visibility of data on a single broker being different based on username too

1 Like

I can’t see two brokers running on the HA install, only Mosquitto.

I do know that the HA integration for Mosquitto uses an auto-generated username and password to connect. I wonder if for some reason that user is now unable to see published topics?

I’ve hesitated editing the MQTT Integration settings as I have no idea what the built in password for it is, but sod it, I’m gonna take a snapshot and edit the Integration to use the MQTT User I added that’s shown in the MQTT Config.

Taras , when he sees this topic I’m sure will have an obvious answer , he’s great on things like this.

1 Like

Well, here’s a bit of an improvement.

  1. Went into Configuration > Integrations.
  2. Clicked on “Configure” for the MQTT Integration
  3. Clicked on Re-configure MQTT
  4. Left broker set to “core Mosquitto” and port to 1883
  5. Changed username and password to my username and password configured in the broker add-on configuration

I can now listen to # from the MQTT Integration’s Configure page, and I could see the same payloads that are visible in MQTT Explorer; and my washing machine state no longer shows as “unknown” it’s back to “Idle”.

Just waiting on the enviro Pi to next publish to MQTT to see if those sensors update.


Update: WOOOOOO! the environment sensors are no longer showing as “Unknown”.

Telling the MQTT Integration to use my MQTT Username and password instead of the configured HomeAssistant ones has fixed it.

No idea still why it was working previously but stopped though!

Thanks @xAPPO for all your help. Your post about visibility of different usernames gave me the idea to try changing the username and password for the MQTT integration. Top man! :fist_right: :fist_left:

1 Like

I’ve revisited this with my Diesel Heater Yesterday.
Regardless of what I’ve tried I’ve still not succeeded in getting all my Topics showing.
Two I’m interested in Simply show up as unknown.
As exoected they are there in MQTT Explorer.

These are the failing code snippets. Many edit have been tried.

  - name: Afterburner Current Usage
    unique_id: afterburner_current_usage
    state_topic: “AfterburnerFF4E3C/sts/FuelUsage”
  #    unit_of_measurement: mL
  #    state_class: total_increasing

  - name: Afterburner Fuel Rate
    unique_id: afterburner_fuelrate
    state_topic: “AfterburnerFF4E3C/sts/FuelRate”
    unit_of_measurement: mL/h
    state_class: measurement

After Reading the above posts I tried having the Integration listen to #
and discovered they are both visible there.

Noted also that QOS & Retain is the same as in another entity that works.

This put me even more at a loss as to what’s going wrong as it’s no proven that HA has them in the integration, but is not passing them to the next level.

Anyone else seen a similar situation?

Edit:-
Also tried using,
homeassistant & the long numeric string in Explorer the that worked fine.

Then tried changing the username I’ve set for MQTT,
mozzie & my password & that changed nothing,