MQTT Broker not working on bootup?

mmiller7 · October 2, 2022, 11:14pm

I’m having a problem with MQTT, it seems like when the server (hypervisor) is rebooted the MQTT service doesn’t properly reconnect to the MQTT broker.

I see the error in my log:

2022-10-02 19:08:55.537 ERROR (Thread-2 (_thread_main)) [homeassistant.components.mqtt.client] Unable to connect to the MQTT broker: Connection Refused: not authorised.
2022-10-02 19:08:55.538 WARNING (Thread-2 (_thread_main)) [homeassistant.components.mqtt.client] Disconnected from MQTT server 192.168.1.221:1883 (5)

I also can verify this I can’t connect to it from a PC with MQTT Explorer.

It seems like the MQTT addon isn’t starting correctly on system boot?

Although it looks like its running in the Addon, it apparently isn’t:

But oddly, if I simply click “Restart” on the MQTT Broker addon it magically all works again?

And then all my errors are gone and sensors magically appear?

msp1974 · October 2, 2022, 11:27pm

Do you have any errors in the log for the mqtt addon? Ie the log tab on your screenshot?

mmiller7 · October 2, 2022, 11:35pm

I looked, but it seemed not particularly useful.

Restart hypervisor, look at addon:

After manual addon restart:

msp1974 · October 3, 2022, 12:06am

I think because you have a 1 sec update the actual part of the log that maybe of interest is rolling of the top. You should have something like this when it starts (that hopefully shows you some error)

s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/mosquitto.sh
[00:58:09] INFO: SSL is not enabled
cont-init: info: /etc/cont-init.d/mosquitto.sh exited 0
cont-init: info: running /etc/cont-init.d/nginx.sh
cont-init: info: /etc/cont-init.d/nginx.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun mosquitto (no readiness notification)
services-up: info: copying legacy longrun nginx (no readiness notification)
s6-rc: info: service legacy-services successfully started
[00:58:09] INFO: Starting NGINX for authentication handling...
[00:58:10] INFO: Starting mosquitto MQTT broker...
2022-10-03 00:58:10: Warning: Mosquitto should not be run as root/administrator.
[00:58:10] INFO: Successfully send discovery information to Home Assistant.
[00:58:10] INFO: Successfully send service information to the Supervisor.

Also maybe turn on debug in the config.

mmiller7 · October 3, 2022, 1:48am

My guess is there are just too many things being published and its pushing out of the log, its like that all the way to the end of the page where the refresh button is, I can’t scroll back far enough to see startup. I tried disabling rtl4332mqtt and now I can see the startup

Hypervisor reboot:

Manual restart addon:

msp1974 · October 3, 2022, 8:49am

Seems this is a known issue.

github.com/home-assistant/addons

Mosquitto broker Add-on is not initialized correctly when HA starts

opened 07:31AM - 17 Aug 22 UTC

rmsmirnov

unsupported

### Describe the issue you are experiencing Hi everyone, I’ve run into the p…roblem with Mosquitto broker Add-on. When I restart the OS the add-on doesn’t start correctly. It shows the following errors in the log: [cont-init.d] executing container initialization scripts... [cont-init.d] mosquitto.sh: executing... parse error: Expected string key before ':' at line 1, column 4 [08:21:01] ERROR: Unknown HTTP error occured [08:21:04] INFO: SSL is not enabled and [08:21:09] INFO: Starting mosquitto MQTT broker... 1660713669: Warning: Mosquitto should not be run as root/administrator. parse error: Expected string key before ':' at line 1, column 4 [08:21:13] ERROR: Unknown HTTP error occured [08:21:13] INFO: Successfully send discovery information to Home Assistant. Because Mosquitto broker starts with errors the other add-ons which depends on Mosquitto also start with errors (for example Zigbee2Mqtt). However if I login to HA web-interface after HA has been fully initialized and manually restart the Mosquitto broker (through Settings->Add-ons menu) then the add-on starts without any error and it begins to operate in normal mode. And then I restart other add-ons (Z2M etc) and they also start working as expected. So to summarize: Mosquitto broker starts normally only from the second attempt. I was able to reproduce the problem on three devices – one is Raspberry Pi 4 and two are running under Armbian. So it seems that we problem is not device-specific. Also I suspect that the problem began when I’ve upgraded OS from Focal to Jammy through the command do-release-upgrade. I use HA in Supervised mode and I use only default add-ons settings. Just to be sure I couple times made “Reset to defaults”, unfortunately it didn’t help. Here is the add-on config in my installation: logins: [] require_certificate: false certfile: fullchain.pem keyfile: privkey.pem customize: active: false folder: mosquitto I did some debugging with cont-init.d/mosquitto.sh and found that the error occurs when script tries to execute bashio::config command. Next I looked at addons/core/mosquitto/config.yaml and found that 1st line of it contained three defis symbols ‘-‘. It is very close interconnected with the error message “parse error: Expected string key before ‘:’ at line 1, column 4”. I’ve tried to remove this 1st line however it seems that the file addons/core/mosquitto/config.yaml is restored from somewhere (Git?) every time HA starts. I would like to disable auto-update for file addons/core/mosquitto/config.yaml (or for some upper folder) just to validate my assumption. How to do it? Thanks in advance. ### What type of installation are you running? Home Assistant Supervised ### Which operating system are you running on? Other (e.g., Raspbian/Raspberry Pi OS/Fedora) ### Which add-on are you reporting an issue with? Mosquitto broker ### What is the version of the add-on? 6.1.2 ### Steps to reproduce the issue 1. Restart OS 2. Check the logs of Mosquitto broker for errors. 3. Restart Mosquitto broker in order to bring it operate normally. ### Anything in the Supervisor logs that might be useful for us? _No response_ ### Anything in the add-on logs that might be useful for us? _No response_ ### Additional information _No response_

There is a possible work around if you read through the comments on the issue.

mmiller7 · October 3, 2022, 3:16pm

Interesting, I hadn’t seen that in my research (and this is a new problem for me at least).

I’ll dig thru that thread, thank you.