Broken mqtt is it log file permissions?

Configuration
1

Hi, I’ve been struggling to fix this for some weeks now. I’m running homeassistant 0.90.2 in a venv on hassbian. Everything was working fine for a few years - then mqtt became a bit unreliable and on investigation I believed I’d run out of disk space and deleted the largest log files I found. I think it is possible that in so doing I’ve created a permissions problem but I cannot for the life of me overcome it.

Below is my attempt to succinctly summarise current issue and actions:

SYMPTOM: attempting to publish to broker (Mosquitto defined in configuration.yaml by including host IP, username, pword, port) results in error connection refused:

login as: pi
[email protected]’s password:
Linux hassbian 4.19.42-v7+ #1219 SMP Tue May 14 21:20:58 BST 2019 armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Jul 12 15:47:48 2019 from 192.168.1.150
pi@hassbian:~ $ mosquitto_pub -d -u *** -P *** -t “mg/zone1” -m “123”
Error: Connection refused

To rule out authentication issue - recreate username and password in a new password file:

pi@hassbian:~ $ mosquitto_passwd -c newpwordfile ***

(password then set for the username above)

mosquitto.conf then edited as follows (conf file is v lengthy so only showing
what I think is relevant here - tell me if you need to see other areas)

log_dest file /var/log/mosquitto.log

allow_anonymous false
(I have run with allow_anonymous true but still get connection refused)
listener 9001
protocol websockets *
listener 1883
user ***
password_file /etc/mosquitto/newpwordfile

after reboot the following symptoms remain. mosquitto seems to have loaded and exited (why?) cleanly.

pi@hassbian:~ $ sudo service mosquitto status
● mosquitto.service - Mosquitto MQTT Broker daemon
Loaded: loaded (/etc/systemd/system/mosquitto.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Sun 2019-07-14 14:12:16 UTC; 1min 10s ago
Process: 365 ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf -d (code=exited
Main PID: 365 (code=exited, status=0/SUCCESS)

Jul 14 14:12:16 hassbian systemd[1]: Started Mosquitto MQTT Broker daemon.

pi@hassbian:~ $ mosquitto_pub -d -u *** -P *** -t “mg/zone1” -m “50”
Error: Connection refused

To try to help you see what is going on, here are the contents of mosquitto.log (much of this was previous, will rerun after deleting to get fresh data). The only way I can get broker to accept connections is to run mosquitto -d and it will be in that scenario that much of the below log traffic would have been generated.

1561238678: mosquitto version 1.6.2 starting
1561238678: Config loaded from /etc/mosquitto/mosquitto.conf.
1561238678: Opening ipv4 listen socket on port 1883.
1561238678: Opening ipv6 listen socket on port 1883.
1561238678: New connection from 192.168.1.166 on port 1883.
1561238678: New client connected from 192.168.1.166 as zone1 (p2, c1, k15, u’’).
1561238679: New connection from 192.168.1.144 on port 1883.
1561238679: New client connected from 192.168.1.144 as ESPClient_84:F3:EB:B3:95:23 (p2, c1, $
1561238679: New connection from 192.168.1.167 on port 1883.
1561238679: New client connected from 192.168.1.167 as zone2 (p2, c1, k15, u’’).
1561238689: New connection from 192.168.1.166 on port 1883.
1561238689: Client zone1 already connected, closing old connection.
1561238689: New client connected from 192.168.1.166 as zone1 (p2, c1, k15, u’’).
1561238692: New connection from 192.168.1.167 on port 1883.
1561238695: Client zone2 already connected, closing old connection.
1561238695: New client connected from 192.168.1.167 as zone2 (p2, c1, k15, u’’).
1561238698: Socket error on client zone1, disconnecting. *
1561238698: New connection from 192.168.1.166 on port 1883.
1561238698: New client connected from 192.168.1.166 as zone1 (p2, c1, k15, u’’).
1561238705: New connection from 192.168.1.167 on port 1883.
1561238705: Client zone2 already connected, closing old connection.
1561238705: New client connected from 192.168.1.167 as zone2 (p2, c1, k15, u’’).
1561238708: Socket error on client zone1, disconnecting.
1561238708: New connection from 192.168.1.166 on port 1883.
1561238708: New client connected from 192.168.1.166 as zone1 (p2, c1, k15, u’’).
1561238714: Socket error on client zone2, disconnecting.
1561238714: New connection from 192.168.1.167 on port 1883.
1561238714: New client connected from 192.168.1.167 as zone2 (p2, c1, k15, u’’).
1561238716: mosquitto version 1.6.2 terminating
1561238716: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.

once log cleared (deleted) and mosquitto restarted (via reboot or via sudo service mosquitto start or via mosquitto -c /etc/mosquitto/mosquitto.conf) , symptoms same but nothing in log - it remains blank

pi@hassbian:~ $ sudo nano /var/log/mosquitto/mosquitto.log
pi@hassbian:~ $ mosquitto_pub -d -u *** -P *** -t “mg/zone1” -m “50”
Error: Connection refused
pi@hassbian:~ $ sudo nano /var/log/mosquitto/mosquitto.log (nothing there just blank)
pi@hassbian:~ $ sudo service mosquitto start
pi@hassbian:~ $ sudo nano /var/log/mosquitto/mosquitto.log (nothing there just blank)
pi@hassbian:~ $ mosquitto_pub -d -u *** -P *** -t “mg/zone1” -m “50”
Error: Connection refused

interestingly if I do the following to prove to myself it is using the conf file I want:

pi@hassbian:~ $ mosquitto -c /etc/mosquitto/mosquitto.conf
1563113985: Error: Unable to open log file /var/log/mosquitto.log for writing.

which makes me remember difficulties started when Pi appeared to run out of disk space and
I hurriedly deleted biggest log files… have I created permissions issues?

to check, first I had to recreate the log file with one space character inside because in ‘clearing’ the file I just deleted it and it seems mosquitto can’t create it when it initialises, then I chmod 777

pi@hassbian:/var/log/mosquitto $ ls -l
total 4
-rwxrwxrwx 1 root root 2 Jul 14 14:22 mosquitto.log
pi@hassbian:/var/log/mosquitto $ mosquitto -c /etc/mosquitto/mosquitto.conf
1563114267: Error: Unable to open log file /var/log/mosquitto.log for writing.

*looking at socket errors in old log contents above I have gone back into
mosquitto.conf and uncommented the websockets and listener 9001 lines and restarted
mosquitto but symptom same as last command shown - error opening log file for writing.

Not sure what to do next…?

2

I assume you installed mosquitto from apt. If so try

sudo apt install --reinstall mosquitto
3

thanks very much for the suggestion - just tried it but unfortunately symptom still Error: Connection refused, and if I try “pi@hassbian:~ $ mosquitto -c /etc/mosquitto/mosquitto.conf” I still get:
1563177225: Error: Unable to open log file /var/log/mosquitto.log for writing.

4

Is it normal to have two mosquitto.conf files? (Hence why I keep trying -c and specifying the conf I want mosquitto to use. Because mosquitto -d works and I understand that uses a default config…

I have conf files at the following locations:

./etc/mosquitto/mosquitto.conf
./opt/hassbian/suites/files/mosquitto.conf

5

How did you install mosquitto in the first place?

6

with apt install

7

I wonder if you backed up your config, did a fresh install of Hass in the venv and copied your config back over would it would save you hours of troubleshooting.

8

How is that going to help?

9

it worked for me on a number of occasions when other suggestions failed. I was trying to assist here. What’s with your comment. Not very helpful. Maybe you have the solution so.

10

The problem is not with home assistant. It is with mosquitto, which is not even in the venv.

11

Well your suggestion didn’t help either.

I’m quite aware where MQTT broker is installed.

As the OP seemed to think it was a full disk causing the problem and deleted files to remedy the problem he may have caused more problems to his install, hence my suggestion to do a fresh install to rule this out.

12

Indeed a fresh install from scratch may be easier than troubleshooting, but I didn’t think that was what you were suggesting.

13

I’ve seen so many Linux installs broken through random file deletions and messing about with file permissions it ends up easier to reinstall the OS after backing up the working folder.

14

Thanks both for your insights - I really appreciate it. For example it wasn’t apparent to me that mosquitto is outside the venv and I had a no doubt wrong suspicion that I was trying to set user and password as the wrong user/environment.
I’ll try the fresh install over the next 48 hours unless there are any other suggestions.

1 Like
15

Yeah just do it!