Unable to change user of Mosquitto

I have installed the Mosquitto broker add-on and also the Mosquitto integration. When I do a simple test with publish/listen it is working fine using the homeassistant user. But as soon as I change to a specific mqtt user that I have created in Home Assistant it stops working. There is nothing that I can see in the logs. Any idea?