How to configure NGINX to reverse proxy to mqtt using SSL

Configuration
,
1

I want to be able to use nginx to reverse proxy (I don’t understand why it’s called “reverse”) to the mosquito aka mqtt add-on so that I can use mqtt.example.com to connect to the broker using SSL. I want nginx to use route all traffic from port 80 to port 443. In other words I don’t want to open new ports for mqtt. The clients don’t specify a port and they get sent to the SSL port.

The following steps are how I got to the point where I’m able to access mosquitto over port 80. However, I’m stuck on getting it even work on SSL, much less having it routed there. With regualr websites, I’m able to get a Let’s Encrypt SSL certificate installed, as well as routing to 443 for SSL by using CertBot. It usually takes care of everything for you.

sudo certbot --nginx -d mqtt.example.com

However that isn’t working for me, and I don’t know why. I’ve tried both Certbot options for allowing HTTP or forcing HTTPS and neither work. In fact, when I try the “both” option I can’t connect with the test service anymore. Here are the steps I took to set things up. Any help would be appreciated, but I’d really like to make it work using CertBot because that will be updating the certs. Thanks for the use of your eyes. :slight_smile:

I put this together from reading the docs and some of the suggestions from this thread and help from @rbray89.

Steps to get mqtt.example.com working with nginx

Install and configure mosquitto add-on broker

Use hassio to install the mosquitto addo-on. Update the configuration settings in the UI. The changes to the default are setting plain_websockets to *true, anonymous to false and adding a login username/password set.

{
  "plain": true,
  "plain_websockets": true,
  "ssl": false,
  "ssl_websockets": false,
  "anonymous": false,
  "logins": [
    {
      "username": "foo",
      "password": "bar"
    }
  ],
  "customize": {
    "active": false,
    "folder": "mosquitto"
  },
  "certfile": "fullchain.pem",
  "keyfile": "privkey.pem"
}

Configure mqtt component in configuration.yaml.

Modify the configuration.yaml to setup the mqtt component.

mqtt:
  client_id: home-assistant-1
  username: foo
  password: bar
  broker: 127.0.0.1

Create nginx server block

At this point, make sure you have nginx installed. This guide is a pretty good walkthrough to installing nginx, a firewall, and creating a test site. This is not using the nginx addon, but rather is a seperate service. I have mine running on a different machine than what is running hassio. Make test site and use certbot (described in the guide) to make sure that http://test.example.com get’s redirected to https://test.example.com. After you have that test working, create a server block for mqtt:

sudo nano /etc/nginx/sites-enabled/mqtt.example.com

server {
   server_name mqtt.example.com;
   listen 80;
   location /
   {
      proxy_pass http://172.16.68.67:1884; #address of home assistant machine
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
   }
 }

Validate the server block

sudo nginx -t

Restart service

sudo service nginx restart

Use HiveMQ to test externally. Use mqtt.example.com as the host, use port 80, and specify the username and password.

Use CertBot to create and install a cert and modify the server block

sudo certbot --nginx -d mqtt.example.com

Unfortunately this last step not only doesn’t work, it breaks port 80 working. @rbray89, do you have any idea of what I’m missing?

2

OK, I see your issue now. You’re tryint to use the nginx plugin for certbot. Most documentation I’ve read says you don’t want to do this as it is explicitly broken in some circumstances. You want to use the standalone certbot and expose the webroot in nginx. You can do this with:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    server_name mqttws.example.com www.example.com example.com;
    listen 80;

    root /var/www/html;
    index index.php index.html index.htm;

    location ~ /.well-known {
        allow all;
    }
}

Then to get certbot working:

sudo wget https://dl.eff.org/certbot-auto 
sudo chmod +x certbot-auto
sudo certbot-auto certonly --webroot --webroot-path /var/www/html  -d example.com -d mqttws.example.com -d www.example.com -d octopi.example.com --email [email protected]

Then in cron, add the renewal service:

sudo crontab -e
12 13  * * *  /usr/bin/certbot-auto renew --quiet --no-self-upgrade --renew-hook "/bin/systemctl restart nginx.service"

Only thing with this is that you must first disable your sites that nginx uses certs for, run certbot, then re-enable your nginx sites and restart nginix. Once you have your first round of certs though, updating them will be automatic. It is just that nginix will fail if it doesn’t have any certs.

This was my server block for mqtt when I was doing this on a manually setup nginx:

server {
    server_name mqttws.example.com;
    listen 443;
    
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
    ssl_dhparam /etc/nginx/ssl/dhparams.pem;

    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
    ssl on;
    ssl_protocols TLSv1.1 TLSv1.2;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;

    proxy_buffering off;


    location / {
        proxy_pass http://localhost:1884/; # The server you want to redirect to
        
        proxy_redirect default;
        proxy_set_header Host $host;
        proxy_redirect http:// https://;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Upgrade $http_upgrade;
#        proxy_set_header Connection "upgrade";

        proxy_set_header Connection $connection_upgrade;
#        auth_basic "Restricted Content";
#        auth_basic_user_file /etc/nginx/.htpasswd;
    }
}
3

@rbray89

No, I’m not using the nginx plugin. I have nginx installed on a separate machine. The only plugin I’m using here is the mosquito addon.

4

@FutureTense
Please re-read. You are trying to use the Nginix “plugin” for CERTBOT (eg. using certbot --nginx) I believe this is where your problems are stemming from.

5

Ah… I’ll try installing Certbot separately. Is there a way to disable the nginx Certbot plugin? It looks like the call to the plugin looks like the same to the standalone.

ETA: Is certbot-auto the standalone?

6

The --webroot argument makes it standalone. It should be functionally very similar to “–nginx”, but without modifying your nginx config. You might be able to use --nginx certonly, but I’ve never used that myself.

certbot-auto is just the latest certbot script. See here:
nginx:


webroot:
7

How do you disable sites? Is this done in the server block? If I decide to start over and just get all my sites running on http, will running certbot-auto modify them accordingly like the cerbot plugin?

8

You should be setting up your sites in /etc/nginx/sites-available and then symlinking to them from in /etc/nginx/sites-enabled. Ideally, each sub domain should be a site, so that you can just remove and re-add the symlink to disable/enable that site. A less ideal approach would be to just comment out the server bock as you mentioned. Using “certbot-auto --webroot” will not touch your nginx sites (which is why it is often preferred in cases like this) All you have to do is provide the .well-known directory in your webroot.

The basic first-time init process is:

  1. Add .well-known and webroot site to nginx.
  2. Disable https sites in Nginx if you have them. (so that when nginx starts up it doesn’t die due to missing cert errors. Note that if you already have all your certs you can skip this step)
  3. Run certbot
  4. Re-enable any disabled sites.
  5. Add cron job for cert renewal
9

I’ve gone through that procedure with the nginx certbot addin. I also have separate files per site and am using symbolic linking. What I don’t understand is what do I need to do to disable a site? Remove the symbolic link, or is it something else?

10

Removing the symlink and restarting nginx is all that needs to be done.

11

I just want to make sure I understand. cerbot-auto does NOT change your serverblocks, correct?

12

As long as you don’t use --nginx, correct.

13

Which server block does that go in? Default? Do I need to add each subdomain in there?

14

I used the default block as I didn’t host anything on port 80. If you disable your other sites this should work for you.

15

@rbray89 still can’t get it working. I added some info logging to nginx

11 [info] 8998#8998: *24 client sent plain HTTP request to HTTPS port while reading client request headers, client: 73.200.34.241, server: mqtt.noakland.com, request: “GET /mqtt HTTP/1.1”, host: “mqtt.example.com:443”

I’m using a web based client test site. Are you able to test your SSL on the following?

16

I don’t think that client supports HTTPS. The log indicates it is trying to do HTTP on the HTTPS socket, so it will be rejected. I was able to test with owntracks on my android device, as well as a few other MQTT apps.

Whatever client you use to test, you shouldn’t be specifying the port manually. instead, specify the protocol. Eg. https://mqtt.example.com

17

I also had this problem. My ESP32 board couldn’t validate any mosquitto-hosted letsencrypt certificates, and I spent way too much time trying to get it to work in itself. Instead, I’m using nginx to forward “stream” traffic. This is nginx handling raw tcp traffic, not http+websockts, with its stream_proxy module.

The nginx configuration is

stream {
        upstream backend {
                hash $remote_addr consistent;
                server localhost:1883;
        }
        server {
                listen       8883 ssl;
                listen       [::]:8883 ssl;
                server_name  hass;

                ssl_certificate "/etc/letsencrypt/live/hass/fullchain.pem";
                ssl_certificate_key "/etc/letsencrypt/live/hass/privkey.pem";
                ssl_session_cache shared:SSLB:1m;
                ssl_session_timeout  10m;
                ssl_ciphers PROFILE=SYSTEM;
                ssl_prefer_server_ciphers on;

                proxy_connect_timeout 1s;
                proxy_timeout 10m; # is default
                proxy_pass backend;
        }
}

at the top level and not in any http {} block.

Blog post at using nginx as a SSL offloading proxy to mqtt

1 Like
18

Thanks, that got me to solve my connection problems!

Out of curiosity, is your nginx running in an LXD container?

Asking, because I am, and I have figured out the following two ways are functioning (with preference for Option 2). Maybe someone else can make use of it. (further SSL and proxy parameters are omitted for brevity)

The setup contains an LXD container “revproxy”, running on a host that is reachable on port 8883 from outside the home network. The Nginx reverse proxy is running inside this container. The MQTT broker is provided by an add-on on the home assistant host (192.168.1.100). The setup should be accessible at mqtt.mydomain.com:8883.

Option 1

Disable proxy_protocol for the LXD proxy device:

lxc config device add revproxy mqtts proxy listen=tcp:0.0.0.0:8883 connect=tcp:127.0.0.1:8883 bind=host proxy_protocol=false

Not specifying proxy_protocol in the listen directive with hash $remote_addr consistent; in the upstream block (everything inside the stream block):

upstream mqtt_backend {
    hash $remote_addr consistent;
    server 192.168.1.100:1883;
}

server {
    listen 8883 ssl;
    listen [::]:8883 ssl;

    server_name mqtt.mydomain.com;

    ssl_certificate /etc/letsencrypt/live/mqtt.mydomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mqtt.mydomain.com/privkey.pem;

    proxy_pass mqtt_backend;
}

Option 2

Enable proxy_protocol for the LXD proxy device (everything inside the stream block):

lxc config device add revproxy mqtts proxy listen=tcp:0.0.0.0:8883 connect=tcp:127.0.0.1:8883 bind=host proxy_protocol=true

Specifying proxy_protocol in the listen directive without hash $remote_addr consistent; in the upstream block:

upstream mqtt_backend {
    server 192.168.1.100:1883;
}

server {
    listen 8883 ssl proxy_protocol;
    listen [::]:8883 ssl proxy_protocol;

    server_name mqtt.mydomain.com;

    ssl_certificate /etc/letsencrypt/live/mqtt.mydomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mqtt.mydomain.com/privkey.pem;

    proxy_pass mqtt_backend;
}

Additionally…

The following can be added to reject connections to subdomains of mydomain.com other than mqtt.mydomain.com, such as www.mydomain.com:

server {
    listen 8883 default_server ssl proxy_protocol;
    listen [::]:8883 default_server ssl proxy_protocol;

    server_name _;

    ssl_reject_handshake on;
}

(omit proxy_protocol if not using LXD proxy with it activated)