I’ve exhausted my google-fu and throw myself upon the mercy of the community.
I have a Windows Server 2019 host running Hyper-V and Docker Linux containers, containing an nginx/let’s encrypt reverse proxying to a Hyper-V VM running my Home Assistant (hass.io).
Everything works fine connecting directly to the HA using HTTP port 80 and static IP address.
I had everything working fine using IIS as a reverse proxy, but I’m playing with docker and wanted to try reverse proxying it with an nginx docker container.
When proxying with nginx, requests to the front end periodically fail. It’s intermittent. Sometimes a dashboard works fine, and next request, I’m seeing connection timeouts in the nginx request log. While timing out through nginx, I can pound the direct HTTP site all day and get absolutely no timeouts.
Here’s what I think might be an important datapoint: A hard browser refresh (Ctrl+Shift+R) induces this failure mode every time. Perhaps it somehow breaks the websocket and it can’t recover gracefully? I’m not sure, I don’t know much about websockets.
Another datapoint: The base pages will usually load, but supervisor plugins like VS Code, Node-Red and others rarely load (but sometimes do!).
Here’s an example of what I find in the log when the choking happens.
172.19.0.1 - - [07/Apr/2021:22:14:01 +0000] "GET /frontend_latest/chunk.117c67b3d79d17756e8a.js HTTP/2.0" 502 559 "https://host.mydomain.com/lovelace-everything/default_view" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.68" "-"
2021/04/07 22:14:01 [error] 165#165: *11936 connect() failed (110: Connection timed out) while connecting to upstream, client: 172.19.0.1, server: host.mydomain.com, request: "GET /static/fonts/roboto/Roboto-Regular.woff2 HTTP/2.0", upstream: "http://192.168.10.69:8123/static/fonts/roboto/Roboto-Regular.woff2", host: "host.mydomain.com", referrer: "https://host.mydomain.com/lovelace-everything/default_view"
2021/04/07 22:14:01 [error] 165#165: *11936 connect() failed (110: Connection timed out) while connecting to upstream, client: 172.19.0.1, server: host.mydomain.com, request: "GET /frontend_latest/chunk.2d509fe62641846afad3.js HTTP/2.0", upstream: "http://192.168.10.69:8123/frontend_latest/chunk.2d509fe62641846afad3.js", host: "host.mydomain.com", referrer: "https://host.mydomain.com/lovelace-everything/default_view"
2021/04/07 22:14:01 [error] 165#165: *11936 connect() failed (110: Connection timed out) while connecting to upstream, client: 172.19.0.1, server: host.mydomain.com, request: "GET /frontend_latest/chunk.81d46ffb42f4f35a54dc.js HTTP/2.0", upstream: "http://192.168.10.69:8123/frontend_latest/chunk.81d46ffb42f4f35a54dc.js", host: "host.mydomain.com", referrer: "https://host.mydomain.com/lovelace-everything/default_view"
2021/04/07 22:14:01 [error] 165#165: *11936 connect() failed (110: Connection timed out) while connecting to upstream, client: 172.19.0.1, server: host.mydomain.com, request: "GET /frontend_latest/chunk.b5b99f95b1f9c2a1c2fa.js HTTP/2.0", upstream: "http://192.168.10.69:8123/frontend_latest/chunk.b5b99f95b1f9c2a1c2fa.js", host: "host.mydomain.com", referrer: "https://host.mydomain.com/lovelace-everything/default_view"
2021/04/07 22:14:01 [error] 165#165: *11936 connect() failed (110: Connection timed out) while connecting to upstream, client: 172.19.0.1, server: host.mydomain.com, request: "GET /frontend_latest/chunk.117c67b3d79d17756e8a.js HTTP/2.0", upstream: "http://192.168.10.69:8123/frontend_latest/chunk.117c67b3d79d17756e8a.js", host: "host.mydomain.com", referrer: "https://host.mydomain.com/lovelace-everything/default_view"
I’m proxying a few different services (but none terribly active, so I don’t think load is the issue). For that reason I’m only including the HA 443 service definition for my Home Assistant.
I’ve tried it with just one location for /, and because “people on the Internet said so” I’ve also tried adding those dedicated location sections for /api/websocket and /api/hassio_ingress (but honestly I don’t understand why they’d be needed, and I don’t see how they’re involved since the endpoints timing out are not in those locations).
server {
server_name host.mydomain.com;
listen 443 ssl;
# Someone on the Internet had this in their HA reverse proxy.
client_max_body_size 0;
ssl_certificate /etc/nginx/ssl/live/host.mydomain.com/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/live/host.mydomain.com/privkey.pem;
ssl_dhparam /etc/nginx/ssl/live/dhparams.pem;
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
ssl_protocols TLSv1.2;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
# Someone on the Internet had this in their HA reverse proxy.
add_header X-Robots-Tag "noindex, nofollow, nosnippet, noarchive";
# Someone said this was a good idea when using docker.
resolver 127.0.0.11 valid=30s;
proxy_buffering off;
location ~ /.well-known/acme-challenge {
allow all;
root /var/www/certbot;
}
location / {
proxy_pass http://192.168.10.69:8123;
proxy_set_header Host $host;
proxy_redirect http:// https://;
proxy_http_version 1.1;
# This is me trying to deal with timeouts
proxy_read_timeout 60;
proxy_connect_timeout 70;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
location /api/websocket {
proxy_pass http://192.168.10.69:8123/api/websocket;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
location /api/hassio_ingress {
proxy_pass http://192.168.10.69:8123/api/hassio_ingress;
proxy_set_header Host $host;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
Thank you!