Cloudflare to HAProxy on OPNSense to Home Assitant resulting in error 400

mrwowsers · June 16, 2021, 1:44am

Hello,
Trying to take care of the warning properly before the next release breaks everything but it just seems to break access via browser and mobile app.

My DNS is hosted through Cloudflare and setup as proxied. It hits my OPNSense router that is running HAProxy for various services. Home assistant is running in HA OS on R Pi 4.

Warning is:

A request from a reverse proxy was received from 192.168.11.1, but your HTTP integration is not set-up for reverse proxies; This request will be blocked in Home Assistant 2021.7 unless you configure your HTTP integration to allow this header

I read through Reverse proxy error but it seems to be more related to NGINX.

I tried adding the relevant http section to my config:

# For reverse proxy
http:
  use_x_forwarded_for: true
  trusted_proxies:
    # Gateway
    - 192.168.11.1
    # Public IP
    - <public IP>
    # Cloudflare IPs
    - 173.245.48.0/20
    - 103.21.244.0/22
    - 103.22.200.0/22
    - 103.31.4.0/22
    - 141.101.64.0/18
    - 108.162.192.0/18
    - 190.93.240.0/20
    - 188.114.96.0/20
    - 197.234.240.0/22
    - 198.41.128.0/17
    - 162.158.0.0/15
    - 172.64.0.0/13
    - 131.0.72.0/22
    - 104.16.0.0/13
    - 104.24.0.0/14

Which resulted in:

2021-06-15 20:43:59 ERROR (MainThread) [homeassistant.components.http.forwarded] Too many headers for X-Forwarded-For: [‘public IP’, ‘random Cloudflare ip’]
2021-06-15 20:44:10 ERROR (MainThread) [homeassistant.components.http.forwarded] Too many headers for X-Forwarded-For: [‘public IP’, ‘random Cloudflare ip’]

I tried the comment by Frenck in Too many headers http 400 Error while using revers proxies · Issue #40421 · home-assistant/core · GitHub that suggested to use http-request replace-value x-forwarded-for ^ "%[hdr(x-forwarded-for)], %[src]". Also tried http-request replace-header x-forwarded-for ^ "%[req.fhdr(x-forwarded-for)], %[src]" to haproxy backend config as suggested on a haproxy github post. Neither worked.

# Backend: HomeAssistant_Pool ()
backend HomeAssistant_Pool
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m  
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 10m
    # WARNING: pass through options below this line
    http-request replace-header x-forwarded-for ^ "%[req.fhdr(x-forwarded-for)], %[src]"
    http-reuse safe
    server HomeAssistant 192.168.11.6:8123

When I try to access home assistant with the above config I get 400: Bad Request in the browser and an entry in the logs (viewing via ssh).

2021-06-15 20:54:15 ERROR (MainThread) [homeassistant.components.http.forwarded] Too many headers for X-Forwarded-For: [‘public IP’,‘random Cloudflare ip’, ‘random Cloudflare ip’]

Hoping another set of eyes can help me get through this. It was working fine before making these required changes.

edit: to fix formatting

koying · June 16, 2021, 6:27am

If I get your config right, you only need your haproxy as trusted proxy. It’s doing the “last mile” between the internet and HA.

mrwowsers · June 16, 2021, 11:22pm

That is the 192.168.11.1. I thought that as long as it was in that list it should be ok. Tried with just 192.168.11.1 and still getting the same error.

koying · June 16, 2021, 11:51pm

Did you revert everything besides the trusted_proxies from haproxy? That’s all you needed to get rid of the original warning.

mrwowsers · June 17, 2021, 1:29am

So I was able to get around the Too many headers for X-Forwarded-For by adding http-request del-header x-forwarded-for to my haproxy config to just remove the Cloudflare X-Forwarded-For entries. Not sure if this is the best option but it seems that Home Assistant doesn’t like to have more than 1 IP in the header value.

HASS relevant config:

# For reverse proxy
http:
  use_x_forwarded_for: true
  trusted_proxies:
    # HA Proxy
    - 192.168.11.1

HAProxy relevant config

# Frontend: Public_Frontend ()
frontend Public_Frontend
    bind 0.0.0.0:443 name 0.0.0.0:443 ssl alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/5f5e55af0a4e38.74070491.certlist 
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 10m

    # logging options
    # ACL: HomeAssistant_Subdomain
    acl acl_5fe8d092262155.65334213 hdr(host) -i hass.mydomain.com

    # ACTION: Remove_Header_to_HomeAssistant
    http-request del-header x-forwarded-for if acl_5fe8d092262155.65334213
    # ACTION: Send_to_HomeAssistant
    use_backend HomeAssistant_Pool if acl_5fe8d092262155.65334213

# Backend: HomeAssistant_Pool ()
backend HomeAssistant_Pool
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m  
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 10m
    http-reuse safe
    server HomeAssistant 192.168.11.6:8123

matthewjporter · July 8, 2021, 7:57am

I use Cloudflare with full proxy and encryption and the following worked for me
I used the following list of IP’s from IP Ranges | Cloudflare and then any others added into my log file

use_x_forwarded_for: true
trusted_proxies:
#log reported ipv4
  - 162.158.159.139
  - 172.69.55.59
  - 141.101.76.195
  - 141.101.98.91
  - 141.101.99.102
  - 141.101.105.87
  - 141.101.105.99
#cloudflare ipv4
  - 103.21.244.0/22
  - 103.22.200.0/22
  - 103.31.4.0/22
  - 104.16.0.0/13
  - 104.24.0.0/14
  - 108.162.192.0/18
  - 131.0.72.0/22
  - 141.101.64.0/18
  - 162.158.0.0/15
  - 172.64.0.0/13
  - 173.245.48.0/20
  - 188.114.96.0/20
  - 190.93.240.0/20
  - 197.234.240.0/22
  - 198.41.128.0/17
#cloudflare ipv6
  - 2400:cb00::/32
  - 2606:4700::/32
  - 2803:f800::/32
  - 2405:b500::/32
  - 2405:8100::/32
  - 2a06:98c0::/29
  - 2c0f:f248::/32

mrwowsers · July 9, 2021, 10:26pm

Are you going through HAProxy at your edge? The main problem I was running into was Too many headers for X-Forwarded-For error.

matthewjporter · July 10, 2021, 2:57pm

I am not, I do full encryption in cloudflare with a direct 443 nat via pfsense.

koying · July 10, 2021, 3:48pm

There was a recent thread with a solution for HAproxy adding its own X-Forwarded-For rather than appending to it, if it still matters.

EDIT: Reverse proxy error - #67 by koying

ra6nar · August 7, 2021, 7:38pm

Thank You! Saved lots of hair!

matthewjporter · August 7, 2021, 7:50pm

@ra6nar

Thats what we are here for

Pay it forward by adding a solution to an issue you have and hopefully it will help someone else out of a hole

Muvo · August 10, 2022, 9:56am

thanks @mrwowsers for the info, for anyone using pfsense gui to set this up I managed to get it working with the following backend ACL action without any of the other settings.

and adding in the local router IP and Cloudflare proxies in the configuration.yaml

LPD · February 11, 2023, 2:23pm

hello, anything else i need to configure in pfsense haproxy?

Muvo · April 10, 2023, 5:59am

Hi Louis,

sorry for the really late reply - you will need to setup haproxy, I followed Lawrence Systems youtube guide - How To Setup ACME, Let's Encrypt, and HAProxy HTTPS offloading on pfsense - YouTube