Fail2ban / ha : still allowing banned IPs

Hi

So I installed fail2ban as per:


Tested by creating a new temp user in HA. Supplied wrong password 3 times. fail2ban log correctly states “Ban…”

2020-03-30 13:12:22,114 fail2ban.filter         [516]: INFO    [ha] Found xxx.xxx.xxx.xxx - 2020-03-30 13:12:22
2020-03-30 13:12:29,277 fail2ban.filter         [516]: INFO    [ha] Found xxx.xxx.xxx.xxx- 2020-03-30 13:12:29
2020-03-30 13:12:30,883 fail2ban.filter         [516]: INFO    [ha] Found xxx.xxx.xxx.xxx- 2020-03-30 13:12:30
2020-03-30 13:12:31,198 fail2ban.actions        [516]: NOTICE  [ha] Ban xxx.xxx.xxx.xxx

iptables correctly shows the drop in the chain…

Chain f2b-ha (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REJECT     all  --  *      *       xxx.xxx.xxx.xxx/0    reject-with icmp-port-unreachable
 8488  730K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

but if I use the correct password., HA allows login and presents overview screen.

I’ve set bantime to -1 so it should be permanent?

What should I look into?

You created a temp user…then what?

You tried logging in from a different ip address? Using a VPN? Cell service?

Is the ip address there correct? Does it match whatever you were trying to connect from? You shouldn’t be able to get to the login page after the ban to input the correct password.

This line is confusing me:

iptables correctly shows the drop in the chain…
but if I use the correct password., HA allows login and presents overview screen.

There should be no ‘but’. Once the chain is in place, whatever you were trying to connect from shouldn’t even get the login page again. It has nothing to do with the password after that. Did you mean

iptables correctly shows the drop in the chain…
but if I use the correct password., HA allows login and presents overview screen. I can still get to the login page

Are you using NGINX?

You only pasted the chain. I’m assuming there is a normal filter rule to redirect packets to the f2b-ha chain (the chain has seen 8488 packets, so something is happening).

There is no good reason to make the ban permanent. Most malicious things are behind vpns/proxies anyway…and each time they try it will be a new address. The purpose of f2b is to slow them down. If they get banned for 1 hour, their bot software will just move on.

Ok, Perhaps a little clearer (on the default filter table)

  • iptables is correct. INPUT chain 1st rule is to branch to the fail2ban chain.
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
14905 1432K f2b-ha     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 0:65535
  • The f2b-ha chain is correct, and the correct IP is blocked (IP hidden here)
Chain f2b-ha (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REJECT     all  --  *      *       xxx.xxx.xxx.xxx      0.0.0.0/0            reject-with icmp-port-unreachable
14905 1432K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
  • Yet I can still log in to HA. using the same machine, same IP as per above that should be blocked.

Just dont’ understand.

There is no good reason to make the ban permanent. Most malicious things are behind vpns/proxies anyway…and each time they try it will be a new address. The purpose of f2b is to slow them down. If they get banned for 1 hour, their bot software will just move on.

Not my experience. I’ll leave that as it is , the issue is not permanent or temporary bans, the issue is iptables/ha allowing logins .
The login route is:

laptop → ISP router → ISP → (publicIP) → internet → cloudDNS → Firewall → cloudVPS → Firewall → VPN → VPN lan client running HA

The IP blocked is the same publicIP all the way through the above route.
No, nginx not running.

See how the pkts is 14905 for both the INPUT and f2b-ha chains. So the flow is correct, but the block just isnt happening. Its like the IP is different than the blocked on but its not, the IP has been added by fail2ban to the f2b-ha chain as soon as I reach the purposely failed third login attempt. It’s the right IP.

EDIT:

I added logging to the f2b-ha chain , manually …

iptables -I f2b-ha 1 -j LOG --log-prefix "DAZ A: "

and now I see that all incoming traffic to my HA server (10.19.49.2) is from my VPN server (10.19.49.1) …

Mar 30 22:13:09 pi kernel: [33257.028351] DAZ A: IN=wg0 OUT= MAC= SRC=10.19.49.1 DST=10.19.49.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=35690 DF PROTO=TCP SPT=58954 DPT=8123 WINDOW=744 RES=0x00 ACK URGP=0

and that’s correct , as per the flow above, but the originating request was from the blocked IP .
Fail2ban correctly identified the blocked IP but ip tables is logging my VPN server IP and that is not the originating IP, hence the block is not happening.
I guess this is what they call “NAT/MASQUERADE” ?? I dont know enough about this topic.
All traffic to my HA server (10.19.49.2) will come directly from my VPN server (10.19.49.1) but its the originating IP that fail2ban correctly identified which is the one I need to block. I’m not sure how to fix this in HA, fail2ban or whereever.

I didnt know much about iptables before coming across this problem !

If the only way to access your HA is via the VPN server, just put the ban functionality on the VPN server…

Huh?
Em, HA instance is on VPN client 10.19.49.2
VPN server is 10.19.49.1
How can fail2ban on VPN server interrogate a VPN client log file to then block an IP on the VPN server.
And as per OP, the VPN server is running on a VPS in the cloud.

fail2ban on the HA instance is working correctly and acquired the correct public IP to block. But somehow, iptables is logging that all packets are coming from my VPN server - which they are, but they originated outside the VPN , from the internet, and somehow I need to modify iptables but I dont know how. A net expert/guru is what’s required here please good folk,

Is the ONLY way to access your HA instance via an authenticated VPN server?

If so, then don’t use fail2ban on HA. Use something similar on VPN server.

I dont think you understand - never mind.
(fyi: why would I run fail2ban on the VPN server when the HA instance is on a VPN client and it is that machine that needs to both identify and ban traffic !)

Metaphor: if you’re trying to keep strangers from the bedroom, you check them at the front door, not the bedroom door. Fail2ban should run on the edge between your network and the internet. Based on the setup you described, that sounds like the login for your vpn server.

This only makes sense if the VPN server is your only pathway to HA - which you didn’t answer above.

ITS THE TARGET OF A REVERSE PROXY.
Honest, doesn;t matter. Please. Thanks for replying anyway. I need something else than your suggestion - but thanks anyway.

Guru needed please. Anyone?

No problem. Just an idea. Good luck with your issue.

Yeah, once the connection hits the NAT, it is managed by the conntrack device. No iptables rules you put in will affect this connection until the conntrack is reset. The NAT table is used only once per flow.

Take a look at the iptables flow diagram (https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg)

NAT and conntrack are hit before the input filter where you have the iptables rule. It’s going to skip it and go straight to the FORWARD table.

The very first packet will traverse the normal rules.

Can you configure fail2ban to go on the prerouting input chain before NAT?

This is a common annoyance I have with iptables and NAT. Just something as simple as changing the conmark is annoyingly hard because of it.

ABSOLUTELY BRILLIANT JIM !
That is sooo helpful. Great pics and as they say “…they speak a 1000 words”.

Yes, I was wondering whether I could do this and now I know how thanks to you… So, I put a log in the PREROUTING chain of the mangle table. And , voila …

Mar 31 09:44:11 pi kernel: [74719.469250] IN=eth0 OUT= MAC=**** SRC=THE.PUBLIC.IP.NEEDED DST=192.168.1.10 LEN=124 TOS=0x00 PREC=0x00 TTL=55 ID=25539 PROTO=UDP SPT=51820 DPT=42612 LEN=104

and there it is. Incoming on eth0 before being forwarded to the VPN interface wg0

So I did a quick test and inserted a LOG rule on the PREROUTING chain of the mangle iptable. Voila, the public IP is in SRC for the packet. Great. So this is where I need to insert the blocking rule. This is fairly trivial and to answer your question , YES, it is possible to configure fail2ban to cover this instance by modifying fail2ban’s commands:

  • fwstart command that sets up iptables and injects a new chain f2b-ha into the filter table, and sets up the input chain that forces all incoming traffic to jump to the f2b-ha chain.
  • fwend command that removes the work done by fwend to “restore” iptables.
  • fwban command that inserts a REJECT rule into the f2b-ha chain to block <HOST>.
  • and fwunban command that removes <HOST> from the f2b-ha chain.

So all that one needs to do is tweak these command definitions. I’ll post the solution on this thread shortly.

Excellent Jim. Thanks a bunch.

One more thing you may know Jim, fail2ban grabs the IP from the HA log file using the placeholder in the regexp from /etc/fail2ban/filter.d/ha.conf. So HA has already inserted the IP into the log file , but how does HA get that [correct] IP - because whether HA is running in a VPN or not, that IP is always correct - and the one I need to use - so just wondering how it acquires the IP?

Daz

EDIT

ok, this is weired and I dont know whats happening.
I set up iptables on the mangle table:-

root@pi:/etc/fail2ban# iptables -vnL -t mangle
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 1269  144K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 4 prefix "** prerouting **: "
 2404  128K f2b-ha     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22
 108K   29M CONNMARK   udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* wg-quick(8) rule for wg0 */ CONNMARK restore

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
29883   13M CONNMARK   udp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0xca6c /* wg-quick(8) rule for wg0 */ CONNMARK save

Chain f2b-ha (1 references)
 pkts bytes target     prot opt in     out     source               destination
 1065 55816 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 4 prefix "** f2b **: "
    0     0 DROP       all  --  *      *       aa.bb.cc.dd          0.0.0.0/0
 2404  128K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
root@pi:/etc/fail2ban#

Fail2ban is correctly inserting a block [DROP] rule on public IP aa.bb.cc.dd.- CORRECT
When packets come into the PREROUTING chain, the LOG correctly identifies SRC as the public IP aa.bb.cc.dd - CORRECT
It then jumps to the f2b-ha chain - CORRECT
The LOG then writes a log but now the SRC has changed to 192.168.1.127 - INCORRECT.
So something is happening to SRC within the jump from PREROUTING to f2b chains. - INCORRECT??

192.168.1.127 is my laptop on my LAN that I am connecting to www.some.site which is then forwarding traffic to my reverse proxy server which is then redirecting traffic through my VPN - CORRECT behaviour

But why is SRC changing during the jump from PREROUTING to f2b-ha chains in mangle table. Otherwise it would be fine if SRC was not modified. I bet this is to do with S/NAT but I dont know enough about networking. At least I’m 50% there in that the SRC in PREROUTING is now as I expect. However, f2b-ha chain is not blocking the IP because SRC has now changed to 192.168.1.127.

??

I may have been misleading with my suggestion after looking a bit more into it. PREROUTING only exists for mangle and nat…neither of which are allowed to DROP a packet. (Well, you’re not supposed to. Some kernels/versions of iptables might just ignore it).

EDIT: Skip to the end to try the easy part first. Only do this if the packets aren’t actually dropped…

So, here are 2 possibilities.

  1. Drop it in the raw table. This one should be able to…
iptables -t raw -A PREROUTING --source aa.bb.cc.dd --jump DROP

I’m guessing the mod to f2b would be just as easy to put there.

If that doesn’t work…

  1. Mark it in the mangle table and drop it in the forward table (and input for new connections)
# f2b will insert a mark rule for all bad ip addresses. If we can't drop on the mangle table, 
# we can at least mark the packet to be dropped by a later rule.
iptables -t mangle -A PREROUTING --source aa.bb.cc.dd -j MARK --set-mark 1

# These rule always exists. Anything with the mark is rejected (both forward and input)
iptables -A INPUT -m mark  --mark 1 -j REJECT --reject-with icmp-port-unreachable
iptables -A FORWARD  -m mark  --mark 1 -j REJECT --reject-with icmp-port-unreachable

The latter is a bit harder. You’d have to get f2b to do the MARK rule on the mangle table. I’ve never used f2b, so I don’t know what that would take.


As for your other questions…

  1. I have no idea how HA knows the right ip address. And you say “it’s always right”, but in my case, all packets look like they come from 127.0.0.1 thanks to my poor NGINX config. I should fix it…but I know that will be a day spent for little gain (until I want to implement f2b or something similar…)

  2. I don’t see how it is possible that the src address changes while in the same PREROUTING mangle table! My only hope (and guess) is the log is catching a reply message and missing the one we care about.

How are you adding those log rules? By default, it should log every packet unless you specify a --rate. Or maybe you added the log rules later and didn’t reset the counters? That would make more sense…

Your chain should be evaluated to completion as soon as the packet jumps to it. After it leaves the chain, it will continue through the rest of the PREROUTING rules. In order. Your order is simple…LOG, f2b-ha (tcp only), CONNMARK.

Though, we should note, you are logging ALL traffic before the chain, and only logging tcp in the chain. I would assume these messages are tcp…but who knows anymore. Maybe add the f2b-ha rule for ALL traffic, not just tcp? Actually, yeah…looking at your log…

Mar 31 09:44:11 pi kernel: [74719.469250] IN=eth0 OUT= MAC=**** SRC=THE.PUBLIC.IP.NEEDED DST=192.168.1.10 LEN=124 TOS=0x00 PREC=0x00 TTL=55 ID=25539 PROTO=UDP SPT=51820 DPT=42612 LEN=104

So after writing allllll of that, it might just be as simple as forwarding both udp/tcp to the f2b-ha chain :stuck_out_tongue:

Wish I was more of an expert in this stuff. Hope I’m not wasting your time leading you in wrong directions lol.

Good luck.

the problem is not so much where and how to drop, that’s sorted , it’s works in the mangle table , the problem is - somehow - SRC IP address is changed when jumping to the f2b-ha chain. I swear. I log the packet when it hits prerouting , it’s got the right SRC , then i log it on reaching the f2b-ha chain, and SRC had changed. honest

that’s the issue now.

Right. I covered that one at the end :stuck_out_tongue:

Your logs show the right ip address on UDP. But you’re only forwarding TCP to the f2b-ha chain. So you’re seeing 2 different messages.

Good spot. Didnt notice that.
But
That is sort of a distraction, because the real point is that the SRC address is changing when jumped to the f2b-ha chain. So even if I was forwarding both tcp and udp, the fact is that the SRC has changed and so will still be allowed through because the block is on the public IP.
Something clever going on here because I am actually testing banning by deliberately entering incorrect login credentials using my laptop. The route for traffic is thus:-

  1. my laptop (192.168.1.127)
  2. my router (192.168.1.1)
  3. router WAN port (aa.bb.cc.dd)
  4. my ISP
  5. my domain host
  6. my cloud server
  7. my VPN host
  8. my VPN client running Home Assistant.

The LOG rule in the mangle table PREROUTING chain at step 8 correctly logs aa.bb.cc.dd.and the next rule is the jump to the f2b-ha and in there the 1st rule is a LOG and at this point SRC has changed to 192.168.1.127 which is indeed the originating IP in my LAN. The next rule is the DROP on aa.bb.cc.dd creating by f2b but this doesnt work because SRC is now 192.168.1.127…
Forwarding both udp and tcp doesnt change matters, the SRC is still mangled to my local LAN IP and thus isnt blocked.

I think the issue now is your modem/router doing its own NAT/routing/DNS cache/ etc.

The first request comes in from laptop, goes to router, goes out to the ISP, does DNS lookup, yadda yadda yadda. Comes back to your modem after the ISP figured it out.

The modem forwards it to the router who sees the packet and says “WTF, this originated from inside the house…and is destined for inside the house on this port forwarding rule”. It forwards it on with its own NAT rule…but then knows all future packets will skip the WAN and stay on LAN. He’s not going to forward your traffic externally now that he knows this connection is a LAN connection. Thus, your source ip will be the laptop.

Put your laptop on a separate VPN before you try this.

If you’re worried about attackers connected to your LAN, I’d harden your network security rather than try to handle this corner case. But I’m sure it could be done.

Yes, that is what is happening Jim, but somehow HA is managing to get the true public IP in this case , the IP assigned to the WAN side of my router. And if it were someone I wanted to ban then it is that IP, the WAN IP, that I want to ban. Fail2Ban is then using detected by HA which is not the SRC address getting logged in iptables. So Fail2Ban is not working because it is setting a rule to ban the public IP when that IP is not in any of the packets received into iptables.

I see what you mean about hardening network, but all I want to do is run fail2ban on my HA server and block IPs that fail to provide correct HA credentials 3x or more, forever.

There surely has to be a way to obtain the originating source public IP ? HA is managing to do it, why can’t iptables? Im stuck.

Yeah, but it is working I think. It should be banning your WAN IP address. Your WAN ip address was 100% the original source address before your router took over.

If you were on a laptop on an external network, ALL packets would be the originators WAN ip address.

If I, from my own network right now tried to log into your network, I would use my laptop (192.168.0.200). You would see my WAN IP address from my ISP attempt and ban that. Then I would 100% not be able to continue.

Try it from your phone. Disconnect wifi, open web browser (incognito recommended), and try to log in. You’ll get banned.

Your phone IP address is always changing though, so don’t expect your phone to be banned forever (which was my original point about not needing to ban ip address forever…though it shouldn’t hurt anything. Unless you start getting thousands of these, then your network could slow down evaluating all of these iptables rules).

You’re just hitting a corner case of packets originating from your LAN. Again, you can try to fix that issue, but it’s not something I would worry about. If some attacker is on your LAN, I don’t think Home Assistant will be your major concern :stuck_out_tongue:

No Jim, that IS the problem. The public WAN IP is not the SRC address, or for that matter - not any address in the packet.
I dont know how to explain this any easier. I’ve tried.
Let me have another go. I’ll try to itemise the traffic flow.

  1. So, I point my browser on laptop to homeassistant.mydomain.net (note homeassistant is a subdomain. I own the mydomain.net domain - not the real name of my domain of course)
  2. First off is my laptop (IP 192.168.1.127 on my LAN assigned by DHCP from router)

next is

  1. My router (IP 192.,168.1.1 on my LAN)
  2. My router (WAN IP a.b.c.d assigned by my ISP) - remember a.b.c.d , I refer to this later
    then out to my ISP
  3. My ISP (several IPs going on here)

then

  1. ISP DNS resolves mydomain.net to my Virtual Private Server (VPS) with Digital Ocean (Public IP w.x.y.z) - remember w.x.y.z

so on to:

  1. My VPS has its WAN IP w.x.y.z on eth0 interface , but it is also my VPN server and so has an internal VPN IP 10.19.49.1 on its wg0 interface.

so

  1. My VPS NGINX is configured to proxy all traffic to homeassistant subdomain (remember homeassistant.mydomain.net above).

and thus

  1. My VPS/VPN server then proxy-passes traffic from eth0 to wg0 interface and then onto my VPN client with IP 10.19.49.2
  2. My VPN client 10.19.49.2 is running the HA instance and this is the target of the original browser request for homeassistant.mydomain.net.
  3. The HA login screen appears on my browser, I enter the incorrect details, 3 times, then I want fail2ban to spot the “Invalid authentication blah blah” error in the homeassistant.log file and then ban my original public WAN IP a.b.c.d from step 4 above. Which it does correctly well done f2b !

so fail2ban correctly inserts a new rule to DROP IP SRC a.b.c.d - correct

  1. What is actually happening by now, thanks to NAT/DNAT/MASQUERADING and so on all over the place, is that in iptables, on my VPN client, running HA, is that the SRC IS NOT a.b.c.d but is the address of my VPS server, w.x.z.y . That is in all cases in all tables in all chains. Clearly I do not want to ban my own VPS/VPN server.

  2. At no time is any packet SRC from a.b.c.d - the original IP from where the browser was pointed to . Thus a.b.c.d has been lost in translation - but HA is somehow obtaining it ??.

  3. Fail2Ban is correctly acquiring a.b.c.d because that is what is in the log file written by HA, so HA certainly knows where the traffic came from, but iptables “does not”.

  4. It doesnt matter who or how or where I try to connect to homeassistant.mydomain.net , the result is always the same, SRC in iptables is always my VPS/VPN server IP. Conceptually this is correct, because traffic has come from there, it always will, but it is not the original WAN IP I need to block - behind which , as you say, will be a phone, a laptop, whatever

Comprende ??
Phew

EDIT

Jim, I don’t understand AF INET TCP IP etc.
But I did find http config for HA.
So, I used this…

http:
  # For extra security set this to only accept connections on localhost if NGINX is on the same machine
  server_host: 10.19.49.2 
  # Update this line to be your domain
  base_url: https://ha.setdet.net
  use_x_forwarded_for: true
  # You must set the trusted proxy IP address so that Home Assistant will properly accept connections
  # Set this to your NGINX machine IP, or localhost if hosted on the same machine.
  #trusted_proxies: <NGINX IP address here, or 127.0.0.1 if hosted on the same machine>
  trusted_proxies:
    - 10.19.49.1
  ip_ban_enabled: true
  login_attempts_threshold: 3

note use of 10.19.49.0/8 addresses above. This makes sure all requests are via the VPN.

AT LAST
3 incorrect logins and the IP is banned. Just what I wanted. So although I never got to understand how iptables is working, I have managed to ban illegal IPs.
Sigh.

Tested and works.
To unban my own IP after testing, I just removed the ban lines from <ha-config-dir>/ip_bans.yaml