SSH Port blocked by iptables - but can still login to SSH port from any address

Question

I have this happening on a few servers with the same firewall rules, so I suspect I am missing something in my iptables configuration but not sure what is wrong. This is happening on some CentOS servers and well as my Ubuntu servers. I've used iptables for years and thought I knew what I was doing... apparently not the case.

I have SSH running on a non-standard port (2022). I have firewall rules to allow access for my personal IPs, followed by blocking specific ports including 2022, then followed by a deny all rule. In the last 3 weeks or so, my logs are showing failed login attempts on SSH from outside IPs that are not in my accept list. I have a VPN service on my laptop so I can try logging in from various countries, IPs, etc and the firewall is not blocking me anymore like it used to. I used my cellphone as a hotspot so I could ensure I had random IP that should be blocked, yet I could still log in to ssh. I used nmap from random IPs and it shows port 2022 as OPEN.

I'm not sure what could be going on, the firewall used to block SSH on not-allowed IP addresses properly and I don't think I've made any changes before this started and I don't have anything like fail2ban complicating things. I've even checked for rootkits as well and nothing shows up. I've Googled a round a lot, but the search results are so noisy with not-quite relevant responses that I've given up and decided to post the question here and hopefully get some better guidance.

This server is running ubuntu 14.04.6 LTS

My allowed IP's are on the 209.xxx and 216.xxx ranges Here are my iptables rules (iptables -L -n):

ACCEPT     all  --  127.0.0.1            0.0.0.0/0
ACCEPT     all  --  209.xxx.xxx.1        0.0.0.0/0
ACCEPT     all  --  209.xxx.xxx.2        0.0.0.0/0
ACCEPT     all  --  209.xxx.xxx.3        0.0.0.0/0
ACCEPT     all  --  209.xxx.xxx.4        0.0.0.0/0
ACCEPT     all  --  209.xxx.xxx.5        0.0.0.0/0
ACCEPT     all  --  209.xxx.xxx.6        0.0.0.0/0
ACCEPT     all  --  216.xxx.xxx.1        0.0.0.0/0
ACCEPT     all  --  216.xxx.xxx.2        0.0.0.0/0
ACCEPT     all  --  74.xxx.xxx.2         0.0.0.0/0
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp flags:0x3F/0x00
RETURN     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 limit: avg 1/sec burst 2
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp flags:0x3F/0x3F
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:110
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:143
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:465
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:993
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:995
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:2022
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Typical Server logs are showing failures like:

Jul 26 05:29:38 SERVERNAME sshd[3536]: Invalid user postgres from 159.89.231.172
Jul 26 05:29:39 SERVERNAME sshd[3536]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=159.89.231.172
Jul 26 05:29:40 SERVERNAME sshd[3534]: Failed password for mysql from 159.89.231.172 port 56352 ssh2
Jul 26 05:29:40 SERVERNAME sshd[3534]: Received disconnect from 159.89.231.172: 11: Normal Shutdown, Thank you for playing [preauth]
Jul 26 05:29:40 SERVERNAME sshd[3538]: reverse mapping checking getaddrinfo for usa1.getlark.com [159.89.231.172] failed - POSSIBLE BREAK-IN ATTEMPT!

Here is the script I created to implement the rules with comments:

          APPEND="sudo /sbin/iptables -A INPUT"
          INSERT="sudo /sbin/iptables -I INPUT"
          OUTPUT="sudo /sbin/iptables -A OUTPUT"
      # drop old rules and start from scratch
      sudo /sbin/iptables -F
      sudo /sbin/iptables -X

      # allow local host
      $INSERT -s 127.0.0.1 -j ACCEPT

      # Allow full access to our approved IPs first:
      $APPEND -s 209.xxx.xxx.1 -j ACCEPT
      $APPEND -s 209.xxx.xxx.2 -j ACCEPT
      $APPEND -s 209.xxx.xxx.3 -j ACCEPT
      $APPEND -s 209.xxx.xxx.4 -j ACCEPT
      $APPEND -s 209.xxx.xxx.5 -j ACCEPT
      $APPEND -s 209.xxx.xxx.6 -j ACCEPT
      $APPEND -s 216.xxx.xxx.1 -j ACCEPT
      $APPEND -s 216.xxx.xxx.2 -j ACCEPT
      $APPEND -s 74.xxx.xxx.2  -j ACCEPT


      # drop Null packets
      $APPEND -p tcp --tcp-flags ALL NONE -j DROP

      # block syn flood attack
             $APPEND -p tcp --syn -m limit --limit 1/s --limit-burst 2 -j RETURN

      # block recon/Xmas Packets
      $APPEND -p tcp --tcp-flags ALL ALL -j DROP

      # don’t lock me out if I screwed up:
      $APPEND -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

      # Allow/Block our legit services
      $APPEND -p tcp  --dport 80  -j ACCEPT
      $APPEND -p tcp  --dport 110 -j DROP
      $APPEND -p tcp  --dport 143 -j DROP
      $APPEND -p tcp  --dport 443 -j ACCEPT
      $APPEND -p tcp  --dport 465 -j DROP
      $APPEND -p tcp  --dport 993 -j DROP
      $APPEND -p tcp  --dport 995 -j DROP
      $APPEND -p tcp  --dport 2022 -j DROP


      # LAst Rule - Block everything else
      $APPEND -j REJECT --reject-with icmp-host-prohibited

Answer 1

Due to my noob reputation I need to ask clarification here.

Can you edit your question to contain also INPUT chain default policy? I ask because of this line:

RETURN   tcp  --  0.0.0.0/0   0.0.0.0/0   tcp flags:0x17/0x02 limit: avg 1/sec burst 2

I'm not an expert, but I have never seen this used in top level. Could be only me though. What is said about RETURN in iptables manpage:

RETURN means stop traversing this chain and resume at the next rule in the previous (calling) chain. If the end of a built-in chain is reached or a rule in a built-in chain with target RETURN is matched, the target specified by the chain policy determines the fate of the packet.

I have a hunch that your problem might have something to do with this functionality.

Answer 2

It's hard for me to give an outright answer given the provided data, but my approach would be to:

1. Trim down iptables down to just two rules: Allow a single IP address range (the one you are connecting from), and deny everything else. Be careful to not lock yourself out of the server! Having console access or a backup plan could be a life saver. If you are confident enough, here's an example on how to proceed. Proceed with caution!:

You can issue commands similar to:

sudo iptables-save > /tmp/original-rules.txt
sudo iptables -F 
sudo iptables -X
sudo iptables -A INPUT -s 209.xxx.xxx.1 -j ACCEPT
sudo iptables -A INPUT -j DROP

The above sequence of commands saves your existing rules, does a basic cleanup of all chains, adds a single IP address or address range to the INPUT chain, and adds a drop "any" rule to the INPUT chain.

Restoring the original rules can be achieved with:

sudo iptables-restore < /tmp/original-rules.txt

2. Test the two-rule setup:

2a. If it works, you know your problem lies somewhere in the original rules. Slowly keep adding rules and test them, until you are satisfied. You can also use logging to understand which rule is matched.

2b. If your server is still reachable while having only two rules from any other IP address besides the one(s) you allow, you then know it's not a rules issue, but something is up with iptables itself. Perhaps it's not loading correctly, or a previous state of iptables is running in memory.

Answer 3

Original poster accepted my earlier answer, but here is a bit more in-depth:

I diagnosed and debugged a bit. Here is the what I did first:

1. Installed Ubuntu 14.04 as VMware virtual machine
2. Modified /etc/ssh/sshd_config to read: Port 2022
3. Rebooted machine
4. Marked down VM IP address, which was 192.168.241.171

Next I took your firewall script and removed the xxx entries. I call the modified script "original" script.

I did a couple of reboot / script modifications / script run / Putty connect cycles. Here are scenarios and results:

• No modifications to "original" script: Putty connection to 192.168.241.171 port 2022 was successful
• Removed the stanza $APPEND -p tcp --syn -m limit --limit 1/s --limit-burst 2 -j RETURN from "original" script: Putty connection to 192.168.241.171 port 2022 was blocked
• Added before #allow local host" this: /sbin/iptables -P INPUT DROP` to the "original" script: Putty connection to 192.168.241.171 port 2022 was blocked
• Added before #allow local host" this: /sbin/iptables -P INPUT ACCEPT` to the "original" script: Putty connection to 192.168.241.171 port 2022 was successful

I think I know what has happened. The user forgot to set the INPUT chain default policy to DROP. Some examples about blocking syn flood using the RETURN statement expect default INPUT chain policy set to DROP and the user had adapted such example without setting the policy. I further speculate that the firewall script has never worked fully and what has been guarding sshd from external users is non-standard port 2022 (but now the bad guys are also knocking on 2022).

PS. If one changes the default policy, it should be done after work hours to negate immediate effects to people, then tested and adapted around new problems if any.