I'm hoping someone suggest a fix for this.
We moved some applications over to a new server which still had the default firewall rules in place which included a rate limiting "drop" rule that looks like this:
iptables -A input_ext -m limit --limit 3/min -m conntrack --ctstate NEW -j LOG --log-prefix "SFW2-INext-DROP-DEFLT " --log-tcp-options --log-ip-options
In short, limit to 3 new connections per minute.
It turns out this was way to short for our application and so I just removed all firewall rules by stopping the firewal (this is on OpenSUSE).
The last firewall log message indicates that a packet was being dropped to a specific IP due to a rate limit but now the server will not send packets to that IP at all! tcpdump shows that the packets are not even attempting to leave the interface.
It seems like netfilter blocked the ip on the rate limit rule and now its "stuck".
I tried specifically allowing that IP and even recreated the limit rule thinking that would "reactivate" the chain but it doesn't work.
My guess is that a reboot would fix it but the server is in production and can not be rebooted without a scheduled outage.
The only other thing I can think of is to reload all of the netfilter kernel modules but again that is too risky on a production system.
Any other ideas on how to clear the filter?
Is there a command to display the current status of what netfilter is tracking and dropping?