On 2012-01-11 13:50, John Lange wrote:

I'm hoping someone suggest a fix for this.

We moved some applications over to a new server which still had the default firewall rules in place which included a rate limiting "drop" rule that looks like this:

iptables -A input_ext -m limit --limit 3/min -m conntrack --ctstate NEW -j LOG --log-prefix "SFW2-INext-DROP-DEFLT " --log-tcp-options --log-ip-options

In short, limit to 3 new connections per minute.

It turns out this was way to short for our application and so I just removed all firewall rules by stopping the firewal (this is on OpenSUSE).

The last firewall log message indicates that a packet was being dropped to a specific IP due to a rate limit but now the server will not send packets to that IP at all! tcpdump shows that the packets are not even attempting to leave the interface.

It seems like netfilter blocked the ip on the rate limit rule and now its "stuck".

I tried specifically allowing that IP and even recreated the limit rule thinking that would "reactivate" the chain but it doesn't work.

My guess is that a reboot would fix it but the server is in production and can not be rebooted without a scheduled outage.

The only other thing I can think of is to reload all of the netfilter kernel modules but again that is too risky on a production system.

Any other ideas on how to clear the filter?

Have you tried this "iptables" option?...

-Z, --zero [chain] Zero the packet and byte counters in all chains. It is legal to specify the -L, --list (list) option as well, to see the coun- ters immediately before they are cleared. (See above.)

Is there a command to display the current status of what netfilter is tracking and dropping?

I don't know if this will give you the information you need...

-L, --list [chain] List all rules in the selected chain. If no chain is selected, all chains are listed. As every other iptables command, it applies to the specified table (filter is the default), so NAT rules get listed by iptables -t nat -n -L Please note that it is often used with the -n option, in order to avoid long reverse DNS lookups. It is legal to specify the -Z (zero) option as well, in which case the chain(s) will be atomically listed and zeroed. The exact output is affected by the other arguments given. The exact rules are suppressed until you use iptables -L -v

Some information is also available under /proc/net, such as /proc/net/ip_conntrack, which shows all connections being tracked. I don't know where the limit counters are available, though.

Have you looked at /proc/net/nf_conntrack ?

Sean

On Wed, Jan 11, 2012 at 1:50 PM, John Lange john@johnlange.ca wrote:

I'm hoping someone suggest a fix for this.

We moved some applications over to a new server which still had the default firewall rules in place which included a rate limiting "drop" rule that looks like this:

iptables -A input_ext -m limit --limit 3/min -m conntrack --ctstate NEW -j LOG --log-prefix "SFW2-INext-DROP-DEFLT " --log-tcp-options --log-ip-options

In short, limit to 3 new connections per minute.

It turns out this was way to short for our application and so I just removed all firewall rules by stopping the firewal (this is on OpenSUSE).

The last firewall log message indicates that a packet was being dropped to a specific IP due to a rate limit but now the server will not send packets to that IP at all! tcpdump shows that the packets are not even attempting to leave the interface.

It seems like netfilter blocked the ip on the rate limit rule and now its "stuck".

I tried specifically allowing that IP and even recreated the limit rule thinking that would "reactivate" the chain but it doesn't work.

My guess is that a reboot would fix it but the server is in production and can not be rebooted without a scheduled outage.

The only other thing I can think of is to reload all of the netfilter kernel modules but again that is too risky on a production system.

Any other ideas on how to clear the filter?

Is there a command to display the current status of what netfilter is tracking and dropping?

-- John Lange _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable

On 2012-01-11 John Lange wrote:

iptables -A input_ext -m limit --limit 3/min -m conntrack --ctstate NEW -j LOG --log-prefix "SFW2-INext-DROP-DEFLT " --log-tcp-options --log-ip-options

In short, limit to 3 new connections per minute.

Better late than never... here's my 2c (I'd consider myself a netfilter wizard, toot toot)

If the rules mostly work except for the small limits, just up the limit for that 1 rule. Set it to like 100-1000 should be ok and still stop floods. If this box is behind a firewall (ie: not exposed to the internet) then deleting the rule completely should be safe.

It seems like netfilter blocked the ip on the rate limit rule and now its "stuck".

Nah, it doesn't do that. As soon as you flush the rules, they are gone. Any conntrack remaining would timeout and disappear as a limitation. Must be something else.

I would strongly suspect that your default policies are set to DROP? You could try setting them to ACCEPT. Dan Martin's script had this near the end.

If you're still stuck, send your output from: iptables -L -n -v