I'm hoping someone suggest a fix for this. We moved some applications over to a new server which still had the default firewall rules in place which included a rate limiting "drop" rule that looks like this: iptables -A input_ext -m limit --limit 3/min -m conntrack --ctstate NEW -j LOG --log-prefix "SFW2-INext-DROP-DEFLT " --log-tcp-options --log-ip-options In short, limit to 3 new connections per minute. It turns out this was way to short for our application and so I just removed all firewall rules by stopping the firewal (this is on OpenSUSE). The last firewall log message indicates that a packet was being dropped to a specific IP due to a rate limit but now the server will not send packets to that IP at all! tcpdump shows that the packets are not even attempting to leave the interface. It seems like netfilter blocked the ip on the rate limit rule and now its "stuck". I tried specifically allowing that IP and even recreated the limit rule thinking that would "reactivate" the chain but it doesn't work. My guess is that a reboot would fix it but the server is in production and can not be rebooted without a scheduled outage. The only other thing I can think of is to reload all of the netfilter kernel modules but again that is too risky on a production system. Any other ideas on how to clear the filter? Is there a command to display the current status of what netfilter is tracking and dropping? -- John Lange
Hi John It's been a long time since I worked with iptables, but one thing that used to trip me up is forgetting to explicitly flush the tables. I eventually wrote a script for this: #!/bin/bash # # iptables.init script # flushes all tables, zeroes counters, resets policies # # Dan Martin University of Manitoba 0599441 # for 74.757 Advanced Networking # IPTABLES="/sbin/iptables" # iptables.init script # modules loaded and tables flushed echo "Previous iptables" echo $IPTABLES -L --line-numbers -v echo $IPTABLES -t nat -L --line-numbers -v echo $IPTABLES -t mangle -L --line-numbers -v echo # # $IPTABLES -F $IPTABLES -t nat -F $IPTABLES -t mangle -F $IPTABLES -X $IPTABLES -Z $IPTABLES -t nat -Z $IPTABLES -t mangle -Z $IPTABLES -P INPUT ACCEPT $IPTABLES -P OUTPUT ACCEPT $IPTABLES -P FORWARD ACCEPT # # echo "Cleaned iptables" echo $IPTABLES -L --line-numbers -v echo $IPTABLES -t nat -L --line-numbers -v echo $IPTABLES -t mangle -L --line-numbers -v echo On Wed, Jan 11, 2012 at 1:50 PM, John Lange <john@johnlange.ca> wrote:
I'm hoping someone suggest a fix for this.
We moved some applications over to a new server which still had the default firewall rules in place which included a rate limiting "drop" rule that looks like this:
iptables -A input_ext -m limit --limit 3/min -m conntrack --ctstate NEW -j LOG --log-prefix "SFW2-INext-DROP-DEFLT " --log-tcp-options --log-ip-options
In short, limit to 3 new connections per minute.
It turns out this was way to short for our application and so I just removed all firewall rules by stopping the firewal (this is on OpenSUSE).
The last firewall log message indicates that a packet was being dropped to a specific IP due to a rate limit but now the server will not send packets to that IP at all! tcpdump shows that the packets are not even attempting to leave the interface.
It seems like netfilter blocked the ip on the rate limit rule and now its "stuck".
I tried specifically allowing that IP and even recreated the limit rule thinking that would "reactivate" the chain but it doesn't work.
My guess is that a reboot would fix it but the server is in production and can not be rebooted without a scheduled outage.
The only other thing I can think of is to reload all of the netfilter kernel modules but again that is too risky on a production system.
Any other ideas on how to clear the filter?
Is there a command to display the current status of what netfilter is tracking and dropping?
-- John Lange _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
-- Dan Martin, MD GP Hospital Practitioner Computer Scientist ummar143@shaw.ca (204) 831-1746 answering machine always on
On 2012-01-11 13:50, John Lange wrote:
I'm hoping someone suggest a fix for this.
We moved some applications over to a new server which still had the default firewall rules in place which included a rate limiting "drop" rule that looks like this:
iptables -A input_ext -m limit --limit 3/min -m conntrack --ctstate NEW -j LOG --log-prefix "SFW2-INext-DROP-DEFLT " --log-tcp-options --log-ip-options
In short, limit to 3 new connections per minute.
It turns out this was way to short for our application and so I just removed all firewall rules by stopping the firewal (this is on OpenSUSE).
The last firewall log message indicates that a packet was being dropped to a specific IP due to a rate limit but now the server will not send packets to that IP at all! tcpdump shows that the packets are not even attempting to leave the interface.
It seems like netfilter blocked the ip on the rate limit rule and now its "stuck".
I tried specifically allowing that IP and even recreated the limit rule thinking that would "reactivate" the chain but it doesn't work.
My guess is that a reboot would fix it but the server is in production and can not be rebooted without a scheduled outage.
The only other thing I can think of is to reload all of the netfilter kernel modules but again that is too risky on a production system.
Any other ideas on how to clear the filter?
Have you tried this "iptables" option?... -Z, --zero [chain] Zero the packet and byte counters in all chains. It is legal to specify the -L, --list (list) option as well, to see the coun- ters immediately before they are cleared. (See above.)
Is there a command to display the current status of what netfilter is tracking and dropping?
I don't know if this will give you the information you need... -L, --list [chain] List all rules in the selected chain. If no chain is selected, all chains are listed. As every other iptables command, it applies to the specified table (filter is the default), so NAT rules get listed by iptables -t nat -n -L Please note that it is often used with the -n option, in order to avoid long reverse DNS lookups. It is legal to specify the -Z (zero) option as well, in which case the chain(s) will be atomically listed and zeroed. The exact output is affected by the other arguments given. The exact rules are suppressed until you use iptables -L -v Some information is also available under /proc/net, such as /proc/net/ip_conntrack, which shows all connections being tracked. I don't know where the limit counters are available, though. -- Gilbert E. Detillieux E-mail: <gedetil@muug.mb.ca> Manitoba UNIX User Group Web: http://www.muug.mb.ca/ PO Box 130 St-Boniface Phone: (204)474-8161 Winnipeg MB CANADA R2H 3B4 Fax: (204)474-7609
Thanks guys. I tried the -Z and the "-t nat -F" and "-t mangle -F" (there was no natting or mangling going on but worth a try) but still the packets won't exit the interface... John
Have you looked at /proc/net/nf_conntrack ? Sean On Wed, Jan 11, 2012 at 1:50 PM, John Lange <john@johnlange.ca> wrote:
I'm hoping someone suggest a fix for this.
We moved some applications over to a new server which still had the default firewall rules in place which included a rate limiting "drop" rule that looks like this:
iptables -A input_ext -m limit --limit 3/min -m conntrack --ctstate NEW -j LOG --log-prefix "SFW2-INext-DROP-DEFLT " --log-tcp-options --log-ip-options
In short, limit to 3 new connections per minute.
It turns out this was way to short for our application and so I just removed all firewall rules by stopping the firewal (this is on OpenSUSE).
The last firewall log message indicates that a packet was being dropped to a specific IP due to a rate limit but now the server will not send packets to that IP at all! tcpdump shows that the packets are not even attempting to leave the interface.
It seems like netfilter blocked the ip on the rate limit rule and now its "stuck".
I tried specifically allowing that IP and even recreated the limit rule thinking that would "reactivate" the chain but it doesn't work.
My guess is that a reboot would fix it but the server is in production and can not be rebooted without a scheduled outage.
The only other thing I can think of is to reload all of the netfilter kernel modules but again that is too risky on a production system.
Any other ideas on how to clear the filter?
Is there a command to display the current status of what netfilter is tracking and dropping?
-- John Lange _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
-- Sean Walberg <sean@ertw.com> http://ertw.com/
I had looked at it.... But your suggestion made me google just "nf_conntrack" which lead me to "conntrack-tools" "that allow system administrators interact with the Connection Tracking System," Now we are getting somewhere... I can see that there is state information for that IP address but sadly, deleting all the state information for that IP (conntrack -D -s xxx.xxx.xxx.xxx) did not solve the problem. There must be something deeper going on. Perhaps the limit stuff was just a red herring... The state tracking looks normal. John
Is it possible to just unload (rmmod) the iptables modules? On Wed, Jan 11, 2012 at 4:17 PM, John Lange <john@johnlange.ca> wrote:
I had looked at it.... But your suggestion made me google just "nf_conntrack" which lead me to "conntrack-tools" "that allow system administrators interact with the Connection Tracking System,"
Now we are getting somewhere...
I can see that there is state information for that IP address but sadly, deleting all the state information for that IP (conntrack -D -s xxx.xxx.xxx.xxx) did not solve the problem. There must be something deeper going on. Perhaps the limit stuff was just a red herring... The state tracking looks normal.
John _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
On 2012-01-11 John Lange wrote:
iptables -A input_ext -m limit --limit 3/min -m conntrack --ctstate NEW -j LOG --log-prefix "SFW2-INext-DROP-DEFLT " --log-tcp-options --log-ip-options
In short, limit to 3 new connections per minute.
Better late than never... here's my 2c (I'd consider myself a netfilter wizard, toot toot) If the rules mostly work except for the small limits, just up the limit for that 1 rule. Set it to like 100-1000 should be ok and still stop floods. If this box is behind a firewall (ie: not exposed to the internet) then deleting the rule completely should be safe.
It seems like netfilter blocked the ip on the rate limit rule and now its "stuck".
Nah, it doesn't do that. As soon as you flush the rules, they are gone. Any conntrack remaining would timeout and disappear as a limitation. Must be something else. I would strongly suspect that your default policies are set to DROP? You could try setting them to ACCEPT. Dan Martin's script had this near the end. If you're still stuck, send your output from: iptables -L -n -v
They were stuck. I flushed all the rules and set the policies to ACCEPT and still contrack was tracking connections and that IP was getting blocked somehow. I suspect unloading the netfilter contrack modules would have resolved it but I wasn't willing to do that until off hours and since I had to wait until off hours I just rebooted to be on the safe side and the problem went away and has not returned. I'll probably never know exactly what went wrong but I definitely will be more careful in the future when removing rules. John
participants (6)
-
Dan Martin -
Gilbert E. Detillieux -
John Lange -
Kevin McGregor -
Sean Walberg -
Trevor Cordes