Hey Everybody,
I recently saw iftop showing a couple "connections" of ~200Kbps persistently on a box and because this wasn't the usual, I looked into it. Turns out it was caused by DNS lookups of type ANY ripe.net repeatedly. I can only assume this is an amplification attack. This box uses BIND 9.9.1-P3 is public facing and does recursive lookups (also authoritative). Now that that's out of the way, I'm looking/thinking of ways the prevent this obviously. This isn't causing a problem on a 100Mb link now but could get there quickly. As far as I know I don't have a lot of options, maybe iptables with some sort of limiting. ACLs would normally help, and would be perfect if I could get it to use a SQL database as the backend, and use that as a whitelist to at least mitigate the issue. If anyone has experience on the subject or an idea, it is much appreciated.
Regards, Paul
My understanding is that open DNS servers that allow unrestricted recursion are frowned upon these days, for the very reasons for which you've expressed concern. I believe best practice nowadays is to specify a limited set of subnets for which you allow recursion. For example, at the U of M we'll typically include something like the following in the "options" section of our named.conf files, to prevent recursive lookups from outside users:
allow-recursion { 140.193.0.0/16; 130.179.0.0/16; };
On 17/09/2012 2:16 PM, Paul Sierks wrote:
I recently saw iftop showing a couple "connections" of ~200Kbps persistently on a box and because this wasn't the usual, I looked into it. Turns out it was caused by DNS lookups of type ANY ripe.net repeatedly. I can only assume this is an amplification attack. This box uses BIND 9.9.1-P3 is public facing and does recursive lookups (also authoritative). Now that that's out of the way, I'm looking/thinking of ways the prevent this obviously. This isn't causing a problem on a 100Mb link now but could get there quickly. As far as I know I don't have a lot of options, maybe iptables with some sort of limiting. ACLs would normally help, and would be perfect if I could get it to use a SQL database as the backend, and use that as a whitelist to at least mitigate the issue. If anyone has experience on the subject or an idea, it is much appreciated.
On 2012-09-17 14:36, Gilles Detillieux wrote:
My understanding is that open DNS servers that allow unrestricted recursion are frowned upon these days, for the very reasons for which you've expressed concern. I believe best practice nowadays is to specify a limited set of subnets for which you allow recursion. For example, at the U of M we'll typically include something like the following in the "options" section of our named.conf files, to prevent recursive lookups from outside users:
allow-recursion { 140.193.0.0/16; 130.179.0.0/16; };
Another way of doing this, which would allow even more flexibility in configuring BIND for internal vs external access is to define two views:
view internal_resolver { match-clients { YOUR.SUB.NET.ADDR/CIDR; }; match-destinations { YOUR.SUB.NET.ADDR/CIDR; }; recursion yes; include "/etc/named.internal.zones"; };
view external_resolver { match-clients { any; }; recursion no; include "/etc/named.external.zones"; };
Another way of doing this, which would allow even more flexibility in configuring BIND for internal vs external access is to define two views:
view internal_resolver { match-clients { YOUR.SUB.NET.ADDR/CIDR; }; match-destinations { YOUR.SUB.NET.ADDR/CIDR; }; recursion yes; include "/etc/named.internal.zones"; };
view external_resolver { match-clients { any; }; recursion no; include "/etc/named.external.zones"; };
I agree this is a much better solution. It also allows a buy of flexibility for dealing with the same domain in different security domains (ie. serverA is 10/X inside and 4.5.6.7 outside). Keepin naming consistent across zones AND insuring internal resolvers for the same domain include external views as well is about 100 times easier to do with bind views.
-- Sean
On Mon, Sep 17, 2012 at 3:07 PM, Sean Cody sean@tinfoilhat.ca wrote:
I agree this is a much better solution. It also allows a buy of flexibility for dealing with the same domain in different security domains (ie. serverA is 10/X inside and 4.5.6.7 outside). Keepin naming consistent across zones AND insuring internal resolvers for the same domain include external views as well is about 100 times easier to do with bind views.
Technically yes it's a sexier solution, but I can not recall a time where I've let a machine have the same name with different IP addresses and ended up not regretting it. Some time down the line we'd find some device or third party that used their own resolver but had a VPN to us, or a DMZ that was just a bit more special with NAT rules than the others, and then we were fighting between various groups as to who had to solve it with which hack. This gets even more fun when only one of the groups can explain what's going on.
BIND views fail my "3am test". After being woken up at 3am I don't want to be thinking about where I have to query from in order to get the "right" answer. You've put yourself in a situation where two people can get different answers for the same question.
As long as you don't take Gilbert's suggestion any further it's good, maybe even clever. But then again, it's no different than Gilles', just more indirect.
Please, for your own sanity, if you want the same machine to be visible at multiple addresses depending on the context, use different zones. Your network guys, sysadmins, and developers can all agree that serverx.example.local is different than serverx.example.com.
Sean
On 17/09/2012 3:07 PM, Sean Cody wrote:
Another way of doing this, which would allow even more flexibility in configuring BIND for internal vs external access is to define two views:
view internal_resolver { match-clients { YOUR.SUB.NET.ADDR/CIDR; }; match-destinations { YOUR.SUB.NET.ADDR/CIDR; }; recursion yes; include "/etc/named.internal.zones"; };
view external_resolver { match-clients { any; }; recursion no; include "/etc/named.external.zones"; };
I agree this is a much better solution. It also allows a buy of flexibility for dealing with the same domain in different security domains (ie. serverA is 10/X inside and 4.5.6.7 outside). Keepin naming consistent across zones AND insuring internal resolvers for the same domain include external views as well is about 100 times easier to do with bind views.
In which version of BIND were views introduced? I wasn't aware of this feature, and I can think of something I implemented this spring that would likely have been MUCH easier to do with this feature. Does bind use the first matching view for a given client address, in cases where an address could match multiple "match-client" patterns?
Sorry for any confusion, of which I'm sure I'm about to add to. But this particular box doesn't have an internal network, just one interface on the internet. Also I think a lot of the problem in my case is the allowed IP addresses change on a regular basis, quite often.
On Mon, Sep 17, 2012 at 3:28 PM, Paul Sierks psierks@sierkstech.net wrote:
Sorry for any confusion, of which I'm sure I'm about to add to. But this particular box doesn't have an internal network, just one interface on the internet. Also I think a lot of the problem in my case is the allowed IP addresses change on a regular basis, quite often.
Then I think we're back at Gille's original response -- don't do it! :) There are many better public DNS servers out there, such as Google/s 8.8.8.8 and 8.8.4.4.
Failing that, mitigate the risk with an iptables filter to prevent your host from being the source of the DDOS.
Sean
On 2012-09-17 15:31, Sean Walberg wrote:
On Mon, Sep 17, 2012 at 3:28 PM, Paul Sierks <psierks@sierkstech.net mailto:psierks@sierkstech.net> wrote:
Sorry for any confusion, of which I'm sure I'm about to add to. But this particular box doesn't have an internal network, just one interface on the internet. Also I think a lot of the problem in my case is the allowed IP addresses change on a regular basis, quite often.
Paul, are you saying that your "allowed" IP addresses are just out there on the Internet at large, and not on an internal network? In that case, I'd have to agree with Sean:
Then I think we're back at Gille's original response -- don't do it! :) There are many better public DNS servers out there, such as Google/s 8.8.8.8 and 8.8.4.4.
Failing that, mitigate the risk with an iptables filter to prevent your host from being the source of the DDOS.
That would be a good strategy, but you have to set this up carefully to make sure you're not interfering with normal DNS activity. You might be able to cobble something together, e.g. using the "recent" module, but setting thresholds might be tricky.
Sean, do you have a working iptables example that you've used? I've used the "recent" module on services like SSH, POP, and IMAP, but not for DNS.
On 2012-09-17 15:31, Sean Walberg wrote:
On Mon, Sep 17, 2012 at 3:28 PM, Paul Sierks <psierks@sierkstech.net mailto:psierks@sierkstech.net> wrote:
Sorry for any confusion, of which I'm sure I'm about to add to. But this particular box doesn't have an internal network, just one interface on the internet. Also I think a lot of the problem in my case is the allowed IP addresses change on a regular basis, quiteoften.
Paul, are you saying that your "allowed" IP addresses are just out there on the Internet at large, and not on an internal network? In that case, I'd have to agree with Sean:
Yes, Exactly.
Then I think we're back at Gille's original response -- don't do it! :) There are many better public DNS servers out there, such as Google/s 8.8.8.8 and 8.8.4.4.
Failing that, mitigate the risk with an iptables filter to prevent your host from being the source of the DDOS.
That would be a good strategy, but you have to set this up carefully to make sure you're not interfering with normal DNS activity. You might be able to cobble something together, e.g. using the "recent" module, but setting thresholds might be tricky.
Sean, do you have a working iptables example that you've used? I've used the "recent" module on services like SSH, POP, and IMAP, but not for DNS.
Ok, I will have to get a rule or two put together, if there's anyone that uses one already that wouldn't mind sharing that would be great too. I'm also curious now how the other open resolvers on the internet do it safely :S
On Mon, Sep 17, 2012 at 3:55 PM, Gilbert E. Detillieux < gedetil@cs.umanitoba.ca> wrote:
Sean, do you have a working iptables example that you've used? I've used the "recent" module on services like SSH, POP, and IMAP, but not for DNS.
No, I've always avoided the problem by using someone else's servers or ACL'ing things to my network.
-m recent is how I'd start, too. Just log the violations instead of dropping them to start.
Depending on what the impact would be to your network, policing/shaping your DNS traffic to an arbitrary limit might also work. Could be done with iptables, the Linux shaper, or an upstream router.
Sean
-
Gilbert E. Detillieux -
Gilles Detillieux -
Paul Sierks -
Sean Cody -
Sean Walberg