Some recurring email bounces have tipped me off to a really strange DNS issue on the boxes I admin. I really need help on this as it's impacting a real customer in a real way: delayed emails.
Often, and somewhat randomly, DNS will fail to resolve a domain. Sometimes it will fail to resolve it for >4 hours and trigger a diagnostic bounce from my sendmail.
When I run tests at the command line (dig) on the domains in question (the ones I've seen in email bounces) I will often very quickly get a resolve failure, but usually 5s later another dig to the same domain will resolve 100% ok!
Every box I am testing on has a similar config with BIND named 9.10 (9.8 on one box) running as the local recursive resolver. /etc/resolv.conf on all is 127.0.0.1. So that means every lookup that isn't cached is going to the root NS's.
When it fails to resolve, named.log logs an entry like: 20-Apr-2016 00:37:28.276 query-errors: debug 1: client 127.0.0.1#33971 (artscouncil.mb.ca): query failed (SERVFAIL) for artscouncil.mb.ca/IN/A at query.c:7769 20-Apr-2016 00:37:28.276 query-errors: debug 2: fetch completed at resolver.c:3658 for artscouncil.mb.ca/A in 0.215778: SERVFAIL/success [domain:artscouncil.mb.ca,referral:1,restart:3,qrysent:2,timeout:0,lame:0,neterr:2,badresp:0,adberr:0,findfail:0,valfail:0]
For manual tests it only logs one, for the real-life sendmail problem ones, I'll see dozen/hundreds of the same thing trying hour after hour (usually around the sendmail queue retry times).
One of my boxes is *extremely* well connected in the US, and while it seems to have errors slightly less often, it still has them. All the rest are on various levels of Shaw or MTS, res or business.
This seems to have just started popping up maybe 6 months ago. It feels like it's getting worse.
I've setup an easy test, on the actual domains with the most problems:
rndc flush dig +short sportmanitoba.ca dig +short gymcan.org dig +short brandoneagles.ca dig +short interactivegym.org dig +short artscouncil.mb.ca
rndc flush dig +trace sportmanitoba.ca dig +trace gymcan.org dig +trace brandoneagles.ca dig +trace interactivegym.org dig +trace artscouncil.mb.ca
Maybe some others can run those tests on their boxes (but only if you're running BIND as caching resolver, which many/most people won't be).
Here's where it gets interesting the +short tests I can get to fail at least 1 of the domains (at random) about 1/8 of the time! On at least 5 different boxes out there! But +trace has never failed once on any box or any domain. It's like +trace does something different, maybe slowing the process down or something, that allows it to always succeed. (Failure for the +short is a missing line in output, +trace you have to look at the domain/ip returned near the bottom.)
So, is it just these particular domains?? Something wrong on their (DNS) side? Or is it more domains, not just these? Is there any way to diagnose what exactly is failing? I find it bizarre that *all* of these domains regularly go down for 4+ hours causing an email bounce!?! Or is there something horribly wrong on my BIND caching DNS servers?
Perhaps they are slow, and I'm my BINDs are just not waiting long enough. Is there a way to tell BIND to be more patient waiting for DNS packets to come in?
Maybe it's something regarding IPv6? (I'm doing this all in IPv4 and have no current interest in 6. And I'm only looking for A records.)
I have packet traces of the above sample commands when the lookups fail, but I can't really figure out what it's doing, other than one boatload of traffic for a tiny dns query. I can provide a trace privately on demand if you think you can help.
Even more odd, I never seem to have a problem with interactive things like web browsing. If this was a problem with all domains, I should see this in firefox all the time, but I don't. Maybe Firefox doesn't even obey resolv.conf and does its own thing, or retries heavily itself?
I also checked to ensure my iptables aren't dropping packets related to this.
Lastly, answers of "just use 8.8.8.8" aren't helpful because I also need to handle dynamic local, and in some cases, external DNS (often with multiple views), all in the same BIND/box (and I like uniformity across boxes for ease of admin). Sure, I could try another resolver, but I see no reason BIND can't be made to work, as it has for me for 20 years. And if this is a BIND bug, I want to submit it to help solve it.
Thanks guys!