[RndTbl] very strange DNS errors

Trevor Cordes trevor at tecnopolis.ca
Wed Apr 20 02:03:59 CDT 2016


Some recurring email bounces have tipped me off to a really strange DNS 
issue on the boxes I admin.  I really need help on this as it's impacting 
a real customer in a real way: delayed emails.

Often, and somewhat randomly, DNS will fail to resolve a domain.  
Sometimes it will fail to resolve it for >4 hours and trigger a diagnostic 
bounce from my sendmail.

When I run tests at the command line (dig) on the domains in question (the 
ones I've seen in email bounces) I will often very quickly get a resolve 
failure, but usually 5s later another dig to the same domain will resolve 
100% ok!

Every box I am testing on has a similar config with BIND named 9.10 (9.8 
on one box) running as the local recursive resolver.  /etc/resolv.conf on 
all is 127.0.0.1.  So that means every lookup that isn't cached is going 
to the root NS's.

When it fails to resolve, named.log logs an entry like:
20-Apr-2016 00:37:28.276 query-errors: debug 1: client 127.0.0.1#33971 (artscouncil.mb.ca): query failed (SERVFAIL) for artscouncil.mb.ca/IN/A at query.c:7769
20-Apr-2016 00:37:28.276 query-errors: debug 2: fetch completed at resolver.c:3658 for artscouncil.mb.ca/A in 0.215778: SERVFAIL/success [domain:artscouncil.mb.ca,referral:1,restart:3,qrysent:2,timeout:0,lame:0,neterr:2,badresp:0,adberr:0,findfail:0,valfail:0]

For manual tests it only logs one, for the real-life sendmail problem 
ones, I'll see dozen/hundreds of the same thing trying hour after hour 
(usually around the sendmail queue retry times).

One of my boxes is *extremely* well connected in the US, and while it 
seems to have errors slightly less often, it still has them.  All the rest 
are on various levels of Shaw or MTS, res or business.

This seems to have just started popping up maybe 6 months ago.  It feels 
like it's getting worse.

I've setup an easy test, on the actual domains with the most problems:

rndc flush
dig +short sportmanitoba.ca
dig +short gymcan.org
dig +short brandoneagles.ca
dig +short interactivegym.org
dig +short artscouncil.mb.ca

rndc flush
dig +trace sportmanitoba.ca
dig +trace gymcan.org
dig +trace brandoneagles.ca
dig +trace interactivegym.org
dig +trace artscouncil.mb.ca

Maybe some others can run those tests on their boxes (but only if you're 
running BIND as caching resolver, which many/most people won't be).

Here's where it gets interesting the +short tests I can get to fail at 
least 1 of the domains (at random) about 1/8 of the time!  On at least 5 
different boxes out there!  But +trace has never failed once on any box or 
any domain.  It's like +trace does something different, maybe slowing the 
process down or something, that allows it to always succeed.  (Failure for 
the +short is a missing line in output, +trace you have to look at the 
domain/ip returned near the bottom.)

So, is it just these particular domains??  Something wrong on their (DNS) 
side?  Or is it more domains, not just these?  Is there any way to 
diagnose what exactly is failing?  I find it bizarre that *all* of these 
domains regularly go down for 4+ hours causing an email bounce!?!  Or is 
there something horribly wrong on my BIND caching DNS servers?

Perhaps they are slow, and I'm my BINDs are just not waiting long enough.  
Is there a way to tell BIND to be more patient waiting for DNS packets to 
come in?

Maybe it's something regarding IPv6?  (I'm doing this all in IPv4 and have 
no current interest in 6.  And I'm only looking for A records.)

I have packet traces of the above sample commands when the lookups fail, 
but I can't really figure out what it's doing, other than one boatload of 
traffic for a tiny dns query.  I can provide a trace privately on demand 
if you think you can help.

Even more odd, I never seem to have a problem with interactive things like 
web browsing.  If this was a problem with all domains, I should see this 
in firefox all the time, but I don't.  Maybe Firefox doesn't even obey 
resolv.conf and does its own thing, or retries heavily itself?

I also checked to ensure my iptables aren't dropping packets related to 
this.

Lastly, answers of "just use 8.8.8.8" aren't helpful because I also need 
to handle dynamic local, and in some cases, external DNS (often with 
multiple views), all in the same BIND/box (and I like uniformity across 
boxes for ease of admin).  Sure, I could try another resolver, but I see 
no reason BIND can't be made to work, as it has for me for 20 years.  And 
if this is a BIND bug, I want to submit it to help solve it.

Thanks guys!


More information about the Roundtable mailing list