I don't think what you are seeing is normal and to me it's all hinting at something local. I feel that there is something common to all your setups causing this. I don't think it's the upstream DNS providers.

One thing that pops to mind is UDP packet fragmentation. Perhaps there is something in the network setup or filtering (iptables) which is causing UDP packets to fragment but is dropping the second part of the fragment? This is a surprisingly common problem on a lot of firewalls, for example Sonicwall. Perhaps force dig to use TCP to see if the results are different (dig +tcp <host>).

That's just one possibility among many. Swap out one of the machines for a totally different system (Windows laptop maybe?) and repeat the tests. If you don't get the same failure rate while on the same connection, then you know it's your setup.

John

On Wed, Apr 20, 2016 at 5:42 PM, Trevor Cordes <trevor@tecnopolis.ca> wrote:

Further, during additional tests while capturing with tcpdump and
visualizing with wireshark, tcpdump consistently tells me it is
dropping 10-45 packets out of around 1200 it seems to capture to dig
+short just 2 domains in a 5s test. Is that level of drop normal? I
couldn't get a single capture that didn't have drops. Is this just
tcpdump not keeping up, or are these drops also not making it to BIND
itself?? (Though I tested and the same # of drops occur on the "good"
domains too.)

Also, to rule out iptables I added an accept all rule for --sport 53
(udp and tcp) super early in my external interface chain. Didn't help
one bit. But I'm now 99% sure it's not iptables.

I found 4 surprising things in wireshark:

1. After rndc flushing it takes about 600 packets to resolve one dinky
domain name?? Wow!

2. AAAA records are coming across the wire, in fact they outnumber A
records. I have ipv6 as "turned off"/blocked on a modern linux box as
you can, so I'm not sure why AAAA is showing up, but I guess it's
neither here nor there as long as the A's are coming back ok. They
certainly add a bucket of useless packets to the 600 total though.

3. Looks like I'm getting back (must be automatic) dns sec stuff in
some of these packets. I don't have any of that configured in my BIND
config, so unless it "just works" with no new config lines, it probably
is just being ignored.

4. There's a fair amount of TCP port 53 traffic going on! My guess
would have been it was all limited to UDP. Guess I'm behind the
times...

Not sure if any of these revelations sheds any light.
_______________________________________________

Roundtable mailing list
Roundtable@muug.mb.ca
http://www.muug.mb.ca/mailman/listinfo/roundtable

John Lange
www.johnlange.ca