Yikes, even weirder than I thought. It's not a cap issue. I put some debugs in it and it's getting around 30-220 pings out the door before it randomly fails on one of them with the "Operation not permitted" error.
If I flush and reload my iptables then it sometimes gets through all 254 pings I'm doing (it's my own subnet, I'm allowed to scan!). After that first scan, the count it allows seems to reduce and then fluctuate.
I checked any iptables rules that could apply and for kicks I changed everywhere I did icmp rate limiting on type 8 (echo) to unlimited (just -j ACCEPT). I also cleared the iptables counters, ran my test, and checked what rules were incrementing and nothing was standing out, and the ones that did increment I got rid of and the problem remains.
Sure feels like an iptable rate-limiting thing, but I just can't spot the problem after extensive checking of my rules and their counters. Besides, I do almost no iptables on OUTPUT table, and default on OUTPUT is ACCEPT.
It's like some other rate limiter has been put into the kernel.
If I add to the script before the send() call a sleep(.002) (hires sleep enabled in perl) then the bug doesn't occur! Works every time! If I set it to sleep(.001) then the bug starts to occur again about half the time. So something got introduced in the newest kernel that is limiting me to around 500s send() calls.
I'll see if I can get an internet-unplugged, iptables completely off test in soon to completely eliminate iptables: tough because I am accessing this (offsite) test box via ssh at the moment.
Anyone aware of anything else/new that could cause this? Anything else in the kernel that rate limits stuff like this? Other ideas to test to rule out? Like I said, this bug does not occur in 4.7.10-100.fc23, just 4.8.8-100.fc23. I hope I don't have to git bisect to solve this one!
Thanks! (Sorry for the copious emails!)