[RndTbl] Roundtable Digest, Vol 69, Issue 11
Brock Wolfe
obwolfe at shaw.ca
Sat Sep 18 20:09:04 CDT 2010
I prefer the DROP packet by IP ADDRESS solution. Simple and to the
point. Worked really well for me on abusive crawlers from china and
russia. 8-)
Brock
On 9/18/2010 12:00 PM, roundtable-request at muug.mb.ca wrote:
> Send Roundtable mailing list submissions to
> roundtable at muug.mb.ca
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://www.muug.mb.ca/mailman/listinfo/roundtable
> or, via email, send a message with subject or body 'help' to
> roundtable-request at muug.mb.ca
>
> You can reach the person managing the list at
> roundtable-owner at muug.mb.ca
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Roundtable digest..."
>
>
> Today's Topics:
>
> 1. slowing httpd access to cgi-bin scripts (Gilles Detillieux)
> 2. Re: slowing httpd access to cgi-bin scripts
> (Helgi Hrafn Gunnarsson)
> 3. Re: slowing httpd access to cgi-bin scripts (Sean Walberg)
> 4. Re: slowing httpd access to cgi-bin scripts (Tim Lavoie)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 17 Sep 2010 16:08:00 -0500
> From: Gilles Detillieux<grdetil at scrc.umanitoba.ca>
> Subject: [RndTbl] slowing httpd access to cgi-bin scripts
> To: MUUG Roundtable<roundtable at muug.mb.ca>
> Message-ID:<4C93D8B0.2010708 at scrc.umanitoba.ca>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Every once in a while, some doofus points a web crawler at our web site
> and, ignoring the disallowed areas in our robots.txt file, starts
> crawling through some of our cgi-bin scripts at a rate of 4 to 8 hits a
> second. This is particularly annoying with some of the more processor
> and disk intensive CGI programs, such as man2html, which also happens to
> generate lots of links back to itself.
>
> Is there anything I can set up in Apache to throttle back and slow down
> remote hosts when they start hitting hard on cgi-bin? I don't want to
> do anything that would adversely affect legitimate users, nor make
> important things like the manual pages hard to find by removing any
> public links to them. But when a client starts making 10 or more GET
> requests on /cgi-bin in a 5 second period, it would be nice if I could
> get the server to progressively add longer and longer delays before
> servicing these requests, to keep the load down and prevent the server
> from thrashing.
>
> I'd appreciate any tips.
>
> Thanks,
> Gilles
>
More information about the Roundtable
mailing list