[RndTbl] Roundtable Digest, Vol 69, Issue 11

Brock Wolfe obwolfe at shaw.ca
Sat Sep 18 20:09:04 CDT 2010


  I prefer the DROP packet by IP ADDRESS solution.  Simple and to the 
point.  Worked really well for me on abusive crawlers from china and 
russia.  8-)

Brock

On 9/18/2010 12:00 PM, roundtable-request at muug.mb.ca wrote:
> Send Roundtable mailing list submissions to
> 	roundtable at muug.mb.ca
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	http://www.muug.mb.ca/mailman/listinfo/roundtable
> or, via email, send a message with subject or body 'help' to
> 	roundtable-request at muug.mb.ca
>
> You can reach the person managing the list at
> 	roundtable-owner at muug.mb.ca
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Roundtable digest..."
>
>
> Today's Topics:
>
>     1. slowing httpd access to cgi-bin scripts (Gilles Detillieux)
>     2. Re: slowing httpd access to cgi-bin scripts
>        (Helgi Hrafn Gunnarsson)
>     3. Re: slowing httpd access to cgi-bin scripts (Sean Walberg)
>     4. Re: slowing httpd access to cgi-bin scripts (Tim Lavoie)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 17 Sep 2010 16:08:00 -0500
> From: Gilles Detillieux<grdetil at scrc.umanitoba.ca>
> Subject: [RndTbl] slowing httpd access to cgi-bin scripts
> To: MUUG Roundtable<roundtable at muug.mb.ca>
> Message-ID:<4C93D8B0.2010708 at scrc.umanitoba.ca>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Every once in a while, some doofus points a web crawler at our web site
> and, ignoring the disallowed areas in our robots.txt file, starts
> crawling through some of our cgi-bin scripts at a rate of 4 to 8 hits a
> second.  This is particularly annoying with some of the more processor
> and disk intensive CGI programs, such as man2html, which also happens to
> generate lots of links back to itself.
>
> Is there anything I can set up in Apache to throttle back and slow down
> remote hosts when they start hitting hard on cgi-bin?  I don't want to
> do anything that would adversely affect legitimate users, nor make
> important things like the manual pages hard to find by removing any
> public links to them.  But when a client starts making 10 or more GET
> requests on /cgi-bin in a 5 second period, it would be nice if I could
> get the server to progressively add longer and longer delays before
> servicing these requests, to keep the load down and prevent the server
> from thrashing.
>
> I'd appreciate any tips.
>
> Thanks,
> Gilles
>


More information about the Roundtable mailing list