[RndTbl] slowing httpd access to cgi-bin scripts

[Gilles Detillieux]

Every once in a while, some doofus points a web crawler at our web site 
and, ignoring the disallowed areas in our robots.txt file, starts 
crawling through some of our cgi-bin scripts at a rate of 4 to 8 hits a 
second.  This is particularly annoying with some of the more processor 
and disk intensive CGI programs, such as man2html, which also happens to 
generate lots of links back to itself.

Is there anything I can set up in Apache to throttle back and slow down 
remote hosts when they start hitting hard on cgi-bin?  I don't want to 
do anything that would adversely affect legitimate users, nor make 
important things like the manual pages hard to find by removing any 
public links to them.  But when a client starts making 10 or more GET 
requests on /cgi-bin in a 5 second period, it would be nice if I could 
get the server to progressively add longer and longer delays before 
servicing these requests, to keep the load down and prevent the server 
from thrashing.

I'd appreciate any tips.

Thanks,
Gilles

-- 
Gilles R. Detillieux              E-mail: <grdetil at scrc.umanitoba.ca>
Spinal Cord Research Centre       WWW:    http://www.scrc.umanitoba.ca/
Dept. Physiology, U. of Manitoba  Winnipeg, MB  R3E 0J9  (Canada)