I prefer the DROP packet by IP ADDRESS solution. Simple and to the point. Worked really well for me on abusive crawlers from china and russia. 8-)
Brock
On 9/18/2010 12:00 PM, roundtable-request@muug.mb.ca wrote:
Send Roundtable mailing list submissions to roundtable@muug.mb.ca
To subscribe or unsubscribe via the World Wide Web, visit http://www.muug.mb.ca/mailman/listinfo/roundtable or, via email, send a message with subject or body 'help' to roundtable-request@muug.mb.ca
You can reach the person managing the list at roundtable-owner@muug.mb.ca
When replying, please edit your Subject line so it is more specific than "Re: Contents of Roundtable digest..."
Today's Topics:
1. slowing httpd access to cgi-bin scripts (Gilles Detillieux) 2. Re: slowing httpd access to cgi-bin scripts (Helgi Hrafn Gunnarsson) 3. Re: slowing httpd access to cgi-bin scripts (Sean Walberg) 4. Re: slowing httpd access to cgi-bin scripts (Tim Lavoie)
Message: 1 Date: Fri, 17 Sep 2010 16:08:00 -0500 From: Gilles Detillieuxgrdetil@scrc.umanitoba.ca Subject: [RndTbl] slowing httpd access to cgi-bin scripts To: MUUG Roundtableroundtable@muug.mb.ca Message-ID:4C93D8B0.2010708@scrc.umanitoba.ca Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Every once in a while, some doofus points a web crawler at our web site and, ignoring the disallowed areas in our robots.txt file, starts crawling through some of our cgi-bin scripts at a rate of 4 to 8 hits a second. This is particularly annoying with some of the more processor and disk intensive CGI programs, such as man2html, which also happens to generate lots of links back to itself.
Is there anything I can set up in Apache to throttle back and slow down remote hosts when they start hitting hard on cgi-bin? I don't want to do anything that would adversely affect legitimate users, nor make important things like the manual pages hard to find by removing any public links to them. But when a client starts making 10 or more GET requests on /cgi-bin in a 5 second period, it would be nice if I could get the server to progressively add longer and longer delays before servicing these requests, to keep the load down and prevent the server from thrashing.
I'd appreciate any tips.
Thanks, Gilles
I'm really looking for an automated solution. I suppose I could set something up using fail2ban to do this, but as I've never installed and configured that either, I don't know if this would be any easier than any of the add-on modules for Apache. For now, the quick addition to my CGI wrapper script will probably do the trick.
On 18/09/2010 8:09 PM, Brock Wolfe wrote:
I prefer the DROP packet by IP ADDRESS solution. Simple and to the point. Worked really well for me on abusive crawlers from china and russia. 8-)
Brock
fail2ban would be a good solution for you, as it is easily configurable -- but as you say, if you can do something similar with your own scripts, no issue. fail2ban is setup on the login/asterisk/apache side of things for me -- robert set it up for me :-)
Dan.
On Tue, Sep 21, 2010 at 4:38 PM, Gilles Detillieux < grdetil@scrc.umanitoba.ca> wrote:
I'm really looking for an automated solution. I suppose I could set something up using fail2ban to do this, but as I've never installed and configured that either, I don't know if this would be any easier than any of the add-on modules for Apache. For now, the quick addition to my CGI wrapper script will probably do the trick.
On 18/09/2010 8:09 PM, Brock Wolfe wrote:
I prefer the DROP packet by IP ADDRESS solution. Simple and to the point. Worked really well for me on abusive crawlers from china and russia. 8-)
Brock
-- Gilles R. Detillieux E-mail: grdetil@scrc.umanitoba.ca Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/ Dept. Physiology, U. of Manitoba Winnipeg, MB R3E 0J9 (Canada) _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
-
Brock Wolfe -
Dan Keizer -
Gilles Detillieux