I started to look into mod_cband for its ability to limit maximal IP's.
This came about due to some issues I read about having to do with the hosting company I'm using, so I wanted to look into it myself and try to figure out how Google, and the other search engines handle these situations.

What happens when they come to a site, and that site only allows X connections from that IP at a time?

Also, how do sites keep from getting overrun with rogue bots and the like?

This is all new to me, I'm simply trying to understand.

Thanks for any and all responses

ap0

What happens when they come to a site, and that site only allows X connections from that IP at a time

They get served a 403 status code usually (or 421 for FTP)

HTTP normally only makes a single connection per IP so it won't generally happen to websites unless somebody is using a scraping tool to "rip" the site pages.

FTP is what can create multiple streams and multiple connections.

What's happening is, we have a site that's getting hammered by bots/spiders and just general traffic. But some of these bots come in, and just eat us alive.

What I'm interested in learning is, is there a normal / common way of handling these situations?

I want to keep things online and stable, but when I get in 140k bots a day in the morning, it makes it real hard

Block the bots you don't want at the firewall NOT the server.

I wish I could. Instead of blocking we'd prefer to throttle them to a max IP per second, so it evens out only those offending bots. We want them, without a lot of that play then we'll not get indexed in the various places that we'd like.

We would just prefer to allow them to do so on our terms.

If you have a linux server with IPtables, you can get it to block IP's if they exceed a certain amount of requests or bandwidth.

( Dont ask how ) I just know there "is" a way to do this.

Thanks lynxus, but how? j/k

I'm surprised that not more people have experience with stuff like mod_cband and the like. I wonder where I can find sysadmins and the like on the web?

Thanks all for your help, hopefully I can find what I'm looking for here soon.

ap0