SEO Tycoon

**sck4784** · 07-10-2008, 02:11 AM

Hi All,

I have been noticing that Google bots and Yahoo bots are using very much traffics on one of my site.

For example:

In the Month of May:

Googlebot uses 30 GB
Yahoo Lurp uses 11 GB

while in the month of May it was Googlebot 12.04GB Yahoo 6GB.

I don't want to waste my bandwidth unnecessary. Can anybody tell me how I can control on it or what can be done to reduce it?

I think by using robot.txt file we can restrict any search engine on the site and do control increase & decrease crawling rate but apart from this is there any other possible way to control on Google bots & Yahoo.

Please do assist me in this regard.

**screenindia** · 07-10-2008, 04:06 AM

If you are using robots.txt then it will affect your whole website, try to avoid this ...
You can restrict the crawler by using meta tag -
<meta name="revisit-after" content="7 days">

I think it will not help more, but you should try this and see the result.

Also you can restrict the crawler by using the firewall.

**VirtuosiMedia** · 07-10-2008, 05:58 AM

You can reduce the crawl rate of the GoogleBot with Google Webmaster Tools. I would imagine that Yahoo has something similar.

**chrishirst** · 07-10-2008, 06:54 AM

Quote:

Originally Posted by screenindia

Also you can restrict the crawler by using the firewall.

Huh? blocking crawlers with a firewall won't just affect one site. It will affect every site on the server.

Quote:

Originally Posted by screenindia

You can restrict the crawler by using meta tag -
<meta name="revisit-after" content="7 days">

No search engine currently in existence uses that meta element

**screenindia** · 03:51 AM

Quote:

Originally Posted by chrishirst

Huh? blocking crawlers with a firewall won't just affect one site. It will affect every site on the server.

The same problem was with my server, it was taking 1500 GB per month instead of 900 GB per month, and I have resolved this problem by using these strategies, this is a real experience, not a guess idea.

**chrishirst** · 07-11-2008, 12:21 PM

No doubt, but blocking search engine crawlers at firewall level is something of a sledgehammer to crack a nut tactic.

but somehow it never fails to amaze me that "marketing experts" and so-called "SEOs" spend a large amount of time getting links to get SE bots to visit and asking question on how to get more SE traffic, THEN complain because they use bandwidth.

such as ->
http://www.webmaster-talk.com/seo-ta...ate-pages.html
http://www.webmaster-talk.com/the-go...ng-google.html

It simply doesn't make sense

**david_john** · 07-25-2008, 04:06 AM

Quote:

Originally Posted by chrishirst

Huh? blocking crawlers with a firewall won't just affect one site. It will affect every site on the server.

No search engine currently in existence uses that meta element

No you are wrong. Recently (may be before 6-7 months) google introduces meta unavailable_after. So search engines does use meta elements. Clearly saying, it doesn't use all the meta elements something like meta keywords.

**chrishirst** · 07-25-2008, 04:35 AM

Huh??

Did you actually bother to read what question the answer was for????

**david_john** · 07-29-2008, 04:44 AM

Oops! Sorry for the misconception. I have misunderstood that u have mentioned abt all meta elements.

http://googlewebmastercentral.blogsp...meta-tags.html

Here u can find a good answer for this!

**digga123** · 08-14-2008, 12:26 AM

I suggest you use robots.txt and monitor the results using Google Webmaster tools. A good overview on the use of robots.txt can be found at http://googlewebmastercentral.blogsp...-protocol.html

**MatrixIT** · 08-14-2008, 01:09 PM

One thing that I found helpful (not just specifically for your query but also robots info in general) was a post on Matt Cutts' blog. I'll link the article:

http://www.mattcutts.com/blog/improv...tation-galore/

**Local SEO** · 08-17-2008, 07:53 PM

Quote:

Originally Posted by chrishirst

No doubt, but blocking search engine crawlers at firewall level is something of a sledgehammer to crack a nut tactic.

but somehow it never fails to amaze me that "marketing experts" and so-called "SEOs" spend a large amount of time getting links to get SE bots to visit and asking question on how to get more SE traffic, THEN complain because they use bandwidth.

such as ->
http://www.webmaster-talk.com/seo-ta...ate-pages.html
http://www.webmaster-talk.com/the-go...ng-google.html

It simply doesn't make sense

I got a good laugh out of this because its true. My advice don't worry be happy and buy more bandwidth. You want your site crawled. That the whole idea! Crawling is good.