Hi,

One of my hosting account has been warned due to high CPU usage. It was 6% CPU avaregae in last 3 days.
Site been crawled by googlebot - about 28000 hits and msnboot about 1000 hits. Is this possible that indexing caused a high cpu usage? I been advised to setup a robots.txt to limit googlebot hits, but I am not sure how to do this.

Googlebot isn't causing it. Most likely a script he is using has a memory leak or something to a similiar extent.

Based on the # of hits, it's possible ( but unlikely ) the bot is causing this. That really is a lot of traffic, and typically Google is pretty light weight about this stuff. Do you know what a bot trap is? Do you have anything like that set up - links to an infinite number of variations on a querystring?

But remember your site should be able to service a LOT of human visitors, so traffic really shouldn't be a problem.

Oh, and robots.txt won't help you here. You can sign up for a free account at Google Webmaster Tools and in there you can ask them to crawl your site less often. But robots.txt is all or nothing - you can say don't ever crawl these set of pages, but that's NOT what you want!

What scripts do you have running on this site? Are they custom-written or commercial/open source?

Have your host provided details of which scripts are responsible for high cpu load? E.g. does it originate from slow/complex database queries or the scripts themselves?

If your host is going to complain about your CPU usage, the least they can do is tell you which particular scripts etc. are the culprits...

No. They told me only about these hits. On other hosting account I have more hits from googlebot and no problem with it. Site is powered by amazon script from associate-o-matic. For now I moved site to different server.

Yeah it's doubtful that Googlebot is causing this...it never affects my CPU%. I'll look into the scripts and get back to you

Originally Posted by sylwester

Hi,

One of my hosting account has been warned due to high CPU usage. It was 6% CPU avaregae in last 3 days.
Site been crawled by googlebot - about 28000 hits and msnboot about 1000 hits. Is this possible that indexing caused a high cpu usage? I been advised to setup a robots.txt to limit googlebot hits, but I am not sure how to do this.

suggest to move out from 6% hoster. its too low.

i suggest bluehost 20% usage. can upgrade to higher cpu if you willing to pay

and hostgator with 25% cpu usage, and not last for 90 seconds.

Or you could find a company that doesn't limit usage but does monitor it and suspend as necessary. That would be your best bet.