The Other Search Engines

**DaveTodd** · 07-19-2002, 06:41 AM

I'm looking for anyone that knows, or can know with a reasonable degree of accuracy, when the next google crawl will be compiled?
see, my site is about 2 weeks old now, and thus is not as yet in the google index. however, i believe that it will be included in the next, and i am attempting to find out when this will be.
anyone with any information?

**david** · 07-19-2002, 08:02 AM

From what I've read in the google.public.support.general newsgroup,, Google spiders the web roughly once a month, usually beginning early on in the month and the updates being made to the engine late in the month. The last update was done in late June.

You might want to keep an eye on http://www2.google.com as I think this site allows you to search the new index a short while before it goes live on the main site.

**DaveTodd** · 07-19-2002, 08:08 AM

thankx david. i've been posting in the google forums for a while, but lots of people there were unsure of when it finished. i figured starting early in the month and relaunching for the beginning of the next month would be the optimum idea. also try http://www3.google.com for a random wildcard draw!

**keithd** · 07-23-2002, 08:22 PM

I just got deep-crawled by google today. I'd guess the crawling will go on for another week or so, then after that google will re-check the bad links and then rebuild the index for the next update.

**TimSchroeder** · 07-23-2002, 11:16 PM

I agree with David.

"usually beginning early on in the month and the updates being made to the engine late in the month. "

I find that they almost always update towards the end of the month between the 25th and 30th....

And the new listing first will appear in www2.google.com or www3.google.com

**DaveTodd** · 07-30-2002, 08:40 AM

righty, thankx guys.

still a bit worried, i've tried google.com where the latest crawl seems to be over, but no luck as to my listing, maybe i missed out this time.

how do you tell if you've been deep crawled by google? what can i do to show me whether or not i've been crawled? is there a file on my server or anywhere else? this i really need to know.

ooh lookie here, i passed 100 posts, go me!!

is www2.google.com a different server to the normal google one? what is www2?

**david** · 07-30-2002, 09:47 AM

www2 appears to be a system which Google have put in place where the new index is 'tested' before being rolled out to the main Google page, so basically you can search on the new index a few days before it is actually available to the public.

As for it being another server, I'm not quite sure how they do this (your question got me thinking about this a bit). I really don't understand at all how they can have two indexes!

Some figures.
In 'The Anatomy of a Large-Scale Hypertextual Web Search Engine' (http://www7.scu.edu.au/programme/ful...21/com1921.htm) which was written before Google became a full search engine, there were the following figures:
Pages Indexed: 24 million
Total Data Stored: 108.7GB

Now, going by Google's home page they now have just over 2 billion pages indexed, so that would be just over 9TB of storage. Admitedly they may have been able to compress it better since then, but I find it hard to believe that they would operate two indexes that size.

That was probably all totally unrelated to the topic, but the document certainly makes interesting reading.

**DaveTodd** · 07-30-2002, 10:39 AM

!"£$%^& thats a lorra lorra space!!
yes they may be able to compress it more, but it will still be min of about 5TB.

I heard a rumor that Google use hundreds of ordinary pcs to store the data so that their response is lightning fast, because it combines them all on a LAN type setup, having lots of small pcs instead of one huge one so that it combines their power and processing speed, ram etc.
has anyone else heard anything about this?