Tycoon Talk
Become a Big fish!
The number 1 forum for online business!
Post topics, ask questions, share your knowledge.
Tycoon Talk is part of Freelancer.com - find skilled workers online at a fraction of the cost.

The Google Forum


You are currently viewing our The Google Forum as a guest. Please register to participate.
Login



Reply
Old 10-10-2006, 10:30 AM How do they do that?
Junior Talker

Posts: 3
Name: Mike77
Trades: 0
I've been meaning to ask this for some time now.

I look at my log files and check to see what pages the 3 main spiders look at. I understand when they do a full scan or when you can see them drilling down from the root or other page.

Most often it is pages here and pages there sort of at random.

Does anyone have an idea of how they decide what pages they want to see?

I know that Google does not know what you click after a search. Did a packet sniff.
Maybe all the URLs of the first page?
Maybe from the Googlebar (if installed) after a search?
__________________

Please login or register to view this content. Registration is FREE
-
Please login or register to view this content. Registration is FREE
-
Please login or register to view this content. Registration is FREE

Please login or register to view this content. Registration is FREE
Mike77 is offline
Reply With Quote
View Public Profile
 
 
Register now for full access!
Old 10-10-2006, 10:59 AM Re: How do they do that?
ADAM Web Design's Avatar
Canadastaninianite

Posts: 5,938
Name: Adam for web page design, not program
Location: Toronto, Ontario, Canada
Trades: 0
Assuming your layout is consistent throughout the site, a very general rule of thumb is "the greater the number of clicks away from the home page, the longer it will take to crawl and index."

For the most part, not much goes on with a new site in the first month. It's usually the home page and maybe 1 or 2 other ones (quite often, it's the first couple of links but that's not always the case). Wait about 3-4 months and see if your site is fully indexed, and then worry if it's still random.
__________________

Please login or register to view this content. Registration is FREE
|
Please login or register to view this content. Registration is FREE
(my blog)


Please login or register to view this content. Registration is FREE
(with proof)
ADAM Web Design is offline
Reply With Quote
View Public Profile Visit ADAM Web Design's homepage!
 
Old 10-14-2006, 04:49 PM Re: How do they do that?
chrishirst's Avatar
Missing! presumed drunk.

Posts: 41,517
Name: Chris Hirst
Location: Blackpool. UK
Trades: 0
Despite their name SE "crawlers" do not crawl. They simply are sent out from the SE datacentre to go and get the source code delivered from a URI.
__________________
Chris. ->> Links are advertising NOT optimising!! <<-
A foolish consistency is the hobgoblin of little minds
Thought for today:- I SEO the only industry where all the cowboys are Indians?
chrishirst is online now
Reply With Quote
View Public Profile Visit chrishirst's homepage!
 
Old 10-15-2006, 08:02 AM Re: How do they do that?
Extreme Talker

Posts: 217
Name: Akash
Location: India
Trades: 0
Well it depends upon which page is easily accesable, I mean the pages having the more quality links are indexed more than others.

Akash
__________________

Please login or register to view this content. Registration is FREE
|
Please login or register to view this content. Registration is FREE





vistadivine.com is offline
Reply With Quote
View Public Profile Visit vistadivine.com's homepage!
 
Reply     « Reply to How do they do that?
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off





   
RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML



Page generated in 0.63267 seconds with 12 queries