SEO Tycoon

**jeffmcsheff** · 09-08-2006, 08:16 AM

does a s/e spider always enter/crawl from the root directory? I heard various opinions that sate that it alwyas crawls from the root and some say that t will crawl from the external link page ie if you had an external link www.blah.com/page would it start its crawl from there or would it start from the root?

**stOx** · 09-08-2006, 08:21 AM

It will start the crawl from whatever page that it is linked too.

**jeffmcsheff** · 09-08-2006, 09:25 AM

that is what my opinion is , i want to a seminar fronted by Online lead generation and internet marketing specialists from LeadGenerators. who said that robots will ALWAYS crawl from the root???

**stOx** · 09-08-2006, 09:44 AM

I can't think of any reason why they would.

**gringo** · 09-08-2006, 11:32 AM

It may just be coincidence if most of your links are to your homepage.

**vangogh** · 09-08-2006, 07:51 PM

Jeff that seminar you went to was wrong. My server logs tell me spiders/robots enter my site anywhere they feel like. They follow links so if they come across a link to my portfolio page that might be where they enter. They will probably enter through the root if most links point there, but they don't have to the same way you or I can enter a site just about anywhere we want.

The first time I go to a site it may very well be that I go in through the home page. As I explore I find I like an article enough to bookmark it so the next time I go to that site my starting point is the page with that article. No different really for spiders and robots.

The only thing I can thnk as to why that seminar said that is because they assume spiders/robots will always stop off first at the robots.txt file. That's not true though. They don't have to ever visit that file and even the major search engines won't check your robots.txt on every visit.

**The Critic** · 09-08-2006, 08:46 PM

If you submit a site to a search engine rather than letting it find you on its own it will start with the root and go from there. If you posted a link to your contact page on this forum or someplace else, a bot crawling the forum will start with your contact page and branch out, following the links it finds until it maps your whole site.

**chrishirst** · 12:17 PM

SE crawlers/spiders DO NOT FOLLOW LINKS in the sense we mean.

yes, they may start discovering a site from the root page, simply because that is where the first links you obtain will point to. But equally if they first discover a link to a interior page they will start there

Once the first page is crawled and sent "home" (to the datacentre) the indexer extracts the links and adds them to the crawl schedule, any subsequent page grabs is done with the initial seed links from your home page. This is why site with "splash" pages or Flash homepages are often more difficult to get crawled.

The only time SE crawlers will actually "crawl" is during a deep index process and if you get one of those you had better hope your server is up to it.