SEO Tycoon

**shanz** · 04-19-2006, 05:38 AM

If google spider crawles particular site .It crawles all webpages in specific site.by using disallow syntax we can block our desired webpage.

BUT,In FTP there will be a few unused webpages .our desired webpages we can view on our site ,how about old and unused pages.Does google crawel all old and unused pages too E.g . index.htm ( current),index1.htm (old)

I'm not getting whether it crawel FTP PAGES or currently running (dispalying)web pages

If google bot crawel all old and new pages what will be the solution.

**chrishirst** · 04-19-2006, 07:01 AM

it can only crawl what is linked.

**stOx** · 04-19-2006, 02:27 PM

Chris if i have a page that was spidered and indexed by google, But then i change the layout of my site and don't link to that page, Would google continue to spider it because it knows where it is even though it's no longer linked to?

**vangogh** · 04-19-2006, 06:26 PM

I may be wrong here, but I think they would still spider the page since they know about it. Just not as often since there's nothing else linking to it.

**chrishirst** · 04-19-2006, 06:45 PM

You are right. If it is in the SE database it will continue to be crawled.

Even after a page is taken down and returning a 404 it will continue to be requested

There seems to be the general principle of " Oh oh, we didn't find it, must be us, we'll keep trying"

**vangogh** · 04-20-2006, 12:01 AM

Makes sense and I was pretty sure I was right. I usually don't leave orphaned pages around so I had never seen it firsthand.