Tycoon Talk
Become a Big fish!
The number 1 forum for online business!
Post topics, ask questions, share your knowledge.
Tycoon Talk is part of Freelancer.com - find skilled workers online at a fraction of the cost.

SEO Tycoon


You are currently viewing our SEO Tycoon as a guest. Please register to participate.
Login



Freelance Jobs

Reply
Old 04-19-2006, 05:38 AM Spider Limitation
Novice Talker

Posts: 10
Trades: 0
If google spider crawles particular site .It crawles all webpages in specific site.by using disallow syntax we can block our desired webpage.

BUT,In FTP there will be a few unused webpages .our desired webpages we can view on our site ,how about old and unused pages.Does google crawel all old and unused pages too E.g . index.htm ( current),index1.htm (old)

I'm not getting whether it crawel FTP PAGES or currently running (dispalying)web pages

If google bot crawel all old and new pages what will be the solution.
shanz is offline
Reply With Quote
View Public Profile
 
 
Register now for full access!
Old 04-19-2006, 07:01 AM Re: Spider Limitation
chrishirst's Avatar
Missing! presumed drunk.

Posts: 41,517
Name: Chris Hirst
Location: Blackpool. UK
Trades: 0
it can only crawl what is linked.
__________________
Chris. ->> Links are advertising NOT optimising!! <<-
A foolish consistency is the hobgoblin of little minds
Thought for today:- I SEO the only industry where all the cowboys are Indians?
chrishirst is online now
Reply With Quote
View Public Profile Visit chrishirst's homepage!
 
Old 04-19-2006, 02:27 PM Re: Spider Limitation
stOx's Avatar
Machine

Latest Blog Post:
Worlds Smallest Car - Peel P50
Posts: 2,111
Name: Matt. (>',')>
Location: London, England.
Trades: 0
Chris if i have a page that was spidered and indexed by google, But then i change the layout of my site and don't link to that page, Would google continue to spider it because it knows where it is even though it's no longer linked to?
__________________

Please login or register to view this content. Registration is FREE
-
Please login or register to view this content. Registration is FREE
-
Please login or register to view this content. Registration is FREE
stOx is offline
Reply With Quote
View Public Profile Visit stOx's homepage!
 
Old 04-19-2006, 06:26 PM Re: Spider Limitation
vangogh's Avatar
Post Impressionist

Posts: 10,688
Name: Steven Bradley
Location: Boulder, Colorado
Trades: 0
I may be wrong here, but I think they would still spider the page since they know about it. Just not as often since there's nothing else linking to it.
__________________
l Search Engine Friendly Web Design |
Please login or register to view this content. Registration is FREE

l Tips On Marketing, SEO, Design, and Development |
Please login or register to view this content. Registration is FREE

l
Please login or register to view this content. Registration is FREE
|
Please login or register to view this content. Registration is FREE
vangogh is offline
Reply With Quote
View Public Profile Visit vangogh's homepage!
 
Old 04-19-2006, 06:45 PM Re: Spider Limitation
chrishirst's Avatar
Missing! presumed drunk.

Posts: 41,517
Name: Chris Hirst
Location: Blackpool. UK
Trades: 0
You are right. If it is in the SE database it will continue to be crawled.

Even after a page is taken down and returning a 404 it will continue to be requested

There seems to be the general principle of " Oh oh, we didn't find it, must be us, we'll keep trying"
__________________
Chris. ->> Links are advertising NOT optimising!! <<-
A foolish consistency is the hobgoblin of little minds
Thought for today:- I SEO the only industry where all the cowboys are Indians?
chrishirst is online now
Reply With Quote
View Public Profile Visit chrishirst's homepage!
 
Old 04-20-2006, 12:01 AM Re: Spider Limitation
vangogh's Avatar
Post Impressionist

Posts: 10,688
Name: Steven Bradley
Location: Boulder, Colorado
Trades: 0
Makes sense and I was pretty sure I was right. I usually don't leave orphaned pages around so I had never seen it firsthand.
__________________
l Search Engine Friendly Web Design |
Please login or register to view this content. Registration is FREE

l Tips On Marketing, SEO, Design, and Development |
Please login or register to view this content. Registration is FREE

l
Please login or register to view this content. Registration is FREE
|
Please login or register to view this content. Registration is FREE
vangogh is offline
Reply With Quote
View Public Profile Visit vangogh's homepage!
 
Reply     « Reply to Spider Limitation
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off





   
RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML



Page generated in 0.26428 seconds with 12 queries