Tycoon Talk
Become a Big fish!
The number 1 forum for online business!
Post topics, ask questions, share your knowledge.
Tycoon Talk is part of Freelancer.com - find skilled workers online at a fraction of the cost.

Website and Server Administration Forum


You are currently viewing our Website and Server Administration Forum as a guest. Please register to participate.
Login



Reply
removing old folders form index (robots.txt)
Old 08-10-2010, 02:32 PM removing old folders form index (robots.txt)
Skilled Talker

Posts: 89
Name: Tony
Location: Thetford, UK
Trades: 0
Since changing my site from a wordpress blog to a static site I need to block all old urls from the search indexes.

What I am unsure of is, if I disallow a folder (eg /category/ ), will all subfolders and files be removed form the indexes in time, or will I need to add the individual url's to the robots.txt??

This might seem a bit blonde to people but I just want to be sure.

Thanks in advance...
__________________

Please login or register to view this content. Registration is FREE
-
Please login or register to view this content. Registration is FREE

Please login or register to view this content. Registration is FREE

Last edited by Kwinzee; 08-10-2010 at 02:34 PM..
Kwinzee is offline
Reply With Quote
View Public Profile Visit Kwinzee's homepage!
 
 
Register now for full access!
Old 08-10-2010, 02:57 PM Re: removing old folders form index (robots.txt)
vangogh's Avatar
Post Impressionist

Latest Blog Post:
Why Responsive Design?
Posts: 10,815
Name: Steven Bradley
Location: Boulder, Colorado
Trades: 0
Yes that will block everything in the folder. You can get most of the information you'll need from the robotstxt.org site. Click the top link on the left for examples of blocking different things.

However why block everything. Why not use 301 redirects to transfer any search value the pages have to an existing page on the site?
__________________
l Search Engine Friendly Web Design |
Please login or register to view this content. Registration is FREE

l Tips On Marketing, SEO, Design, and Development |
Please login or register to view this content. Registration is FREE

l
Please login or register to view this content. Registration is FREE
|
Please login or register to view this content. Registration is FREE
vangogh is offline
Reply With Quote
View Public Profile Visit vangogh's homepage!
 
Old 08-10-2010, 03:09 PM Re: removing old folders form index (robots.txt)
chrishirst's Avatar
Missing! presumed drunk.

Posts: 42,390
Name: Chris Hirst
Location: Blackpool. UK
Trades: 0
I agree with Steve, redirect NOT block.
__________________
Chris. ->>
Please login or register to view this content. Registration is FREE
<<-

A foolish consistency is the hobgoblin of little minds
Thought for today:- Is SEO the only industry where all the cowboys are Indians?
chrishirst is online now
Reply With Quote
View Public Profile Visit chrishirst's homepage!
 
Old 08-11-2010, 05:07 AM Re: removing old folders form index (robots.txt)
Skilled Talker

Posts: 89
Name: Tony
Location: Thetford, UK
Trades: 0
I thought of leaving the pages in index. But to be honest, the site has been a project for learning more then anything. And I have been concentrating on web design lately. I haven't really spent much time marketing it or generating traffic til now. Search engine traffic for the pages which I have blocked is minimal at present. So I am not too worried about blocking them from search engnes. But I will bear what you 2 have said in mind for future reference so thanks.

I have however set up a .htaccess to redirect 301 all the old pages already. So I will still get traffic from any links I have dropped in the past.

But I now know that I can block whole directories rather then individual url's, which will save time in future. Cheers
__________________

Please login or register to view this content. Registration is FREE
-
Please login or register to view this content. Registration is FREE

Please login or register to view this content. Registration is FREE
Kwinzee is offline
Reply With Quote
View Public Profile Visit Kwinzee's homepage!
 
Reply     « Reply to removing old folders form index (robots.txt)
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off





   
RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML



Page generated in 0.13414 seconds with 12 queries