Tycoon Talk
Become a Big fish!
The number 1 forum for online business!
Post topics, ask questions, share your knowledge.
Tycoon Talk is part of Freelancer.com - find skilled workers online at a fraction of the cost.

HTML Forum


You are currently viewing our HTML Forum as a guest. Please register to participate.
Login



Post a Project »

Find a Professional HTML Freelancer!

Find a Freelancer to help you with your HTML projects

FREE Outsourcing eBook!

Reply
Old 09-30-2006, 10:50 AM robots.txt
snipergrunge's Avatar
Experienced Talker

Posts: 44
Trades: 0
I am absolutely unfamiliar with /robots.txt what is it? what should i put and how it is important for search engines?
snipergrunge is offline
Reply With Quote
View Public Profile
 
 
Register now for full access!
Old 09-30-2006, 11:00 AM Re: robots.txt
LadynRed's Avatar
Defies a Status

Posts: 10,017
Location: Tennessee
Trades: 0
It's not all that important to search engines, they will spider your site w/o one, though if it's there they will use it - or should. There is a ton of information on the internet about robots.txt, do a search for it. The file should go into the root directory of your site. It can be used to STOP spam bots and spiders from indexing all or part of your site. Unfortunately, there are many bots that will ignore the robots.txt file too.

Here are a few sources:
http://www.javascriptkit.com/howto/robots.shtml
http://www.robotcop.org/
http://www.robotstxt.org/wc/robots.html
http://www.robotstxt.org/wc/active/html/index.html
http://www.psychedelix.com/agents/index.shtml
__________________
Web Goddess & Web Standards Evangelist :) - Tables Be Gone !!

Please login or register to view this content. Registration is FREE


Please login or register to view this content. Registration is FREE

LadynRed is offline
Reply With Quote
View Public Profile
 
Old 09-30-2006, 12:31 PM Re: robots.txt
stOx's Avatar
Machine

Latest Blog Post:
Worlds Smallest Car - Peel P50
Posts: 2,111
Name: Matt. (>',')>
Location: London, England.
Trades: 0
You only need one if you want to stop bots from accessing certain parts of your site.
__________________

Please login or register to view this content. Registration is FREE
-
Please login or register to view this content. Registration is FREE
-
Please login or register to view this content. Registration is FREE
stOx is offline
Reply With Quote
View Public Profile Visit stOx's homepage!
 
Old 10-02-2006, 12:45 AM Re: robots.txt
Novice Talker

Posts: 6
Trades: 0
www.robotstxt.org
nizhongmin is offline
Reply With Quote
View Public Profile
 
Old 10-02-2006, 04:27 AM Re: robots.txt
Kirtan's Avatar
Who Am I?

Posts: 376
Name: Venkat Raj
Location: Salem, South India
Trades: 3
Generally speaking, whenever a spider visits your page, it follows all the links in your page. so all of your pages linked in a logical manner then all the pages in your site will be indexed by that particular search engines. If you want to allow spiders to index particular pages and not to index some other area of your site then you have to use robots.txt.
For example, you may not want to allow spiders to index your images, scripts,etc for that you have to disallow spider using robots.tx
__________________
All the Buddhas of all the ages have been telling you a very simple fact: Be -- don't try to become.
Kirtan is offline
Reply With Quote
View Public Profile Visit Kirtan's homepage!
 
Reply     « Reply to robots.txt
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off





   
RSS Feed  Feeds: RSS   JS   XML
RSS Feed  Feeds for this forum: RSS   JS   XML



Page generated in 0.59194 seconds with 12 queries