Hi, Not much of the webmasters are known to the role of Robots.txt. SO here is my try, please also share your views about this one.

Not many web master take the time to use a robots.txt file for their website. For search engine spiders that use the robots.txt to see what directories to search through. The robots.txt file which can be created simply in a notepad can be very helpful in keeping the spiders indexing your actual pages and not other information, such as looking through your stats and the files that you are going to provide the visitors to download after making the payment etc.

Please let me know if I am going in wrong direction.

Spiders can not access your parts folders with robots.txt file it is also useful in keeping your spiders from accessing parts folders and files in your hosting directory that are totally unrelated to your actual web site content, for example you might have also uploaded those files which are in progress and not fully developed and completed.

In short we can say that it is a tool with the help of which you can make the visitors' search customized for your website.

What are your ideas about using this file?

Feedback Appreciated!!

Maybe before writing a post about other people understanding robots.txt YOU should have taken the time to understand what it is instead of copy and pasting INCORRECT information and trying to tell us "SO here is my try,"

http://www.isnare.com/?aid=642&ca=Internet

Read http://www.robotstxt.org/

Originally Posted by chrishirst

Maybe before writing a post about other people understanding robots.txt YOU should have taken the time to understand what it is instead of copy and pasting INCORRECT information and trying to tell us "SO here is my try,"

http://www.isnare.com/?aid=642&ca=Internet

Read http://www.robotstxt.org/

Pretty funny. I guess there is one in every crowd. Nice

If it was just the one, it wouldn't be a problem

wisdom is knowing what you don't know?

The robot exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard complements Sitemaps, a robot inclusion standard for websites.


http://en.wikipedia.org/wiki/Robots.txt
http://www.webconfs.com/what-is-robo...article-12.php