What is robots.txt? Is it a part of on-page?

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is unrelated to, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.
From Wikipedia, the free encyclopedia

I think it's mandatory for websites which has directories needed to hide from search spiders.

It can use with meta tags or as a separate text document with your site. Using this you can decide which page of your site do index or dont index by spider....

Robots.txt inform the search engine bots about which pages and folders are for crawl and which are not for. We can also provide this information only for particular SE bots.

In robot text file we can define which file of folder need not to be indexed.Robot,txt can define on webmaster tool it will be easy to index your webpages easily.

It can useless with meta tags or as a distinguish text document with your site.expending this you can decide whip page of your sit down do index or dont indicator by spider.

Originally Posted by jameskodo85

It can useless with meta tags or as a distinguish text document with your site.expending this you can decide whip page of your sit down do index or dont indicator by spider.

The argument goes something like this: "I refuse to prove that I exist," says God, "for proof denies faith, and without faith I am nothing."
"But," says Man, "the Babel fish is a dead giveaway isn't it? It could not have evolved by chance. It proves you exist, and therefore, by your own arguments, you don't. QED."
"Oh dear," says God, "I hadn't thought of that," and promptly vanishes in a puff of logic.

!!!!!!!