Could anyone give me a clear explanation the difference between robots meta tag and robots.txt. For the past few years I have been controlling the bots by uploading the robots.txt file to the root folder.
Any suggestions??

The difference is in the scope of the instructions;

robots.txt is at site/folder level while the robots meta directives are at page level.

AND
Any suggestions on what?

Originally Posted by chrishirst

AND
Any suggestions on what?

Suggestion on what should be implemented? Should I go for the meta tags. I am aware of the security issue that the pages we are hiding from the bots could be accessed by anyone by typing the Page URL.
Does including robots meta tag clears the issue?

Nothing will prevent anyone typing in the URI to access the page. If you are concerned about security you need to implement more reliable methods of preventing unauthorised visitors

Pages blocked by robots.txt may appear in results as PIPs (Partially Indexed Pages), which shows a URL only if links are found to the page.
With a "noindex" directive in the robots meta this should not occur, as bots can read the page content and the directive for not showing the page in the index.

Originally Posted by chrishirst

With a "noindex" directive in the robots meta this should not occur, as bots can read the page content and the directive for not showing the page in the index.

Thank you Chris Hirst that helped.