The Other Search Engines

**Famous16** · 04-20-2010, 11:55 AM

Hi guys,

Can anyone please tell me what is robot.txt and how its working ?

Regards,
Famous16

**lynxus** · 04-20-2010, 12:00 PM

Its a file you have on your server / root of your website.

Spiders or Robots ( search enginer bots mainly ) look for this.

The .txt file has a ruleset that tells the bots what they can and cant look at and index.

This is only useful for robots that actually abide by the rules.
Custom bots or smaller SE's may not follow the rules in the file

**Knight13** · 04-20-2010, 12:01 PM

It is a file you put on your website and it tells the search robot what pages you do not want them to visit. or at least that is what i think it is.

**chrishirst** · 04-20-2010, 01:05 PM

Not robot.txt but robots.txt

www.robotstxt.org

**alexjoan** · 04-22-2010, 05:27 AM

robots.txt is a file you put in the root of your folder. in this file you can specify which files are accessible by which search engine. For example :

user-agent: *
disallow: /pdf/
disallow: /avoid*.html

By these lines you are saying that you do not want any search engines index all files in folder "pdff" and also all files which start with "avoid" and end with html are not accessible.

**casius** · 04-23-2010, 01:54 AM

Hello,

Robots.txt is a normal text file that during its name has particular meaning to the best part of honorable robots on the web. By defining a few rules in this text file, you can initiate robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

**Winson** · 04-28-2010, 07:04 AM

Hi Friend,
As far as I know robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.

Thanks,
James

**chrishirst** · 04-28-2010, 07:10 AM

Quote:

Originally Posted by Winson

Hi Friend,
As far as I know robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.

WRONG! You can ONLY tell Search Engines NOT to crawl certain section.