The Other Search Engines

**Michel Samuel** · 06-02-2007, 02:35 PM

I probably already know the answer to this question but what the hell...

Can I do this in my robots.txt file ?

User-agent: robot-I-want-to-crawl-my-site.
allow: /

User-agent: *
Disallow: /

The objective is to give permission to the robots I like.
And disallow all the rest.

**chrishirst** · 06-02-2007, 09:28 PM

Nope

robots.txt is an exclusion protocol only. It is also a voluntary protocol to follow so not all bots honour the exclusions.

**Michel Samuel** · 06-03-2007, 06:44 AM

Quote:

Originally Posted by chrishirst

Nope

robots.txt is an exclusion protocol only. It is also a voluntary protocol to follow so not all bots honour the exclusions.

I had hope that I was wrong.
C'est la vie.

OK,
In this case anyone have a master list of american search engines ?
I'm going to have to do this the hard way.

**Lucy** · 06-05-2007, 09:49 AM

What is the benefit of Robots.txt file?

**tripy** · 06-05-2007, 10:17 AM

[quote]
What is the benefit of Robots.txt file?
[/quotes]

You can use it to give hints on the search engines agents, and blacklist some parts of your site, for example.
You can see the robots.txt of this site there:
http://www.webmaster-talk.com/robots.txt

**Learning Newbie** · 06-08-2007, 03:55 PM

Quote:

Originally Posted by Michel Samuel

In this case anyone have a master list of american search engines ?
I'm going to have to do this the hard way.

No. But can you use geo-IP blocking instead? You can actually ban robots (return an unauthorized message) as opposed to hanging a "do not enter" sign.

**Thomas Schulz** · 06-20-2007, 10:40 AM

Quote:

Originally Posted by chrishirst

Nope

robots.txt is an exclusion protocol only. It is also a voluntary protocol to follow so not all bots honour the exclusions.

Although Google, Yahoo, Ask etc. have some extensions.

Here's my blog post about having the robots.txt link xml sitemap file.

Basicly you can now write: Sitemap: http://www.example.com/sitemap.xml

**Vasity** · 06-26-2007, 01:42 AM

Why would you want to disclude something from showing from your website. please let me know as I would like to know the functions of this protyocol.

**chrishirst** · 06-26-2007, 04:29 AM

There are many reason you would want to exclude pages or folders from a site.

eg:
Many forums have several ways of getting to the same page, using the disallow you can stop compliant bots from accessing the print versions

If you have tracking links attached to many of your external links you can exclude these with;
disallow: /folder/pagename.ext?track=*

see http://www.robotstxt.org for more on the protocol and http://www.highrankings.com/forum/in...p?showforum=62 for more examples and instances.

**EGS** · 07-01-2007, 03:49 AM

Quote:

Originally Posted by chrishirst

Nope

robots.txt is an exclusion protocol only. It is also a voluntary protocol to follow so not all bots honour the exclusions.

Ditto. Can't include permission on robots.txt - they open themselves to anything and everything on your server if you don't exclude it!