The Google Forum

**Slf** · 10-10-2004, 07:16 AM

Can somebody make me a robot text...? i have this,

User-agent: *
Disallow: /cgi-bin/

is that all i need?

**praveen** · 10-10-2004, 08:07 AM

this will help u

http://www.webmasters-cavern.com/gen...robots-txt.htm

**Dark-Skys99** · 10-10-2004, 09:30 AM

Quote:

Originally Posted by Slf

is that all i need?

Its really upto you.. You can prevent any directory you want from being indexed using the same method

--James

**Slf** · 04:37 AM

Yeap thats all i want thanks,
it seems stupid that i need. When its practically telling it to do nothing...

**reciprocal** · 10-11-2004, 04:25 PM

if you do a search for robots.txt on google it will give you some good info about robots.txt and how to have google's spider have a hay day with your site

**Ark** · 10-11-2004, 04:58 PM

what is this robot stuff???

**Dark-Skys99** · 10-11-2004, 05:05 PM

Quote:

Originally Posted by ark

what is this robot stuff???

You create a text file named robot.txt..

This then tells search robots.. Such as the GoogleBot what they are not allowed to access and index...

More info.. http://www.searchengineworld.com/rob...s_tutorial.htm

--James

**Ark** · 10-11-2004, 05:08 PM

Oh ok now I get it... thanks