SEO Tycoon

**3DProf4online** · 10-07-2008, 11:48 AM

Hey guys, I just would like to discuss with what benefits can be gained from robots.txt file when promoting website? Could you please explain how to upload this file to hosting? Is it uploaded to public_html root directory as well?

**chrishirst** · 10-07-2008, 01:52 PM

Quote:

Hey guys, I just would like to discuss with what benefits can be gained from robots.txt file when promoting website?

Absolutely NONE at all.

Quote:

Could you please explain how to upload this file to hosting? Is it uploaded to public_html root directory as well?

FTP and Yes if that's where the root of your website is.

**yoonoo** · 10-09-2008, 10:07 AM

BY using robots.txt file you can command search engine bot which file to indexed and to crawl. This way your other files, which you don't want to be publish on web are safe.

**chrishirst** · 10-09-2008, 10:16 AM

Quote:

BY using robots.txt file you can command search engine bot which file to indexed and to crawl.

No you can't.

You can ONLY indicate where compliant bots shouldn't go

**lightningfast** · 10-09-2008, 11:37 AM

Using Robots.txt file you can restrict the search engines from crawling any Web pages or directories in the Web site. Also, you can re-direct the search engine bot to locate the Sitemap file if you are using. It is just guiding the search engines.

**TheRazor** · 10-10-2008, 08:58 AM

Upload it to the root of your domain yes.

With robots.txt you don't get any extra benefits but with a sitemap you will gain WAY WAY more impact on the search engines, google expecially.

A well done sitemap (dynamic sitemaps are the best) can boost your indexing speed up to a few hours.

**dweb** · 10-14-2008, 09:30 PM

The robot.txt file is a good tool to use to allow search engines know which pages you want them to crawl and which ones you don't.

**okili** · 07:48 PM

I have another question regarding robots.txt.

Its simple to disallow any file or directory, such as:
disallow: /forums/
disallow: /forums/somefile.php

can it allow or disallow urls using some predefined pattern?

I mean, I have some urls like this:
/forums/index.php?act=Post&CODE=02&f=15&t=13&qpid=15
/forums/index.php?act=Post&CODE=02&f=15&t=13&qpid=85
/forums/index.php?act=Post&CODE=02&f=15&t=14&qpid=16

and I do not want bots to index them, and I want to disallow the urls, which start on:
/forums/index.php?act=Post&CODE=02 ?

if I write the following in robots.txt
disallow: /forums/index.php?act=Post&CODE=02
or
disallow: /forums/index.php?act=Post&CODE=02*

will it work? or it will disallow the whole /forums/ directory or block /forums/index.php file totally?

I do not want to test it till I'm not sure, while google has indexed more than 35000 urls from my webpage and there is a big threat in modifying robots.txt.

**mrandrei** · 04-27-2009, 11:13 PM

Robots.txt won't have a direct effect in site promotion. It's just a text-based file that gives specific instructions to spiders.

**edgray** · 04-28-2009, 08:18 AM

Google likes to see robots.txt files so they're definitely worth adding, even if you have no disallows in it.

**pow-wow** · 04-28-2009, 11:13 AM

Quote:

Originally Posted by yoonoo

BY using robots.txt file you can command search engine bot which file to indexed and to crawl. This way your other files, which you don't want to be publish on web are safe.

A robots.txt file is not to keep your information secure and safe from prying eyes rather to tell visiting spiders from indexing areas of your website. A robots.txt file is not used to tell search engine spiders what to do, only what not to do.

**swity** · 04-29-2009, 12:30 AM

Robot.txt file is a simply text file. You need to create a notepad file to upload on server. By the txt file you can let search engines know what file have to indexed and what not.

**HivelocityDD** · 04-29-2009, 01:27 AM

Hope this links may help

http://www.webmasterworld.com/robots_txt/
http://www.seoconsultants.com/robots-text-file/

**shivanikapoor** · 04-29-2009, 06:44 AM

Upload robot in root and get benifit like if you don't want some folder not cr awl by search engine write that folder name and type command disallow

**journey08** · 04-29-2009, 06:49 AM

The main function of the robots.txt file is to instruct robots not to crawl and index certain files most particularly password protected folders and folders which contain only images.