SEO Tycoon

**fontmagic** · 12-28-2007, 04:50 PM

This came as a surprise to me. I was talking to one of my clients and their web site gets 10K uniques each day (page views are more than double this number). However, they don't have a robots.txt file.
So is robots.txt file "required" for goo SEO?

**Nathand** · 12-28-2007, 07:00 PM

It's not required. I think it's usually just used to let search engines know where not to crawl.

Nathan

**bowraven** · 06:03 AM

Hi - Yes I would be interested in whether a robot text file is required, as I have a couple of sites that I have optimised and if this will help me I would love to know, thank you.

Russell
< link drop removed >

**ForrestCroce** · 12-29-2007, 03:20 AM

It's absolutely not required. Robots.txt lets you - or really your server - announce which pages and folders different bots aren't allowed to crawl. Like Nathand said. It has very little actual seo benefit.

You should have one, though. If you don't care about pages being indexed, make an empty, zero-byte file. All well behaved bots will try to grab the file before crawling over a site, so if the file doesn't exist, the server will send down a 404. That's much bigger than an empty file, and if you have any custom logic on your 404 page, it can put a burden on the server. Not a large one, but if they're sending down 20+ thousand pages a day to human clients, scalability is probably at least a minor concern.

If there's no robots.txt file, it means 'bots are allowed to crawl where they please. If you have one, and it's more than one line, it technically needs to be in Unix format.

**dotservant.com** · 12-29-2007, 03:28 AM

I'll have to agree with ForrestCroce, robots.txt is not a compulsory file to your website but it helps if you want to limit the search engines crawlers activities in your website, you can use it to instruct the crawlers to not crawl certain files or sections of your website.

For example, blocking crawlers to pages that is in https:// , including a sitemap: for search engines that reads XML sitemaps etc.

**chrishirst** · 12-29-2007, 06:05 AM

You can't block compliant bots from https: pages with robots.txt.

**dotservant.com** · 12-29-2007, 07:41 AM

Hi Chris,

It is actually a little complex than that for blocking https://, my apology for that.

You can check out this google support doc for the implementation :-

http://www.google.com/support/webmas...n&answer=35302

There must be two different robots.txt in place, one for the http and one for the https

and combining it with a rewrite rule in to serve the correct file :-

RewriteEngine on
RewriteCond %{SERVER_PORT} ^443$
RewriteRule ^robots\.txt$ robots_ssl.txt [L]

**chrishirst** · 12-29-2007, 11:23 AM

You can also do the same with PHP and ASP by using a custom 404 page.

**cyberdog** · 12-30-2007, 11:29 PM

robots.txt are not required, but it MAY help in SEO. i use terms such as crawl delay, because remembers, robots sees www.2.com/page/ different from www.2.com/page. You better give your pages a while to load to redirect them to the proper link, and increase your chances of getting better results for seo.

**ADAM Web Design** · 12-31-2007, 02:36 AM

By spamming a web design forum board with about half a dozen links. It works every single time!

As others said, robots.txt isn't required at all. I often use the robots meta tag (since I find it's the simplest thing to use with dynamically generated content and will block anything I don't want them to find by password-protecting it. If you really want to tell any bot (compliant or otherwise) to go away, password-protect your stuff in a means that makes it extremely difficult (if not impossible) to crack.

**chrishirst** · 12-31-2007, 06:22 AM

Quote:

Originally Posted by cyberdog

because remembers, robots sees www.2.com/page/ different from www.2.com/page. You better give your pages a while to load to redirect them to the proper link, and increase your chances of getting better results for seo.

And what has any of that got to do with robots.txt?

Oh and BTW. domainname/page and domainname/page/ are exactly the same to a search engine

**dman_2007** · 01-01-2008, 02:48 AM

Quote:

Originally Posted by chrishirst

And what has any of that got to do with robots.txt?

Oh and BTW. domainname/page and domainname/page/ are exactly the same to a search engine

I think he meant to say www.domainname.tld/page and domainname.tld/page and forgot to remove www from one of the url. If you aren't already redirecting one of them to the other one, then you can use robot.txt to pick which one you to show up SERPs but redirection method is way better.

**ADAM Web Design** · 01-01-2008, 02:54 AM

I don't think he did. I think he was saying that the final slash made a difference.

**banjosteve** · 01-05-2008, 03:17 PM

I've not noticed a difference in seo for my sites with or without robots.txt -- but I do have one that allows everything because I was tired of seeing it show up in my 404 error stats.