SEO Tycoon

**earnsomecoin** · 09-05-2006, 05:37 AM

Are robot.txt files still necessary for websites?

If so, why would one use them?

How do you know what to include in the file?

Is there someone out there that can write these files spacifically for my websites?

Thanks!

**imported_Koz** · 09-05-2006, 05:52 AM

This article might help you: Googlebot Behaviour

Robot.txt files would be used to block search engines (you can specify which ones) from crawling a number of pages on your site. It depends on what kind of site you have whether it is neccessary for you or not.

**A.N.Onym** · 09-05-2006, 06:12 AM

Using robots.txt file to block some stuff can be useful, such as:
- PDF folders
- CSS folder
- scripts folder

Basically, anything that you dont want crawled, you need to block in robots.txt. It will save you traffic and won't put it online (important for client documents which are sometimes online).

**stefanjuhl** · 09-05-2006, 11:20 AM

Remember not to list / block very secret folders in your robots.txt, since this is a good place to find stuff that people don't want to share. E.g. don't block your administration folder etc. Instead use the robots meta tag for those individual pages.

**Bookworm-SEO** · 09-09-2006, 11:39 PM

You don't need it.
If you process credit cards or run anything requiring confidentiality (and which could get you sued if you lose the info to others), use robots.txt. Another way I just read about is blocking everything but googlebot (and their other crawlers, such as the adsbot, if you use adwords), slurp (yahoo) and the msn spider (I don't think it has a name). Avoids allowing malicious crawlers to steal/scrape your content.

**T-L** · 09-10-2006, 05:22 AM

I use one for all of my sites. Here's a few things to note:

On forums, many different links lead to the same content, which could be harmful with the search engines due to duplicate content.

Another thing - If you have a nice template, keep the bots out of your image folders. You really don't want your template images showing up on Google Images because some people seem to think that G-Images is their design resource playground.

**andrewbowe** · 09-10-2006, 07:53 PM

just use it to keep the crawlers out of your images folder or anything elese you store on your webspace because it will look unproffesional if they start showing up in the search engines

**stefanjuhl** · 09-11-2006, 03:02 AM

Quote:

Originally Posted by Bookworm-SEO

You don't need it.
If you process credit cards or run anything requiring confidentiality (and which could get you sued if you lose the info to others), use robots.txt.

That's NOT the right way to do it. Anything containing user data or whatever confidential should be blocked so it is in no way accessible. Robots.txt won't do that for you. It will do the opposite because as soon as the competitor hires a skilled SEO consultant they'll check robots.txt for competing websites to see what kind of stuff they don't want us to know about. So you're actually more opening for exploits than blocking anything!

Quote:

Originally Posted by Bookworm-SEO

Another way I just read about is blocking everything but googlebot (and their other crawlers, such as the adsbot, if you use adwords), slurp (yahoo) and the msn spider (I don't think it has a name). Avoids allowing malicious crawlers to steal/scrape your content.

Malicious crawlers that scrape content/e-mail addresses etc. don't obey the robots.txt. (Even the search engines doesn't obey the robots.txt at all times...) They need to be banned server-side based on ip's, behavior etc. The common way to do this is with spider-traps.

**stefanjuhl** · 09-11-2006, 03:04 AM

Quote:

Originally Posted by safron

just use it to keep the crawlers out of your images folder or anything elese you store on your webspace because it will look unproffesional if they start showing up in the search engines

So you don't want the traffic that Google image seach and others can provide..?

You should only block image folders if the crawlers use too much bandwidth, or if you're really sure that you don't want/need traffic from image search.

**stefanjuhl** · 09-11-2006, 03:06 AM

Quote:

Originally Posted by TriteLife

On forums, many different links lead to the same content, which could be harmful with the search engines due to duplicate content.

That's one of the few good reasons to block crawlers with robots.txt

Though you'd be better of if you used cloaking for the crawlers to redirect the duplicate URL's to the primary URL's.

**imported_WoodiE55** · 09-14-2006, 04:46 PM

If I'm not mistaken Lee wrote either here or on ForumTrends.com that he actually blocks all SE's via his robot.txt file from his forum and points them only to the forum archive so the risk of dup content isn't there.

Anyone else do this?