The Google Forum

**mark754** · 09-29-2004, 01:35 AM

I want to prevent my main index page (index.html) from being crawled and indexed by Google, while visitors should continue to access this page when they type in my domain name.

If I set up a robots.txt file to disallow index.html (or use a no index, no follow meta tag), then include a second index page (e.g. "index2.html") that is not disallowed, will the search engines crawl and index the second page? I believe the SE's will look for index.html first and if it doesn't find one, will it pick up index2.html even if it is not linked to index.html?

What's the better method to use, robots.txt or no index, no follow? Thanks.

**KeithMcL** · 09-29-2004, 09:49 AM

http://www.google.com/webmasters/faq.html#nocrawl

**mushget** · 10-15-2010, 06:03 AM

use robots.txt file
the content of the file can be :
User-agent: *
Disallow: /

**robcook** · 10-15-2010, 08:15 AM

Quote:

Originally Posted by mushget

use robots.txt file
the content of the file can be :
User-agent: *
Disallow: /

this will deffinatly work other wise you can also do that from webmaster toll also

**suichiro** · 10-15-2010, 08:34 PM

Here's an article about how to prevent indexing by Google. http://ezinearticles.com/?How-to-Pre...che&id=2012242 hope it helps.

**ustings** · 10-18-2010, 02:04 AM

y would u want to do such a thing?

**Benmoore47** · 10-18-2010, 05:25 AM

why? while lots of people is doing their best to be indexed by Google.

**bermuda** · 10-18-2010, 05:48 AM

In addition to the robots file techniques, in some cases the webmasters might also use password protected directories and areas that would be only visible to the people who would log into the panels to access them. The example could be about certain forums that are not visible to people or search portals until when you manually log into them using your username and password.

**KeshaSparks** · 10-18-2010, 06:00 AM

This was a very informative and interesting discussion.......thanx

**Benmoore47** · 10-19-2010, 04:54 AM

Oh I see, that's why I decided to delete the robot.txt file at my sites.

**maheshpandit** · 10-19-2010, 08:41 AM

You can read the complete help here: http://www.google.com/support/webmas...?answer=164734

**suichiro** · 10-19-2010, 06:39 PM

There is something that bothers me about this thread. If lots of people having a hard time getting their page indexed by search engine specially Google, Then what's the purpose, why do you want not to be indexed?

**Benmoore47** · 10-20-2010, 04:59 AM

Quote:

Then what's the purpose, why do you want not to be indexed?

Hmmm.. It still confuses me. What do you think the reason behind? May it is a spammed site or a pornographic site.

**diyakapoor** · 10-21-2010, 10:05 AM

in robots.txt file write this content:
User-agent: *
Disallow: /

Upload this file to root folder. By doing this Google will not be able to crawl any of your website page.

**kbrkal** · 10-22-2010, 10:33 AM

Go to Google Webmaster Tools and then you can find the page that helps you create a robots.txt file. Then upload it to the root.

**Social-Media** · 10-22-2010, 04:31 PM

If your index.html is already indexed then adding a robots.txt DISALLOW will NOT prevent your URL from showing in the SERPs and will NOT get the URL de-indexed. Even when you block a page using robots.txt, if enough sites link to that page and Google thinks based on those inbound links that the page might be relevant, they can STILL show that URL in the search results. It will typically have a made up title that matches the search phrase and will be missing the search snippet when this happens.

If your want to absolutely prevent your URL example.com/index.html from being shown in the SERPs (and to get it removed from Google if it is already there) then your best bet is to allow the crawlers to crawl the example.com/index.html (i.e. do NOT block it with robots.txt) and add a <meta name="robots" content="noindex" /> element to the <head> of the page. When Google sees this they will:

1) Remove it from their index if it is indexed and
2) NEVERY show it in the results no matter how relevant they think it is for a search query.

Here's a post with more info on how to prevent Google indexing.

**siteion** · 10-22-2010, 04:35 PM

Quote:

Originally Posted by suichiro

There is something that bothers me about this thread. If lots of people having a hard time getting their page indexed by search engine specially Google, Then what's the purpose, why do you want not to be indexed?

Maybe it's a private website or he just doesn't like Google.