I am experiencing this problem with phpbb website.
I have set robots.txt right on first day as

User-agent: *
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /templates/
Disallow: /common.php
Disallow: /groupcp.php
Disallow: /memberlist.php
Disallow: /modcp.php
Disallow: /posting.php
Disallow: /privmsg.php
Disallow: /viewonline.php
Disallow: /faq.php
Disallow: /search.php
Disallow: /config.php

Now as i know Google read and follow instructions in robots.txt file.
But it has indexed more than 3 times the originally allowed webpages with posting.php and privmsg.php which are disallowed in robots.txt .

And more to this problem when google starts to drop webpages then instead of dropping those pages with privmsg.php and posting.php (which are disallowed in robots.txt) it drops other allowed pages more.

What could be the problem?
I think it must be some serious problem in Google algo.

Google as of now having lots of problem and its better not to be taking any decisions based on its results but just to wait and watch.

Originally Posted by shabbirbhimani

Google as of now having lots of problem and its better not to be taking any decisions based on its results but just to wait and watch.

No Google is fine. It is the webmaster who are having the problems. Do a little research back in 2003 when this same type of thing happened. Familiar?

Google is trying to keep the results relevant in a world of webmasters who are trying to press their site to the top whether it is most relevant or not. Be relevant and keep quality content stocked up and you will have no problems.

Originally Posted by Visio

No Google is fine. It is the webmaster who are having the problems. Do a little research back in 2003 when this same type of thing happened. Familiar?

Google is trying to keep the results relevant in a world of webmasters who are trying to press their site to the top whether it is most relevant or not. Be relevant and keep quality content stocked up and you will have no problems.

Try reading A hypothesis about current google problems.

shabbirbhimani: the word "hypothesis" implies that it's not a conclusive theory by any stretch of the imagination. It's just one of those things people are throwing out there with no real evidence or proof of anything.

Part of the problem with "Google has problems" is that, if they do (and I'm not sure if they do or don't), it cannot be conclusively determined. The vast, VAST majority of sites out there have some form of error or another preventing them from getting indexed properly.

Besides, in Mong's case there may be a fix that exists with the robots.txt file.

Mong, try this. Under your disallow, put:

User-Agent: Googlebot
Disallow: /posting.php*

According to http://www.mattcutts.com/blog/new-robotstxt-tool , the wildcard is supported within the disallowed URLs. You may be getting querystrings after it that are indexed.

Failing that, if you can edit your forum template, add the following to your meta tags for each disallowed page (I prefer this over robots.txt since it can be applied dynamically):

<meta name="robots" content="noindex, nofollow" />

If the first thing doesn't work, the second always does.