Google's John Mueller has stated that Google crawlers rely on the cache of the robots.txt file. This cache is updated approximately every 24 hours.

So before you add new content that is "disallowed", you need to add it to your robots.txt file for 24 hours before adding the disallowed content to get crawled. This will insure Googlebot has the updated cache of your robots.txt file and obeys the "Disallow" command.

Ty, seems a bit silly that it doesn't honour the file in real-time though.

Actually it would be impossible to honour it in "real time"

It would mean TWO requests to your server for every page. and then a delay while the robots.txt was analysed.

Originally Posted by chrishirst

Actually it would be impossible to honour it in "real time"

It would mean TWO requests to your server for every page. and then a delay while the robots.txt was analysed.

You beat me to it Chris! Analyzing the robots.txt right before each URL request would be a mess.

Originally Posted by chrishirst

Actually it would be impossible to honour it in "real time"

It would mean TWO requests to your server for every page. and then a delay while the robots.txt was analysed.

Fair enuf, makes sense.

Probably wont change the way i work.. But yanoo..... "The More You Know"

So they are giants sites

Originally Posted by mrmr

So they are giants sites

What the ?????????

You don't get the same quality of sig link posters any more!!

Originally Posted by chrishirst

You don't get the same quality of sig link posters any more!!

Right you are! I deleted one once (no joke), that was something like:
"Thank you very much for the advice. My daughter will be pleased to know that her prom dress worries will seem trivial now."
IN THE INTERNET MARKETING FORUM.

First I laugh, then get angry that I am wasting time removing this s#it. I just want the slap them silly!