Just recently, and completely out of nowhere, I've been having issues submitting my sitemap to Google via webmaster tools.

Upon further inspection, it seems that Googlebot has been denied access for an unknown reason. And meanwhile, on top of this problem, nearly 2,000 pages in my arcade have been removed from Google's index.

This is the message according to Google webmaster tools:



And this is the information inside my robots.txt file:

User-Agent: *
Disallow:

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-login.php
Disallow: /wp-register.php

Sitemap: http://www.wordcade.com/sitemap.xml

And on the other hand, my webmaster blog is having nothing wrong with it:



And the contents of robots.txt for my webmaster blog:

User-Agent: *
Disallow:

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-login.php
Disallow: /wp-register.php

Sitemap: http://www.blackwaterblog.com/sitemap.xml

from your robots.txt (http://www.wordcade.com/robots.txt)

User-Agent: *
Allow: /

All compliant dots are blocked from the entire site

Originally Posted by chrishirst

from your robots.txt (http://www.wordcade.com/robots.txt)

User-Agent: *
Allow: /

All compliant dots are blocked from the entire site

Both my arcade and my webmaster blog are using identical robots.txt (exception of url), yet Googlebot is crawling and indexing my webmaster blog without any issues. It's only denied on my arcade.

I've gone and changed the Disallow to Allow: for both robots.txt to see if that makes any difference.

allow: is NOT a valid directive. The robots.txt is an exclusion only protocol.

remove the Disallow: / line completely or remove the "/" from that line.

Originally Posted by chrishirst

allow: is NOT a valid directive. The robots.txt is an exclusion only protocol.

remove the Disallow: / line completely or remove the "/" from that line.

Ok, edited the file to just "Disallow:" removing the / part.

Forgive me, it's been a long time since I've worked with robots.txt stuff.

Note: It may take up 2 days for crawling to restart, robots.txt is normally cached for 24 hours with Google

Originally Posted by chrishirst

Note: It may take up 2 days for crawling to restart, robots.txt is normally cached for 24 hours with Google

Thanks for the help.