My website is up for 2 years (or something like this). I changed the Language, Software and Design several times, but since November I'm pretty much happy with what I have and I want to leave it like that now.

I also started promoting my Website a bit, learn something about SEO and stuff and realized, that Google stopped crawling me back in November.

I tried to upload my Sitemap with the Google Webmaster Tools, and I get the following error.

URL restricted by robots.txt
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.

I checked my robots.txt file, with a validator, and it told me there was an error. I had something like Sitemap: http://www.nino-net.org/sitemap.xml in it, which was not valid, so I deleted it.

Now the robots.txt was valid, but still,the same error in Google Webmaster Tools. Can someone help?

Ahh, btw. if you want to look at my robots file: http://www.nino-net.org/robots.txt

the trick is to NEVER tell bots that you DON'T want your pages indexing.

<meta name='robots' content='noindex,nofollow' />

Thanks! Didn't saw this was still enabled in my Wordpress Settings

Still the Google Webmaster Tools don't work yet, do you think they somehow cache my robots.txt?

Yeah, that's the worse thing to do. I'm sure it's easy to overlook at times, but it is one thing you should hopefully never overlook, especially for a client. It would be no bueno!

@JohnnyD What the hell did you try to say???

The sitemap is now in Google Webmaster Tools, but Google still tells me it is not able to crawl because robots.txt restricted it...

I wouldn't worry about it. GWT often gives out conflicting "error" messages.

Wait until the site is being crawled and indexed again, which is starting to happen, so give it a couple or three weeks before being concerned about the sitemap.

Ok, thanks, I really love missleading information! :-)

chrishirst has hit the nail on its head. That's exactly your problem.

is it possible that bots stop crawling ur page ....or ur site stopped indexing ......i mean how such thing can happen even if u r not banned by google .....

@mikecriss. There is a knack to getting the best out of forums;

It's called READING THE THREADS!!!!!

Hi, is there a way to speed it up so that google will reindex the pages? I've just updated my site over the weekend, and am waiting for google to update it. How can I speed the process along if it's possible?

Thanks in advance.
T

Don't mean to hijack but if I may piggyback. Is there anything in or that isn't in this robots.txt file that prevent indexing all pages other that those disallowed?

User-agent: *Disallow: /cgi-bin/Disallow: /images/Disallow: /temp/Disallow: /contact/

Thanks.

Nope
Apart from the file would be invalid without the line breaks.

Apart from the file would be invalid without the line breaks.

That would fall under user error (me) or a VB quirk with cut and paste.

Originally Posted by chrishirst

@mikecriss. There is a knack to getting the best out of forums;

It's called READING THE THREADS!!!!!

I actually lol'ed when I read that.

Thanks Chris.

==============

Its amazing how many people do not take the time to look at their own code. When someone ask for a review of their wordpress blog, 75% of the time their site is missing the meta description tag, and they do not even know it.

This thread is a good example of how just one little setting could really ruin your ranking.