I have been having an issue with google over over a month now.

They indexed me, in 8 hours flat. I never did anything other than submit the site, the sitemap and submitted a link to a few social bookmarking sites.

All was fine for about a week.

Sadly, now the bot keeps getting robots.txt unreachable, sitemap unreachable and network unreachable.

The site is reachable.

I've removed the robots.txt and just entered meta tags in the template of the site, to ease crawling. But its refusing to give me a 404 not found on the robots.txt - instead I get the unreachable error - therefore the bot gives up.

I have also removed the sitemap as well, since this was giving me errors.

The site is http://www.xfilesnews.com and here's the last of the webcrawl report from google I have no errors there, but the URL unreachable section:

http://www.xfilesnews.com/ robots.txt unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php robots.txt unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?option=com_contact&Itemid=3 robots.txt unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?op...ask=category&sectionid=2&id=3&Itemi d=32 Network unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?op...com_content&task=section&id=2&Itemi d=32 robots.txt unreachable [?] Nov 29, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=10&Itemi d=25 Network unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=14&Itemi d=32 robots.txt unreachable [?] Nov 27, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=16&Itemi d=35 robots.txt unreachable [?] Nov 27, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=17&Itemi d=25 robots.txt unreachable [?] Nov 29, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=18&Itemi d=32 Network unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=19&Itemi d=32 robots.txt unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=2&Itemid =32 robots.txt unreachable [?] Nov 23, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=4&Itemid =32 Network unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?option=com_newsfeeds&Itemid=7 robots.txt unreachable [?] Nov 23, 2007


So, i have listings in google, which cannot be updated since googlebot has decided it dosent like my site. I have e-mailed the webhost about this, and its gone on for over a month, and I'm nowhere near understanding what the problem is, or how to fix it.

Anyone have any ideas?

If you deleted your robots.txt file, of course it's going to 404!

Unreachable sounds like something on the server is misconfigured and preventing Googlebot from accessing the URLs. The AdSense bot is obviously able to visit the site though.

Have you been seeing those unreachable reports all month? Or just a few days?

Originally Posted by Thinkey

I have been having an issue with google over over a month now.

They indexed me, in 8 hours flat. I never did anything other than submit the site, the sitemap and submitted a link to a few social bookmarking sites.

All was fine for about a week.

Sadly, now the bot keeps getting robots.txt unreachable, sitemap unreachable and network unreachable.

The site is reachable.

I've removed the robots.txt and just entered meta tags in the template of the site, to ease crawling. But its refusing to give me a 404 not found on the robots.txt - instead I get the unreachable error - therefore the bot gives up.

I have also removed the sitemap as well, since this was giving me errors.

The site is http://www.xfilesnews.com and here's the last of the webcrawl report from google I have no errors there, but the URL unreachable section:

http://www.xfilesnews.com/ robots.txt unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php robots.txt unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?option=com_contact&Itemid=3 robots.txt unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?op...ask=category&sectionid=2&id=3&Itemi d=32 Network unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?op...com_content&task=section&id=2&Itemi d=32 robots.txt unreachable [?] Nov 29, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=10&Itemi d=25 Network unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=14&Itemi d=32 robots.txt unreachable [?] Nov 27, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=16&Itemi d=35 robots.txt unreachable [?] Nov 27, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=17&Itemi d=25 robots.txt unreachable [?] Nov 29, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=18&Itemi d=32 Network unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=19&Itemi d=32 robots.txt unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=2&Itemid =32 robots.txt unreachable [?] Nov 23, 2007 http://www.xfilesnews.com/index.php?option=com_content&task=view&id=4&Itemid =32 Network unreachable [?] Nov 30, 2007 http://www.xfilesnews.com/index.php?option=com_newsfeeds&Itemid=7 robots.txt unreachable [?] Nov 23, 2007


So, i have listings in google, which cannot be updated since googlebot has decided it dosent like my site. I have e-mailed the webhost about this, and its gone on for over a month, and I'm nowhere near understanding what the problem is, or how to fix it.

Anyone have any ideas?

Well........if everything was going fine then there's no need to do these things.

I don't understand why have you removed robots.txt file and sitemap. beacuse they help a lot in indexing and not the meta tags.

Never mind what the very often flaky "webmaster tools" says.

What do your site logs tell you??

Originally Posted by chrishirst

Never mind what the very often flaky "webmaster tools" says.

What do your site logs tell you??

My logs tell me googlebot visits 1 page and then disappears. He never spiders. Yahoo Slurp goes through my site once a day - Googlebot? I saw him 4 times last month by looking through my logs.

BTW - if it was getting a 404 it would still spider.

I'm getting unrachable, which means its something else. I'd be glad if it were reading 404. That means SOMETHING will get indexed. This way my last cache is nov 7th 2007.

Why don't you put your robots.txt back but with note to G. to crawl your site?
It should solve your 404 problem






fastreplies

Or even create an empty (0 byte) robots.txt file to upload. This will stop the 404 errors without blocking anything.

My logs tell me googlebot visits 1 page and then disappears. He never spiders. Yahoo Slurp goes through my site once a day - Googlebot? I saw him 4 times last month by looking through my logs.

Yep, that's correct

SE spiders don't spider, SE crawlers don't crawl.

They are sent to one url, grab the source code and leave, if another page on your site is scheduled to be visited another hit from the bot will occur.

It sounds like the site is new, if so how long has it been live?

The normal pattern for googlebot visits is several single hits to the home page in the first 2 - 4 weeks, this will correspond with an external link being found.

then 4 - 6 weeks on there is usually a flurry of activity where the pages will be read. then at 12 - 16 weeks it will drop to home page visits for a couple of months.

So what you are seeing is perfectly normal.

Do you get 404 in your site logs when you have a robots.txt at all and does the googlebot request it once on the days it hits your site?

Originally Posted by chrishirst

Yep, that's correct

SE spiders don't spider, SE crawlers don't crawl.

They are sent to one url, grab the source code and leave, if another page on your site is scheduled to be visited another hit from the bot will occur.

It sounds like the site is new, if so how long has it been live?

The normal pattern for googlebot visits is several single hits to the home page in the first 2 - 4 weeks, this will correspond with an external link being found.

then 4 - 6 weeks on there is usually a flurry of activity where the pages will be read. then at 12 - 16 weeks it will drop to home page visits for a couple of months.

So what you are seeing is perfectly normal.

Do you get 404 in your site logs when you have a robots.txt at all and does the googlebot request it once on the days it hits your site?

A little over a month.


I went in adn checked my RAW logs (i got that desperate i even started looking at those) My november raws are gone auto delete by teh server, but i checked teh december logs..and so far:

[03/Dec/2007:15:29:49 -0500] "GET / HTTP/1.1" 200 38163 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
76.193.166.76 - -


[03/Dec/2007:18:16:44 -0500] "GET /index.php?option=com_rss&feed=RSS2.0&no_html=1 HTTP/1.1" 200 8932 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.70.220 - -

[03/Dec/2007:22:49:01 -0500] "GET / HTTP/1.1" 200 38101 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
12.201.51.177 - -

Is that google IP's? And if so it looks like it sreading right, but notice how it was only my RSS feed with a URL?

Always make sure you download your logs before they get dumped, they are a vital part in your marketing armoury

Only the 66.249 is a Google IP

the other two, 76.193 is from Dallas & 12.201 is from New York

Originally Posted by chrishirst

Always make sure you download your logs before they get dumped, they are a vital part in your marketing armoury

Only the 66.249 is a Google IP

the other two, 76.193 is from Dallas & 12.201 is from New York

So, the other two, are....?

Either way, I think you see my point, and what was the google bot trying to access anyway? It looks like it ididn't access anything.

and what was the google bot trying to access anyway

A RSS feed.

So, the other two, are....?

No idea, could be anything, cloaking detection from a directory maybe. Did the same IP access anything else?

Originally Posted by chrishirst

A RSS feed.


No idea, could be anything, cloaking detection from a directory maybe. Did the same IP access anything else?

No, that's all it did.

FeedFetcher visits my feeds perfectly fine. And the adsense bot may need some time to do the rest of the site, but the index page seems on par with the topic of the site, so assume that will correct itself over the few weeks.

The search bot? I swear, the movie will be over and I'll still be scratching my head. Even my other site hosted on this service hasn't been indexed since oct 23rd, and i know that makes no sense.

Something died along the way, and my webhost is too stupid to know what. Have you guys ever encountered that, and what was the webhost's technical explanation...i think that would help me at this point.

Well your "home" page was last crawled on the 3rd of December so it doesn't appear to be a major problem.

I'd suspect it's merely a glitch, so doing nothing at all about it for a few weeks would be best.

BTW. Get rid of the hidden text, it's not big and it's not clever and will come back around to hurt you eventually.

removed what I had left. I think its cleaned up now. Btw, how does google see the difference between

x-files and x files

that's what I was trying to balance out, however I dont it overall in the main site text. Does google ignore the dash or not?


Also, do you know of any php script out there that would take my RAW logs and search just for search engine robots, and all other robots? Just curious.

don't know of any PHP scripts, but I use FunnelWeb to download and analyse my log files.

the hyphen "-" is used as a word separator by the SEs, so is treated as a space. Therefore x-files = x files. Most punctuation marks are ignored in search patterns unless the pattern is quoted by the user.

Originally Posted by chrishirst

don't know of any PHP scripts, but I use FunnelWeb to download and analyse my log files.

the hyphen "-" is used as a word separator by the SEs, so is treated as a space. Therefore x-files = x files. Most punctuation marks are ignored in search patterns unless the pattern is quoted by the user.

Thanks for that, I use SEO quake to analyze some stuff, and it seems to note it as two different words.

I use PHP-visites within Joomla to look at my stats, unfortunately there is no clear "browser" part that I can view - I'd have to have more robots than people to see it listed, so I guess that's a good thing its not.

I'll take a look at funnelweb, I know I have it on another website of mine, thanks.

Originally Posted by chrishirst

don't know of any PHP scripts, but I use FunnelWeb to download and analyse my log files.

the hyphen "-" is used as a word separator by the SEs, so is treated as a space. Therefore x-files = x files. Most punctuation marks are ignored in search patterns unless the pattern is quoted by the user.

I'm downloading the funnelweb program now, I assume I download the RAW files from the server and load them into the program?

I'm still having the issues with googlebot - it visits my site pretty much every day, unfortunately according to the webmaster tools it cannot crawl. I can see the boost in engine traffic from being indexed again on the 3rd, ,however I can see that just falling out if i don't get indexed generally....

On the flip side, google analytics is telling me that although i get 50% less traffic from yahoo than google, the traffic is far better since they do stay on the site longer and visit more pages. Yahoo bounce rate for the site is 31% vs. 47% for google. Yahoo to me, at the moment seems far more targeted than google is.

Could this be because of the indexing issue? What are your thoughts?

For now, ignore what the "webmaster tools" tells you, IF your site logs show there is a problem ie: UAs getting 404's and 403's instead of 200's, then look for a cause.

If GA shows REAL visitors, compare these against your logs.

Your site is new, it takes time . The days of putting a site on-line and it being successful in weeks are long gone.