SEO Tycoon

**cgnlive** · 10-26-2011, 04:41 PM

Are the sitemap and robots.txt file a must have for a website?

PS: I'm a new and willing to learn webmaster, sorry for the dumb questions.

**chrishirst** · 10-26-2011, 05:35 PM

No and No.

Robots.txt is only required to block compliant bots from some pages.

A sitemap is only required if your navigation causes problems for crawling.

**IraMency** · 04:45 PM

I'm an idiot as to what this means, robots.txt but recently this did something to my site. I really think it screwed up my page rank. My site was newer and got a page rank one, I posted relevant content and good keywords. Now, I use a blog which I post all kinds of things to it, but recently got a "google checkup" that said HUGE ERRORS on one blog page.I look and the whole blog went down to a PR0 and I asked under webmaster central what happened. When I looked up the report "free health report" it said that the page that showed everything under a "KITCHENALIA" label had robots.txt "blocking it." What on earth did this mean? So I remove the label and was hoping that would fit it. f,a nd said that robots.txt was on it and something wasn't crawling it etc. I really didn't know what on earth they were talking about but to fix it I deleted everything with that particular label on it and put a new label on. I guess maybe they thought that label was something x rated I don't know but it's just kitchen stuff. This was on my vintage blog and I don't know how it got there to start with. Can you explain who puts the robots.txt on it, and how to get them off there? Or what it is exactly?

**chrishirst** · 10-27-2011, 05:51 PM

Ever thought of using the enter key to make your posts readable?

**seolman** · 10-28-2011, 04:35 PM

Quote:

Originally Posted by chrishirst

Ever thought of using the enter key to make your posts readable?

**IraMency** · 10-30-2011, 11:51 AM

Sorry guys I'm new here. Still trying to work my ways around.

I'm an idiot as to what this means, robots.txt but recently this did something to my site. I really think it screwed up my page rank.

My site was newer and got a page rank one, I posted relevant content and good keywords. Now, I use a blog which I post all kinds of things to it, but recently got a "google checkup" that said HUGE ERRORS on one blog page.

I look and the whole blog went down to a PR0 and I asked under webmaster central what happened. When I looked up the report "free health report" it said that the page that showed everything under a "KITCHENALIA" label had robots.txt "blocking it."

What on earth did this mean? So I remove the label and was hoping that would fix it. NOPE, all my tags or catergories basically says that robots.txt was on it ....

I don't want to put an example, because then someone will get me for an infraction of putting in a link since I'm new but here is an example:

the website / LABEL TAG 1

the website / LABEL TAG 2

All the label tags had the robots.txt showing.

What does this mean. Does this mean on the pages that aren't a direct post themselves, but are lumped into labels, that the web bots aren't crawling them?

I am not as advanced as tailoring sitemaps, I am using basic HTML and Google's help.

Thanks
Newbee Cindy

**chrishirst** · 10-30-2011, 12:18 PM

First of all SGB PR is absolutely nothing at all to be interested in never mind "concerned about"

And this

Quote:

I posted relevant content and good keywords

means what exactly

Quote:

the website / LABEL TAG 1

the website / LABEL TAG 2

All the label tags had the robots.txt showing.

What does this mean. Does this mean on the pages that aren't a direct post themselves, but are lumped into labels, that the web bots aren't crawling them?

No idea what you are asking about and what is a "label tag"????

**IraMency** · 10-30-2011, 12:45 PM

Okay I guess I should first explain I use blogger and send my actual web addy to the blog. So it's a blog and site in one. With blogs you can use "keywords" or "labels" at the bottom of each post. So for instance to make things cleaner looking when one pulls up your site. If they want to view articles about Decorating they hit the Decorating label tag and they all come up.

So the issue here is when you post a new post on blogger, the post itself has it's own HTML that goes off into cyberspace. Let's say today's post is Kitchenware Decorating Ideas. This .html link does get indexed in Google and my site is indexed and all is fine.

But since I labeled that post "Decorating" if anyone were to go to my site and pull up the Decorating label, they would get ALL POSTS ON DECORATING. This is where the robots.txt comes in.

When my site lost page rank, which I was not sure why, during Google's last page rankings in August, I went into Google's websmaster central which told me I had CRITICAL ERRORS. Eegads, what's this I thought?

So the errors were pointing to each LABEL page and saying that the robots.txt wasn't crawling it.

I guess the answer is I can't have it both ways. If the original post is getting crawled, why should I care, but it's sad to think I have lot of standalone "pages" that are label directories so to speak, and they won't be indexed.

I would put the links here to show you what I'm talking about but someone out there will just infract me so I'm hoping this all makes sense.

Thanks for listening and would appreciate any advice you have on this.

**IraMency** · 10-30-2011, 01:00 PM

P.S. Please be gentle with me I'm operating on 1.0 and you guys are 12.0 , I'm trying to learn !

Little minnow in big pond.
Not scared though.

**chrishirst** · 10-30-2011, 01:06 PM

Quote:

So the errors were pointing to each LABEL page and saying that the robots.txt wasn't crawling it

It's always a good idea to keep the bots away from the spammier looking parts of the site.

And the "label" / "tag" lists do create duplicated URL structures which will give more problems that the useless SGB number will ever do.

Because REAL PR is a probability distribution there is only a value of 1 that is spread around EVERY URL that is indexed by Google, so in any particular period if Google adds MORE documents that are removed EVERYBODY'S share or REAL PR goes down alittle bit, so by the time that it is exported to an integer (0 to 10) it may be that 0.000000000001 difference to make the SGB go down 1 or up 1.

**IraMency** · 10-30-2011, 01:26 PM

Thanks.

Page Rank is a strange monster.......but that's the strange part, the site I'm talking about was a 1 , is indexed by Google, but now is a 0, still indexed by Google. Odd huh?

Was not sure why that would happen. Actually talked to a real GOOGLE person (YES THEY EXIST) who said they didn't flag it or manual knock it down for any reason.

I guess I won't worry about it. But I just don't know why they woudn't crawl a particular label. Labels can be real helpful in people finding what they need. For instance if I were on a SEO site and needed BACKLINK help I would click on that tag cloud for Backlink and read all the 12 associated articles. I would hope that page would be indexed in Google if I were searching Google for "Backlink" help but I guess they don't think like that since it's really a false-positive ie: mirror cloud??? so to speak of the original 12 articles.

Strange though the labels on my other blogs aren't getting blocked by bots. So makes me wonder what the difference is.

Or they haven't been blocked YET.....................

**chrishirst** · 10-30-2011, 01:41 PM

A SGB of 1 could actually somwhere in the range of 0.49 to 1.49 with rounding up

**mechanicaldesig** · 10-31-2011, 02:27 AM

A robot.txt file on a website will function as a request that specified robots ignore specified files or directories in their search.

sitemap.php is a tree structure of website.

sitemap.xml allows a webmaster to inform search engines about URLs on a website that are available for crawling

**ajaysinha10** · 11-22-2011, 06:37 AM

robot.txt help to block some of the folders or pages of your sites. sitemap is a xml file that help crawler to identify those pages that were not indexed by search engine.

**infotech rules** · 11-24-2011, 01:50 PM

Sitemap is something that helps you to be indexed at search engines. Whatever new you add to your site that is submitted.

**Atcomaart** · 11-28-2011, 06:36 AM

If you want to increase your indexing page in Google so sitemap xml is important and if you want to your unwanted pages not crawling in Google so robots.txt is important.

**khalidherbal785** · 12-04-2011, 12:11 AM

Increasing your page indexing in Google so sitemap.xml is important and if you want to your unwanted pages not crawling in Google so robots.txt is important.

**home8** · 12-04-2011, 01:26 AM

Robots.txt is only needed if you want to block some bots from some pages.

**serandfae** · 12-04-2011, 03:28 AM

Wow. The actual answer to the OP's question buried in multiple fluff responses. That's helpful.

tim

**chrishirst** · 12-04-2011, 09:33 AM

Well just as long as people stop reading at post #2 (which is where home8 (obviously NOT a sig link poster) got the reply from.