I have a tutorials dynamic site, my site is getting bigger so I will write a PHP script to dynamically generate sitemap from database.
My question is: How much and how deep should I generate urls?
In my site, you can view tutorials by sitename, by category, by sub-category...etc which are considered different urls, Should I generate urls for all those?

Also, how about page numbers? should I generate urls for all pages like: http://mysite.com/photoshop/1/, http://mysite.com/photoshop/2/,
http://mysite.com/photoshop/3/ ....etc?

I'm asking because automated sitemap generators seem to do that, they follow every possible url in your site. I just don't know if this is the right thing to do if you have more control.

include any page that has good content or has potential to get links to it. Include any page you want to be considered for indexing.

What about sorting? you can view tutorials by site, category, sub-category ...etc. Those pages return the same content but with different tutorials order spread into multiple pages, should I include all of those or only one form, for example: by category?

Actually if the different URLs are displaying the same content you don't want to include them all. Search engines don't want to present duplicate content in their results. If you have 3 URLs that all show the same content you should block two of them through robots.txt.

If the content on each URL is unique then sure include them all in your sitemap to Google, but if the content is the same pick the URL you think best to include and block the others from getting indexed.

Thank you for your answers, I know about duplicate content but I'm confused because it's not exactly "the same content" it's more like other views of the content because of sorting. compare those three links to have a more clear idea of what I'm talking about:

http://www.tutorialsroom.com/tutoria...otoshop/all/1/
http://www.tutorialsroom.com/tutoria...nline.co.uk/1/
http://www.tutorialsroom.com/tutoria...oto_effects/1/

2 questions to ask yourself.

do the pages have exactly the same text?

if Yes, it's a duplicate.

do the pages have very similar text, or the same text with just the odd wording change?

if Yes there is every chance of it being a "duplicate"

Hard to say based on the links you posted. I can see how you'd end up with very similar pages with the main difference being the results are listed in a different order. They're pretty thin on content in general though.

It looks like each subcategory will be unique when compared to the others and the more general category will share some of the resources with it's subcategories. I would think they'd end up being unique enough and it would be ok to include them.

You're not really going to be penalized if they are determined to be duplicate.

@vangogh, Thank you for your answer, now you know why it was not easy to decide and that's why I have included the links, thanks for your time visiting them instead of providing "general" answer. Google is saying you won't be penalized for duplicate content (maybe except if they think you are trying to trick the system) and the only thing that might happen is they will choose one version which might not be the one you wanted. As you have seen, this won't be a problem in my case.

Following your words, I think I will add them all. I really appreciate your answer and time. Added some green

Glad to help. Doesn't look like the green came through, but no worries.

The links did help clarify why you're having a hard deciding. I was thinking the the pages would be exactly the same at first, but now I see they aren't. One thing you can always try at a later point is blocking one set of URLs (maybe the general category ones) in robots.txt and see if you notice any change in how the other pages rank. You can block and unblock them as you want just to see what effect if any there is.

Doesn't look like the green came through

The cheque (check) is in the post

Funny. I've heard that line before.

Originally Posted by chrishirst

The cheque (check) is in the post

Sent another cheque

I just cashed it. Thanks

Originally Posted by vangogh

I just cashed it. Thanks

I think you mean cached...

Originally Posted by vangogh

I just cashed it. Thanks

I want some cash too.

Will you settle for come cache?

I believe any string of eight words or more that exists exactly on more than one page counts as duplicate content (not exactly, but I have been recommended to use this as a guide). I think in your case you should include them all since you would want those pages to be found for queries such as grunge tutorials, web 2.0 tutorials, layout tutorials. You could probably leave out the searches by site name or even nofollow to them since it will be very similar content and anyone search for "www.site.com tutorials" will most likely end up on that site anyway. As for the page numbers I would focus on targeting the page 1 for its terms but nothing else. I would still index them though as you could get traffic from some long tail searches that may be listed in a description of one of the tutorials.

I am no professional, this is just my opinion, what I would do if I were in your situation. If I am wrong about anything, someone else will correct me hopefully and I will learn too.

any string of eight words or more that exists exactly on more than one page counts as duplicate content

Highly unlikely unless the pages in questions only have about 8 words each on them. If all it took was a string of 8 words then every time you quoted another web page your page would become a duplicate. That's just not the case.

If anything it's more likely a % of the duplicate words than any absolute number of words.

I would also add the warning to check which links in your sitemap are being indexed closley (via webmaster tools). I had an issue with my site where I was using tabs about half way down the page. Each tab was available via a different url but because there was text repeated above the tabs google saw them all as dup content. This was a problem since the majority of the content was under the tabs but not only that, google did not pick one to index - it simply did not index any of them. Soon as I changed the tabs to render all content and use js to switch view all pages got indexed.

Woow, a lot of points here, you are the best forum

1- About the 8 words rule, I don't think it exists -or even near that value- too because I have always taken the first 4 of 5 sentences of every tutorial and put it in home and index pages with no penalty or whatever. Also, some sites -with my permission - included some of my tutorials in their sites with no penalty because they have included a link to the original tutorial.

2- I will not include the tutorials by site name in my sitemap because I don't care if Google indexes them but I won't nofollow them either, I will leave the decision to Google to decide whether to index them or not

3- I want all the category pages to be indexed so I have included all these pages in my sitemap, but not the sub-categories because they are already included in the links under every tutorial in the category pages so Google should know about them and I think I will get the search queries targeting them too.

4- About keeping an eye on what pages are indexed, I'm happy to say that almost all my site pages are index