We all know the drill. You post the same thing on 2 pages of your site, only 1 of them will show up in the SERPs. Post something from a different site on yours, and you won't (or shouldn't) turn up at all. I've never done any testing, but I assume this is true. That means Google is able to filter duplicate content, but it also means they're doing something more than just hashing the HTML code, or different templates would seem to be different content.

A good friend of mine wants to create a personal database system. It should capture all types of documents, from voice recordings to email. And especially because of that last one, he's asking me for ideas how he can filter out the duplicate content from his own system. But in a fuzzy way, instead of binary comparisons that would give false negatives. And I haven't been able to come up with one, but I do know that Google is pretty good at "organizing the world's information" so it seems like borrowing their ideas would be a good start.

I don't suppose anyone has any ideas, or thoughts where I should send him to look?

I have an experimental blog that rank on major SERP but it has duplicate content I build more quality link to my site and all of a sudden it came out 5 at first page of Google SERP.

Duplicate content turns up on Google all the time, like when I submit an article to multiple directories.

So... How Google know which site has the unique content? By means on when it was indexed? I think!

I don't care so much how they know which is the original source - to worry about that, first I'd need to know how to identify whether X ~= Y?

Originally Posted by Learning Newbie

I don't care so much how they know which is the original source - to worry about that, first I'd need to know how to identify whether X ~= Y?

There are a few ways. By running documents into a tree, you could just decide what level of duplication is enough to be considered duplicate.

Another option could be a Bayesian classifier.

You could even do word matching. First strip all the prepositions and short words punctuation,etc out of a doc, generate an index of counts for the remaining words. If your indexes match, it's probably duplicate.

None of these are 100% foolproof but, they would probably work for your purpose

Using the registered verison of www.copyscape.com would be my suggestion, however I'm not clear on the question. Do you want to find out how you can find duplicate content within your own site or if the content on your site is located else where?