Re:The Ultimate Guide To Duplicate Content
by teyc on Wednesday April 06, 2011 @06:25PM
(#2417248)
Interesting that you mentioned it is 8 words in a row. The usual way duplicate content is checked for is using a moving hash. I had one that I put up online a few years ago.
Other content that should be checked is the printable versions should not be indexed. Wordpress also generates some crawlable urls that cause the same content to be returned, e.g. archives, categories.
Presence of session ids in urls used to be a big problem, but search engines seems to have matured and a bit cleverer. However, if you aren't using a well known CMS, then it is better to make sure you link to a canonical version and don't put the one with session id in the search engine index.
Other content that should be checked is the printable versions should not be indexed. Wordpress also generates some crawlable urls that cause the same content to be returned, e.g. archives, categories.
Presence of session ids in urls used to be a big problem, but search engines seems to have matured and a bit cleverer. However, if you aren't using a well known CMS, then it is better to make sure you link to a canonical version and don't put the one with session id in the search engine index.