Re:De-duplicating Hacker News
by crystalgiver on Sunday July 19, 2015 @09:29PM
(#9913590)
Why can you not just use the canonicalized URL to detect dupes? That is infinitely simpler than doing text analysis.
[ Parent
| on HN ]
Re:De-duplicating Hacker News
by shishir456 on Monday July 20, 2015 @03:47AM
(#9914542)
It will work for simple cases like https vs http or other cases of URL normalization but won't work for complex cases where they refer to the same content but with different title.
I think it could work with the canonical tag[1], not the url itself.
[1] http://googlewebmastercentral.blogspot.com.ar/2009/02/specif...