Show HN: Mirror of current articles on HN front page

Re:Show HN: Mirror of current articles on HN front page by enan on Thursday September 10, 2015 @08:01PM (#10201282)

Thanks for the comments. Our crawler does respect the robots.txt standard and the nofollow tag. Seems like noarchive is what google recommends. Will look more into it.

Although we do put a banner on the index page - we don't have them on each page. Thanks for pointing it out - will fix!

[ Parent | on HN ]

Re:Show HN: Mirror of current articles on HN front page by Paulods on Friday September 11, 2015 @12:19AM (#10202000)

Even more important than that for me (possibly for you too) is that you make sure that none of these pages make it into googles index.

The duplication of content (potentially sending the original pages down in search ranks) and the fact that you are polluting the organic search results for the sites you mirror could be a big issue for the owners of the pages.

[ Parent | on HN ]

Re:Show HN: Mirror of current articles on HN front page by enan on Friday September 11, 2015 @04:21AM (#10202553)

Good point! There is a robots.txt that prevents the site from getting indexed now: http://hn.getpageback.com/robots.txt

[ Parent | on HN ]