News for hackers. Stuff that matters.
Re:Robots.txt meant for search engines don’t work well for w... by Houshalter on Friday April 21, 2017 @12:59PM (#14167270)
Consider this very website, hacker news. Every single comment has it's own url. Which creates a huge number of redundant urls. There's no reason for archive sites to be scraping a separate copy of every single comment after it's indexed the thread. And it makes it harder to use google to search HN.
Re:Robots.txt meant for search engines don’t work well for w... by bigbugbag on Friday April 21, 2017 @02:27PM (#14168020)
Aw man you should have been there 10 years ago, this website hacker news had a url to access every single comment. And quite often some of the comment were more informative than the link poster, I've bookmarked quite a few of those myself. But with hacker news gone they're gone too, because archive.org failed: though they do have the data they broke the links and it is now inaccessible to me.
Re:Robots.txt meant for search engines don’t work well for w... by Houshalter on Friday April 21, 2017 @03:38PM (#14168712)
I see your point. Perhaps a better solution would be to get rid of permalinks to individual comments entirely and point to them with fragment identifiers.