News for hackers. Stuff that matters.

A comment on Testing 3 million hyperlinks, lessons learned. Parent comment

Re:Testing 3 million hyperlinks, lessons learned by rsofaer on Thursday June 07, 2012 @05:35PM (#4081467)
The sort of tarpit you're talking about wouldn't even affect this link validator. You really think Stack Exchange should have given up on validating links because Github's robots.txt has:

User-agent: *

Disallow: /

in it?

[ Parent | on HN ]
Re:Testing 3 million hyperlinks, lessons learned by lazugod on Thursday June 07, 2012 @08:02PM (#4081981)
They could ask for Github's permission.
[ Parent | on HN ]
Re:Testing 3 million hyperlinks, lessons learned by JasonPunyon on Friday June 08, 2012 @09:21AM (#4084011)
Yeah, we did. http://meta.stackoverflow.com/a/132677/6212
[ Parent | on HN ]