About Linkrot: Proportion of Working Links on Pinboard.in, 1997-2011

Re:About Linkrot: Proportion of Working Links on Pinboard.in... by patio11 on Thursday May 26, 2011 @11:18AM (#2588027)

There is an easy way to increase your detection of redirects and parked pages: make two requests, one to the real URL and one to a URL which is intentionally broken. (example.com/i-am-a-link and example.com/fklsdfasdifo for example) Run a heuristic for difference on the resulting content. This won't catch all of them, particularly if you use a really naive heuristic that can't deal with e.g. ads changing, but it's a heck of a lot quicker than comparing manually.

[ Parent | on HN ]

Re:About Linkrot: Proportion of Working Links on Pinboard.in... by hcho on Thursday May 26, 2011 @01:14PM (#2588536)

I suspect that's what Google does too. I see lots of example.com/fklsdfasdifo type requests from Big G in my logs.

[ Parent | on HN ]

Re:About Linkrot: Proportion of Working Links on Pinboard.in... by bauchidgw on Thursday May 26, 2011 @05:55PM (#2589576)

if you see a lot of them go to webmaster tools, if you see them there too its not some kind of test but some other reasone, mostly their shitty js parsinf, which treats anything with an / as a relative url...

[ Parent | on HN ]

Re:About Linkrot: Proportion of Working Links on Pinboard.in... by benhoyt on Thursday May 26, 2011 @02:37PM (#2588918)

Yeah, that's a good method to detect "soft 404s". I wrote a Python library for DecentURL to do this some time ago: http://blog.brush.co.nz/2008/01/soft404s/ (Ironically, the original paper I got the idea from seems to be a dead link: http://www.tomkinshome.com/andrew/papers/decay/final/p444-ba...)

[ Parent | on HN ]

Re:About Linkrot: Proportion of Working Links on Pinboard.in... by ible on Thursday May 26, 2011 @04:33PM (#2589336)

I had a project/startup working on dealing with link rot for a while. It would not just tell you a link was 404ing, but recognize when the content of the page had changed significantly and let you know. The fun part was to automatically recommend a good replacement page from within the site, nearby, or the internet archive.

Based on a quick test it seemed that it would take an site owner about 5-10 minutes per link to find a good replacement once they knew it was broken . That's fine for a personal portfolio with 50 links, but for a site like Boing Boing getting all the broken links working again looked like full time work for a year.

I'm curious if you have many outbound links in your 'scalable content'? Do you spend much time maintaining them?

[ Parent | on HN ]

Re:About Linkrot: Proportion of Working Links on Pinboard.in... by true_religion on Thursday May 26, 2011 @10:49PM (#2590275)

I just want to point out that this won't really work if the site tries to redirect you to a search engine. Naturally broken links will come up with all kinds of search results, but deliberately broken links will come up with "None".

[ Parent | on HN ]