News for hackers. Stuff that matters.
Re:Testing 3 million hyperlinks, lessons learned by mattmanser on Thursday June 07, 2012 @04:55AM (#4078114)
A variable called stuff? Seriously?

Shame as the rest of the article is quite good, but that really flags me that this is a little bit cowboy code.

Also interesting to read some sites are taking a 'white-list' approach to robots.txt, as he says this is resulting in people starting to ignore it.

Re:Testing 3 million hyperlinks, lessons learned by sams99 on Thursday June 07, 2012 @06:09AM (#4078267)
There you go, it is called items now, personally I don't see much difference this is a tiny demo class and the var is private, you better understand before copying and pasting. Anyway, lets not argue about semi columns. The class is correct for multi threaded access and the API is pretty much perfect for my needs.

Glad you liked the rest of the article, I hope this helps others

Re:Testing 3 million hyperlinks, lessons learned by jackalope on Thursday June 07, 2012 @07:22AM (#4078413)
Starting to ignore robots.txt? Unfortunately, there's no way to prove that anyone respects it. A well-identified bot can back off, then return from a different IP address with a different User-Agent and attempt to mimic a human user. Webmasters really have no defense against policy violations. If you run a bot of any kind, including a link checker or SSL tester, please respect robots.txt. If not, be prepared to be identified as malicious and blocked by an IDS.
Re:Testing 3 million hyperlinks, lessons learned by jackalope on Thursday June 07, 2012 @07:27AM (#4078427)
Let me add: Since the purpose of your bot is to verify links and protect/serve your users, consider removing the links from your site if robots.txt prohibits you from checking them. That's what I would prefer as a webmaster who explicitly set that policy on a site, since I have no control over who posts the links.
Re:Testing 3 million hyperlinks, lessons learned by grecy on Thursday June 07, 2012 @01:06PM (#4080085)
That doesn't make sense.

The point of blocking a link with robots.txt is to say "Hey, web crawlers, please don't load and index this page". it does not mean "Hey, users, please don't come and load and read this page".

So the script written, for all intents and purposes, is just the same as a regular old user clicking the link and reading the page then keeping a list of the links that work and those that don't. It's not a crawler, it's an automated user.

If you are a webmaster than wants to block people from posting links to your page all around the web allowing others to come and read it, make the page 403.

Re:Testing 3 million hyperlinks, lessons learned by mdemare on Thursday June 07, 2012 @08:46AM (#4078690)
So stackoverflow should remove all links to github? Unwise.
Re:Testing 3 million hyperlinks, lessons learned by mapleoin on Thursday June 07, 2012 @05:15AM (#4078158)
I like the profuse amounts of comments, especially when documenting a function's purpose.
Re:Testing 3 million hyperlinks, lessons learned by pluies on Thursday June 07, 2012 @08:01AM (#4078525)
I work for stuff.co.nz (biggest news website in New Zealand). You wouldn't believe the amount of 'stuff' in our codebase. :)
Re:Testing 3 million hyperlinks, lessons learned by hamidpalo on Thursday June 07, 2012 @09:55AM (#4079021)
I didn't even notice the stuff variable, but the lock(this) really irked me.
Re:Testing 3 million hyperlinks, lessons learned by scott_s on Thursday June 07, 2012 @01:07PM (#4080090)
Why? I'm not familiar with C# idioms, but it looks like a sane way to ensure that "items" and "expireOverride" are only modified by a single thread at a time.