Re:How to crawl a quarter billion webpages in 40 hours
by duaneb on Saturday August 11, 2012 @12:27AM
(#4369434)
Not to mention if you were malicious, the crawler would probably be inferior to:
while :; curl "http://www.ycombinator.com"; done
That said, there are many crawlers out there, many of them probably more sophisticated in their ability to ruin someone else's day. Unless you're releasing an exploit, malicious users probably know how to abuse the internet more than you do.
Re:How to crawl a quarter billion webpages in 40 hours
by TazeTSchnitzel on Saturday August 11, 2012 @12:17PM
(#4370372)
Or even the infamous LOIC
That is not a good reason. There are many crawlers out there. Anyone can easily modify the string "robots.txt" in the wget binary to "xobots.txt".
Release your code so that others can learn. Stop worrying that you are giving some special tool to bad people - you aren't.