Re:Robots.txt is a suicide note
by mmaunder on Tuesday May 10, 2011 @02:20AM
(#2531590)
I wonder if honeypots that auto-block rogue crawlers have occurred to these yoyos.
[ Parent
| on HN ]
Re:Robots.txt is a suicide note
by sbierwagen on Tuesday May 10, 2011 @02:41AM
(#2531630)
We typically use wget running on consumer machines. This isn't like the spider farm of a typical search engine, with hundreds of machines consuming gigabytes/second of bandwidth. AT hits are specific and targeted.