News for hackers. Stuff that matters.
Re:Songkick's robots.txt has a few special rules. by dibarra on Sunday April 22, 2012 @12:41PM (#3875289)
It's interesting they're disallowing 008. I work at a popular webhost, and we've had problems with this crawler too (It's distributed, and can crawl things a little too fast).
Re:Songkick's robots.txt has a few special rules. by nintax on Sunday April 22, 2012 @04:20PM (#3875945)
Did you notice the bottom 3 rules? Maybe I should resubmit and change the title...