News for hackers. Stuff that matters.
Re:Show HN: ParseHub – Extract data from dynamic websites by tsergiu on Tuesday September 23, 2014 @03:02PM (#8357423)
Sorry, we'll make the wording more clear.

At the moment, ParseHub does not respect robots.txt. We do expect to add this + features for webmasters in the future, but have not had the developer cycles to do this yet.

Re:Show HN: ParseHub – Extract data from dynamic websites by spacefight on Tuesday September 23, 2014 @04:30PM (#8357987)
Please add this sooner than "in the future". Obeying the robots.txt is a _MUST_ for any robots out there. This also from a content owner. Thank you.