Re:Show HN: ParseHub – Extract data from dynamic websites
by spacefight on Tuesday September 23, 2014 @04:30PM
(#8357987)
Please add this sooner than "in the future". Obeying the robots.txt is a _MUST_ for any robots out there. This also from a content owner. Thank you.
At the moment, ParseHub does not respect robots.txt. We do expect to add this + features for webmasters in the future, but have not had the developer cycles to do this yet.