Re:Ask HN: Should I consider a startup based on scraped data?
by davidjairala on Tuesday May 05, 2015 @06:15PM
(#9495351)
Just to make sure I understand your reply correctly, are you saying that if a robot.txt file disallows your specific crawler but allows googlebot you'd see no problem with crawling it?
[ Parent
| on HN ]
There are no comments on this comment yet.