Ask HN: How Much Can I Scrape?

Re:Ask HN: How Much Can I Scrape? by redorb on Monday November 10, 2008 @11:55PM (#360237)

1 request per a minute I don't think yahoo would even notice.

[ Parent | on HN ]

Re:Ask HN: How Much Can I Scrape? by maximilian on Tuesday November 11, 2008 @12:23AM (#360258)

1 request per second would be still be pretty slow. Computers think in nanoseconds. The internet thinks in milliseconds.

[ Parent | on HN ]

Re:Ask HN: How Much Can I Scrape? by aneesh on Tuesday November 11, 2008 @01:59AM (#360311)

True, but they might notice 1440 requests in the same day from the same IP.

[ Parent | on HN ]

Re:Ask HN: How Much Can I Scrape? by flashgordon on Tuesday November 11, 2008 @04:34AM (#360393)

actually the way i worked around this with both yahoo and google was to roundrobin my requests over a list of 1500 publicly available proxy servers :D

[ Parent | on HN ]