News for hackers. Stuff that matters.

A comment on How to crawl a quarter billion webpages in 40 hours. Parent comment

Re:How to crawl a quarter billion webpages in 40 hours by neurotech1 on Friday August 10, 2012 @04:38PM (#4368138)
Wouldn't the cost of transferring the crawled data (compressed/archived) to AWS/EC2 partially offset the cost saving?

Edit: AWS still has free imbound BW.. my bad

[ Parent | on HN ]
Re:How to crawl a quarter billion webpages in 40 hours by jdrock on Friday August 10, 2012 @04:46PM (#4368165)
No - inbound data transfer cost to AWS is free.
[ Parent | on HN ]
Re:How to crawl a quarter billion webpages in 40 hours by brey on Friday August 10, 2012 @04:38PM (#4368140)
nice thing about running crawlers from EC2 - all inbound bandwidth is free
[ Parent | on HN ]
Re:How to crawl a quarter billion webpages in 40 hours by zerop on Saturday August 11, 2012 @07:01AM (#4369871)
Like running crawlers what can be some other use cases where free inbound is a blessing?
[ Parent | on HN ]
Re:How to crawl a quarter billion webpages in 40 hours by bigiain on Saturday August 11, 2012 @08:40AM (#4369986)
Backup/archive, where uploading data is frequent and downloading it is only very occasionally required.
[ Parent | on HN ]