How to crawl a quarter billion webpages in 40 hours

Re:How to crawl a quarter billion webpages in 40 hours by jdrock on Friday August 10, 2012 @04:22PM (#4368053)

Being the CEO of a firm that offers web-crawling services, I found this post very interesting. On 80legs, the cost for a similar crawl would be $500, so it's nice to know we're competitive on cost.

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours by BarnabasLAL on Friday August 10, 2012 @04:58PM (#4368219)

Came here to suggest 80legs too, happy to see you're on it. I would add that if the project was being done for actual work and not just curiosity, then 80legs has the added advantage of not spending all that dev time building and testing it.

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours by neurotech1 on Friday August 10, 2012 @04:38PM (#4368138)

Wouldn't the cost of transferring the crawled data (compressed/archived) to AWS/EC2 partially offset the cost saving?

Edit: AWS still has free imbound BW.. my bad

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours by jdrock on Friday August 10, 2012 @04:46PM (#4368165)

No - inbound data transfer cost to AWS is free.

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours by brey on Friday August 10, 2012 @04:38PM (#4368140)

nice thing about running crawlers from EC2 - all inbound bandwidth is free

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours by zerop on Saturday August 11, 2012 @07:01AM (#4369871)

Like running crawlers what can be some other use cases where free inbound is a blessing?

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours by bigiain on Saturday August 11, 2012 @08:40AM (#4369986)

Backup/archive, where uploading data is frequent and downloading it is only very occasionally required.

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours by vlad on Friday August 10, 2012 @08:07PM (#4368911)

I notice the welcome message and action buttons on the home page are all jpeg images rather than text, which makes it harder for search engines to make a judgement about the content. (The text appears fuzzy as well.) As a developer of a crawler, is there a reason you found this to be beneficial?

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours by joshu on Friday August 10, 2012 @05:07PM (#4368254)

(Completely aside: Your front page's colors look strange on my Mac. https://skitch.com/joshua/ecykp/80legs-custom-web-crawlers-p... -- see the blue border?)

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours by ecaron on Friday August 10, 2012 @06:30PM (#4368622)

Not just your Mac - my Firefox & Chrome on Linux. Seems the color palettes in http://80legs.com/_images/bg-home.jpg vs http://80legs.com/_images/the-most-powerful-web-crawler-ever... don't match up. Looks great on all browsers on Windows, so I'm guessing the jpgs have a color profile attached.

[ Parent | on HN ]

Re:How to crawl a quarter billion webpages in 40 hours by showerst on Friday August 10, 2012 @06:17PM (#4368575)

Looks like http://80legs.com/_images/the-most-powerful-web-crawler-ever... uses slightly different gradient stops than http://80legs.com/_images/bg-home.jpg

[ Parent | on HN ]