Re:Webcrawling at Scale with Nokogiri and IronWorker: Part 2
by carimura on Monday September 17, 2012 @02:45PM
(#4534146)
Awesome. Love the options.
FWIW - the value prop of IronWorker is never having to deal with servers again -- and only pay for the seconds you're actually crawling. Fire up a million crawlers (workers) and it's auto distributed across large sets of machines behind the scenes (no "spin up tear down" either).
So in essence, it's a completely hosted version of what you described. The power is seen by "just trying it". No software (except for your code), no installations, no servers, etc.
FWIW - the value prop of IronWorker is never having to deal with servers again -- and only pay for the seconds you're actually crawling. Fire up a million crawlers (workers) and it's auto distributed across large sets of machines behind the scenes (no "spin up tear down" either).
So in essence, it's a completely hosted version of what you described. The power is seen by "just trying it". No software (except for your code), no installations, no servers, etc.