News for hackers. Stuff that matters.
Re:Pros and Cons of Redis-Resque and SQS by PaulHoule on Monday July 30, 2012 @09:52AM (#4312283)
I looked pretty hard at systems to do parallel batch jobs.

One thing I looked at was to store intermediate results in Amazon S3 and control the execution of job steps with SQS.

Although the system couldn't do everything Hadoop does, it worked for a wide range of problems and had the desirable quality that it could transparently parallelize multiple processing phases when it was possible.

I went with Hadoop, however, because I can run a Hadoop cluster in the office or in the cloud, and that's a big advantage.

There are no comments on this comment yet.