News for hackers. Stuff that matters.
Re:Don't use Hadoop – your data isn't that big (2013) by evancasey on Thursday May 21, 2015 @01:32PM (#9584142)
Huge spark fan here. Love the execution model, API, supporting libs etc.

Unfortunately, Spark doesn't scale well on large datasets (10TB+). Sure, it's possible (and has been done), but right now there are too many rough edges to make it a better choice than Scalding/Cascading for data processing at scale. Most of this boils down to fine tuning certain Spark parameters, which is a pain when you're dealing with long-running, resource intensive workflows.

Re:Don't use Hadoop – your data isn't that big (2013) by kod on Thursday May 21, 2015 @07:49PM (#9586005)
You have any references to support that 10tb claim?
Re:Don't use Hadoop – your data isn't that big (2013) by bsg75 on Thursday May 21, 2015 @09:10PM (#9586198)
What is the upper limit you have hit with Spark?