Re:Don't use Hadoop – your data isn't that big (2013)
by kod on Thursday May 21, 2015 @10:19AM
(#9582861)
I honestly don't know why anyone would use Cascading or Scalding when Spark exists.
[ Parent
| on HN ]
Re:Don't use Hadoop – your data isn't that big (2013)
by evancasey on Thursday May 21, 2015 @01:32PM
(#9584142)
Huge spark fan here. Love the execution model, API, supporting libs etc.
Unfortunately, Spark doesn't scale well on large datasets (10TB+). Sure, it's possible (and has been done), but right now there are too many rough edges to make it a better choice than Scalding/Cascading for data processing at scale. Most of this boils down to fine tuning certain Spark parameters, which is a pain when you're dealing with long-running, resource intensive workflows.
Re:Don't use Hadoop – your data isn't that big (2013)
by kod on Thursday May 21, 2015 @07:49PM
(#9586005)
You have any references to support that 10tb claim?
Re:Don't use Hadoop – your data isn't that big (2013)
by bsg75 on Thursday May 21, 2015 @09:10PM
(#9586198)
What is the upper limit you have hit with Spark?
Re:Don't use Hadoop – your data isn't that big (2013)
by fs111 on Saturday May 23, 2015 @08:27AM
(#9592719)
Because Cascading does not break your apps by changing the API every minor release, that would be one. Also, being production grade software for many years is another.