Re:Don't use Hadoop when your data isn't that big (2013)
by alexchamberlain on Tuesday May 23, 2017 @01:38PM
(#14403272)
I like that as a rule of thumb; I think the following tend to bucket storage and processing solutions well enough to be a starting point
Small data: fits in memory
Medium data: fits on disk
Big data: fits on multiple disks
I've yet to come up with a rule of thumb for throughput though, and this can never replace the expertise of an experienced, domain knowledgeable engineering team. As always, there are lots of things to balance, including cost, time to implement and the now Vs the near future. Rules of thumb over simplify, but also give you a way to discuss different solutions without coming over as one size fits all.
Small data: fits in memory Medium data: fits on disk Big data: fits on multiple disks
I've yet to come up with a rule of thumb for throughput though, and this can never replace the expertise of an experienced, domain knowledgeable engineering team. As always, there are lots of things to balance, including cost, time to implement and the now Vs the near future. Rules of thumb over simplify, but also give you a way to discuss different solutions without coming over as one size fits all.