[dead]
Re:How does a relational database work?
by parasubvert on Thursday August 20, 2015 @09:53AM
(#10091779)
He's talking about Apache Spark.
While it can handle 100 MB easily there probably are faster ways to handle that small amount of data. But yes, Spark can handle many PB and doesn't require a ton of changes in the code as you scale up from say 10 TB to 100 PB. The underlying cluster would change, and the performance profile would change a lot (10 TB can be done in-memory ... many PB, not so much)
Please don't make technical comments (or any comments) in this inflammatory, nasty way.
We're lucky that you didn't spark a horrible flamewar, but instead got patient, factual replies.
I know. And I'm sorry, bitterly sorry, but I know that... no apologies I can make can alter the fact that in our thread you have been given a dirty, filthy, smelly piece of technical argument.
I'm not following you here. The point is that you have a history of doing this and if you won't stop doing it you can't comment on HN.
What can I say, being an asshole is one of my diversity dimensions. We just had that training at work, it was very helpful.
Also, I hope you aren't a Python dev in your spare time. Guido would be appalled.
Re:How does a relational database work?
by MichaelGG on Thursday August 20, 2015 @12:30AM
(#10090029)
He's right. Pretend you have 100PB. Write code for that. It'll work for 100MB but have terrible overheads.
Re:How does a relational database work?
by parasubvert on Thursday August 20, 2015 @09:52AM
(#10091770)
He's talking about writing it for Apache Spark.
Which really isn't intended for 100 MB (I bet I could write a unix pipe & filter script that's faster than Spark), but is intended for 10 TB through several PB.
Teradata, Oracle, PostgreSQL for example are reasonably complex databases to cluster and manage yourself. Just as easy/hard as setting up HDFS and installing Hive. In all cases people who are at big data scale are buying OTS solutions e.g. Cloudera appliance. They aren't rolling their own.
And if you are using Hive then I can understand why you are not feeling the buzz. But play around for Spark for a while and it's easy to see the future. Being able to write Scala/Python/SQL/R against a data set that can be anywhere from 100MB to 100PB without any changes is pretty compelling.