Hadoop is also about distributed computing, but if the process requires 250x as much RAM, you're going to have a very high TCO regardless of how cheap RAM or servers can be.
J is an opensource APL-derived language, but different in some important ways, but it is not as fast as KDB+/Q/K. J is truly an array-based language, whereas K is list based, so it has more in common with Lisp in that singular respect.
Kerf is a new language being worked on by one of the creators of the Kona language, an opensource version of the K3 language [1].
It seems these type of articles ignore non-opensource solutions even if they are more efficient in many ways including TCO. How many man-years need to be spent to try and duplicate an existing solution, and still not be a better solution?
Since it has not been mentioned yet, let me point out that Barry Jay's Fish should be a part of this conversation. Sadly its quite hard to find. I don't know why those pages have disappeared.
I beat a databricks stack (hand tuned by one of the authors of big-DF) running on a large cluster using one jd node by factors of "a lot." And K is faster than J.
http://quant.stackexchange.com/questions/3156/is-there-any-t...
$25k for 2 core setup is a little crazy. They should definitely add a free tier if they want more adoption and more people investigating their tech. Spark is open source and free.
As I replied below, I am just surprised that similar technology has not been created by the opensource community that is closer in performance to KDB+.
$25k may seem absurd to you, but if it delivers results 230x faster using 1/250th of the RAM, decision makers in business see the TCO (total cost of ownership) as being worth it. It all depends on your needs and how they are met.
Jd is the commercial database for the opensource language J, that is also fast, and I believe it costs less than KDB, so there are other faster options.
If free as in opensource is your criteria, and not actual cost-benefits analysis, you only have those free choices, but in business time is money, so the 'free' option would be more costly.
True, since MonetDB came of out of the Netherlands around 1993, but KDB+ came out in 1998, well over 8 years before C-Store.
K4 the language used in KDB is 230 times faster than Spark/shark and uses 0.2GB of RAM vs. 50GB of RAM for Spark/shark, yet no mention in the article. It seems a strange omission for such a sensational sounding title [1]. I don't understand why big data startups don't try and remake the success of KDB instead of reinventing bits and pieces of the same tech with a result in slower DB operations and more RAM usage.
[1] http://kparc.com/q4/readme.txt