> Map-reduce is naturally built into the language
I hear this often about both array and functional languages but I think this kind of slogan leads to a misunderstanding of the MapReduce framework (and what's special about it).
In a functional or array language, map and reduce have the following signatures:
map : key list -> (key -> value) -> value list
reduce : value list -> (value, value -> value) -> value
These are strictly less powerful than the functions used in MapReduce: read : block -> (input_key, input_value) list
map : (input_key, input_value) -> (output_key, output_value) list
reduce : (output_key, (output_value list)) -> (output_value, output_value -> output_value) -> (output_key, output_value)
Combining this API with a (1) a distributed file system which lets you co-locate computation with data across many computers and (2) a parallel partitioning algorithm enables petabyte-scale computation. It's qualitatively different than in-memory array processing.kdb+ map/reduce implementation operates across data in memory or on machines in a cluster.
Does the Q/K implementation actually execute in parallel? I've always had a hard time figuring out exactly why K is supposed to be so fast (except for column stores, interpreter fitting in L1 cache, and fast hand-written primitives).
Re:The K Language
by bladecatcher on Monday April 25, 2016 @03:41AM
(#11562955)
Yes, you can start the server with any number of slave processors. As long as you're not writing data to global namespace (i.e, if you can keep your code purely functional), then as pointed out above, one can use the "peach" (parallel each) operator [1] to parallelize execution. Further you also get the "distributed each" to distribute load amongst multiple processes instead of threads [2]
[1] http://code.kx.com/wiki/Reference/peach
[2] http://code.kx.com/wiki/Reference/peach#Peach_using_multiple...
The biggest advantage of a language like Q is that it's inherently parallel. Map-reduce is naturally built into the language, so this allows the users to think in terms of breaking problems down into sub-problems and parallelizing them via the "parallel apply" construct[1]
Also another very powerful feature is the braindead simplicity of Inter process communication between KDB+ instances [2]
[1] http://code.kx.com/wiki/Reference/peach [2] http://code.kx.com/wiki/Startingkdbplus/ipc
This makes a real difference in ad-hoc environments where you want to setup and tear down complex trading software in real-time