I suspect there is a religious/mythological aspect to K/Q's reputation for speed. During the year that I was using Q I often found it to be slower than the equivalent Matlab code (which benefits from a JIT) and or even NumPy (which has many naively implemented operations).
The biggest advantage of a language like Q is that it's inherently parallel. Map-reduce is naturally built into the language, so this allows the users to think in terms of breaking problems down into sub-problems and parallelizing them via the "parallel apply" construct[1]
Also another very powerful feature is the braindead simplicity of Inter process communication between KDB+ instances [2]
[1] http://code.kx.com/wiki/Reference/peach [2] http://code.kx.com/wiki/Startingkdbplus/ipc
This makes a real difference in ad-hoc environments where you want to setup and tear down complex trading software in real-time
I hear this often about both array and functional languages but I think this kind of slogan leads to a misunderstanding of the MapReduce framework (and what's special about it).
In a functional or array language, map and reduce have the following signatures:
map : key list -> (key -> value) -> value list
reduce : value list -> (value, value -> value) -> value
These are strictly less powerful than the functions used in MapReduce: read : block -> (input_key, input_value) list
map : (input_key, input_value) -> (output_key, output_value) list
reduce : (output_key, (output_value list)) -> (output_value, output_value -> output_value) -> (output_key, output_value)
Combining this API with a (1) a distributed file system which lets you co-locate computation with data across many computers and (2) a parallel partitioning algorithm enables petabyte-scale computation. It's qualitatively different than in-memory array processing.[1] http://code.kx.com/wiki/Reference/peach
[2] http://code.kx.com/wiki/Reference/peach#Peach_using_multiple...
E.g. http://kparc.com/q4/readme.txt (Trillion Row Benchmarks)
It depends what you're doing, but the vast superiority of kdb+ versus its competitors has been demonstrated:
1) What happens when you run these benchmarks on a machine with only 16GB of RAM?
2) How does the KDB performance compare to doing the equivalent operations on a Pandas DataFrame (which, since these are simple in-memory operations, seems like the only fair comparison).
kdb+ uses ~ 1.2 MB of Resident RAM at startup. In addition, kdb+ storage model in memory and on disk have very little overhead (a few hundred bytes) per column over the raw binary data.
The above, combined with memory mapping, allow for large databases to be queried with great performance.
I query billion row futures databases on a Macbook Pro with 16 GB RAM with kdb+.
> 2) How does the KDB performance compare to doing the equivalent operations on a Pandas DataFrame (which, since these are simple in-memory operations, seems like the only fair comparison).
Could someone provide the equivalent Pandas code running against similar TAQ data for the queries in the following benchmark?
Do you have any more specific use cases for the kind of analysis that kdb+ is used for? I'm working with timeseries sources that represent streams from internal variables from embedded systems and would like to evaluate whether kdb+ brings something to the table that we could use. (What we are doing probably falls into the category Complex Event Processing, i.e. windowed operations on the data stream with some predicates being evaluated globally.)
kdb+ does this very efficiently.
0 - https://blog.step.com/2016/04/08/an-open-source-project-for-...
That sounds like the low-end base salary for ~entry level. It probably gets topped up with ~100k bonus. There are a couple places that hire many but don't pay well.
At what I think is the high end, I see about an offer per quarter of $500-900k base with $1-2M bonus (usually guaranteed for the first year) for pure technology role. Add more front office work and the bonus potential shoots up (but it's a tough gig at the moment at least).
For context, non-kdb principal level roles I've seen on west coast top out at around $1M, mostly in RSUs, from a say a $250k base at the usual names.
I've seen offers in the $1-10M range on the east coast to build competitive technology. Amusingly, on the west coast, I've only seen the stereotypical "be my technical cofounder and get screwed on comp" offers to do so.
It's not just knowing an APL though -- far from it. For the chi/nyc it's a mix of hardcore low-level stuff, and some market knowledge. For the west coast, a mix of low level stuff (less hard core) and more distributed systems stuff.
I'm interested to know more about this, assuming you have time.
Occasionally someone pops up wanting to do a company to compete with them but they're typically offering paper of dubious value, covered in slime.
In any event, I'm not particularly interested in going to war to steal their market share. I do worry that they're going to go into a long decline to irrelevancy though -- I'd very much prefer that not to happen. I'd rather they/FD write a new generation building on the best of arthur's work plus some things more suited for new platforms and workloads. There are big opportunities there.
Maybe kOS will be open-source. Maybe it will be the big one, Arthur's chance to change the course of software development history.
http://code.jsoftware.com/wiki/JDB/Announcement
http://www.jsoftware.com/jdhelp/overview.html
https://scottlocklin.wordpress.com/2012/09/18/a-look-at-the-... comes very close to capturing the ingenuity of J language.
After going through the J primer, KDB+/Q feels like a distillation of J - it basically took the most accessible ideas of J and packaged them for mainstream production use - some of their features such as these make it easy to setup a computing harness for electronic trading in no time:
http://code.kx.com/wiki/Cookbook/LoadingFromLargeFiles
http://code.kx.com/wiki/Startingkdbplus/tick
Also, the footprint of the kdb runtime and installation is unparalleled. I've run systems 24 x 7 x 365
they tried. About a year ago they released 32 bit version for free with shockingly liberal license - anything goes, including commercial use, unlimited. About couple of months ago they pulled it back and replaced with free non-commercial use only. well, they are the owners, it is their call, but still k will remain as "one of the most impressive software achievements around", orthogonal to most of the [fucked up] IT trends of the past 15 years. It is a shame if it fades into a quantitative elitist oblivion.
Although mongoDB is a demonstration of how to build a successful business around open-source, it's very different to start out that way rather than move that way after you're already established as closed/licensed product. Kx would have to essentially kill-off their existing profit stream with the goal that they could build a larger business around FOSS kdb+. That's a risky proposition, fraught with many pitfalls.
It is a shame they backed-off of the fully-open 32-bit version. At least that had some potential to spur some user-base/ecosystem growth (at a minimum, this could have encouraged the development of novel clients/editors/REPLs/debuggers/charting/etc...), without threatening their core profit stream.
You are correct though that their model over the years has been to extract a large amount, up front, from a small number of users. They (mostly FD) have failed to make the leap to users outside finance due to what I'd call cultural reasons. They also lack strong technical leadership imnsho (they're, at heart, not an engineering company).
I'd certainly take a swing at doing an open source version for them but not clear to me that they'd know how to play it.
I do know in addition to changing the license they have made changes to the software since that time. The size of the binary has increased.
I worry more about Kx being acquired by some large company, maybe a competing database vendor, that cares little about software quality.
Ask the commenter one down from me, 'wsfull, he knows what I'm talking about!
RE: 36MM - that makes sense. q/kdb is the definition of "low volume, high margins". You only have so many IB's in Midtown, wealth management funds in Stamford, and a PIMCO here and there in Newport to shop your product to. (As opposed to say, Oracle, where there's broad appeal, residual government contracts as a result of legacy PL/SQL code from 20 year old code that's kept in production.)
[1] E-mails in the profile if you want to hear how I got a symbol table in, but I'm sure you already guessed by now.
They're not directly comparable, but kdb+ would crush InfluxDB head to head in an apple/orange comparison if used for the same thing. I actually can't think of any time series-like store that remotely comes close to the feats I have seen kdb+ with Q accomplish. Too bad the productivity and knowledge overhead is so high.
Seriously, I'm working in realtime metrics right now and I spent a week playing with kdb+ as the central store. I can tell there's a gold mine there, but I'm just not swinging a big enough pickaxe. If there was a dumb Python/Go/etc typical ops hacker frontend to that whole system I would be throwing every dollar I have to my name at it; my kingdom for burying that system under a few layers of abstraction and paying some quant Q hacker a Ferrari or two a year to hide the sausagemaking from everyone else.
(Related aside: Like building furniture from stacks of cash? Learn K/Q/kdb+ and head for NYC.)