I think KDB might be what you’re looking for. It’s very efficient and widely used in finance. It’s pretty expensive, but I believe you can use it for personal use for free (there might be a single core restriction I think).

As an aside, KDB devs make bank, like salaries >500k

I've always been curious how people get into this. I've been interested in KDB ever since I read http://archive.vector.org.uk/art10501320, but it's not exactly a language they teach in college or you'd pick up in independent study. Do people just apply to fintech places with relevant experience and then get training for kdb?

Re how to get started: what was effective for me was to pair kdb+/q with another language and constrain how much I tried to do with q itself. q-sql for joining, filtering, pivoting and grouping (particularly date grouping and grouping with custom aggregate functions) is incredibly fast and powerful and also easy to learn. (though certainly, q experts will be able to point out all sorts of sub-optimal things in your queries)

Initially, we used F# to shovel data into kdb and to orchestrate queries and process results -- while keeping our q-sql simplistic -- and still saw astonishing speedups in our data processing. Over time, wherever we needed more speed, we'd do more and more in q itself (the 'Q Tips' book is very helpful).

One can get a lot of power with just a little kdb+/q.

Typically, one would start off in a quant trading desk which uses kdb+ for their analytics or join First Derivatives [0] that trains graduates

[0] https://www.firstderivatives.com/

The latter. Like everything else in finance, you have to learn on the job. The good news is that since no one who hasn't worked on Wall St (and even most who have) knows anything about the industry (and that the talent pool is pretty small compared to Silicon Valley, at least for non shit-analyst jobs), they are willing to accept people who don't know much, providing you have the basic chops (basic calculus, statistics, and good programming skills).

I really wish more programmers would get into the industry, since the status quo is pretty terrible. The financial industry needs more quantitative folks so we can kill off the dinosaurs of the "greed is good"/pre-decimalization world :)

I have no idea about it, but Kx Systems have a YouTube playlist updated this year, "Introduction to kdb+ and q" -

https://www.youtube.com/playlist?list=PLypX5sYuDqvrwBD2EMWad...

There seem to be a couple of free versions, a 64-bit and a 32-bit version, with the latter being less restricted than the former.[1]

I was curious about it, but they really don't make it easy to get started. There are no links to any documentation anywhere on their site. The download is hidden behind a licensing agreement and a form asking for the user's personal information. They mention GitHub (in the context of "community support") on their download page, but there are no links to any GitHub repos.

[1] - https://kx.com/connect-with-us/download/

https://code.kx.com/q/ has learning material and documentation . Hope it helps

I highly recommend the “Q for Mortals” book as an excellent introduction to using K and it’s available online:

https://code.kx.com/q4m3/

Gnu APL has multi core support for certain computations and you can fork APL child processes and interact with them. So not quite multi threaded but still very useful for heavy duty work where you want to have a great deal of concurrency.

Yes! I'm imagining a value proposition where APL is as fast as C or Fortran, on distributed systems, because at that point it would probably evaporate most of Matlab, Mathematica, and probably TensorFlow's users.

Can you imagine? APL is an amazingly powerful language. It's easy to write nearly any mathematical function in it. What if it was faster than every other mathematical language? It would probably dominate the market overnight.

You mean Dyalog APL? It already regularly outperforms normal, handwritten C code. It also has support for distributed computing, multi-threading, and the Co-dfns compiler can be used to compile your APL code to the GPU. For example, consider the following talk, which discusses sub-nanosecond lookups/search using the Dyalog APL interpreter.

https://dyalog.tv/Dyalog18/?v=paxIkKBzqBU

Supposedly numpy was inspired in part by J.

I don't think it will ever dominate the market - some things such as FFI remain weak points compared to C.

Plus a lot of what makes Mathematica good is that they have specific algorithms that they've optimized quite a bit, for things like computer algebra &c.

As mruts pointed out above, kdb+/q provides what you're imagining - it provides fantastic distributed computing support via its IPC protocol and also has multi-core support.

Multi-core is priced for big financial institutions, though. Only thing you or I could afford is the 32 bit single-core.

There is a free 64-bit version available with limits that don't seem harsh at all:

The 64-bit kdb+ On-Demand Personal Edition is free for personal, non-commercial use. Currently it may be used on up to 2 computers, and up to a maximum of 16 cores per computer, but is not licensed for use on any cloud – only personal computers. It requires an always-on internet connection to operate, and a license key file, obtainable from ondemand.kx.com

You can start the 32 bit instance with the -s switch (for q to be started with multiple slaves) for parallel execution of a function over data using the "peach" command [0].

But yes, I do agree that the 32 bit instances have limited use, as they can't be used in for-profit projects.

[0] https://code.kx.com/wiki/Reference/peach

Is the pricing published? I haven't been able to find it on the website.

It’s definitely one of those “if you have to ask, you don’t need it” kind of things

somewhat related: https://github.com/Co-dfns/Co-dfns