Most programmers find k interesting. k is fast. k is small. k & APL are great as a tool of thought. It has a "lost art" feel. It's an intellectually satisfying endeavor.

Do many people here deal with array computation?

Like three Dyalog employees occasionally contribute to discussion, fintech employees make up like 10% of the people regularly writing comments on the site, and array programming is incredibly popular when in disguise and made less efficient as Python (numpy), pretty much essential for multiple subfields of computer science, so forth.

K is also supposedly fast—not once have I seen an explanation of what makes it fast

People talk about it all the time here, though. 'geocar mentions why in almost every thread, and he should know: he helped write an operating system in the language.

and why those techniques aren't looted for open-source languages.

Because they don't want to believe it to be true! When told you're messing up something simple, you have to question your assumptions a bit. People hate doing this, especially when they've invested time in seeing them through.

Is K in fact a thin layer on top of assembler (specifically SIMD and other extensions), for number crunching in the vein of shaders/CUDA?

k is higher-level than C, Haskell, Rust, etc.

Frustrating. Obtuse. Overall: fun.

https://github.com/kevinlawler/kona

I do not know how good/performant they are.

Here's a list of a few different k implementations. Most aren't aiming for performance.

https://ngn.bitbucket.io/k.html

Kinda, yeah. It's more interesting--along some axes--than 'how we went from 100 customers to 101 customers' (which some how required a re-write in Java?)

But also I feel like a lot of people look at their C/Python/SQL/JS/HTML/CSS stacks and their thousand person company and hundred thousand lines of code that kind a sorta just directs people at a Stripe widget and think: We Clearly Are Doing Something Wrong.

Not that K/APL or Prolog or Smalltalk or whatever would actually magically make people do-the-right-thing or do-a-thing-better but they're sort of different worlds so they inspire a bit more wonder and reflection than 'how we moved from python to go to rust'

People like k for different reasons. For me, it's the right blend of fast to develop and runs fast enough. Someone recently made a database engine called okon which is designed to process HIBP and looking at their github it took 20+ days of C++ code to do. And they managed to get under 49µsec so they pat themselves on the back for it being so good.

But my k(q) solution was written in five minutes and runs in under 5µsec -- almost 10x faster. I used binary search instead of b-tree (a decision the okon author considered but eventually went the b-tree route instead), but it's otherwise the same basic strategy.

So for me, the decision is, 5 minutes, or 20 days, and that's a no-brainer.

> How does this free K from loading and writing data?

Your CPU has a shockingly small amount of directly-attached memory (around 64kb). That's been true since the 1970s, and is likely to be true for another fifty years.

This used to be called ram, but now we call anything directly addressable by the CPU as ram, and we call this thing cache (or L1 or whatever) because that's basically how most people use it.

Now what your CPU does all day, is look at a memory address, and if that's not in the cache, it sends a message to a memory controller and waits for that memory to show up (somewhat a simplification). Those instructions show up, now your CPU can execute them, and those instructions require more memory so again the CPU sends a message to the memory controller and waits for that memory to show up. All day long, alternating between these two wait states, we wait a bit for code, we wait a bit for data, and then we repeat.

Because K is small: Because the interpreter, the application source code, and all your state variables fit into that 64kb, you can see that memory waiting drops by half.

> Is K in fact a thin layer on top of assembler (specifically SIMD and other extensions), for number crunching in the vein of shaders/CUDA?

No. It's a very simple interpreted language that (in most implementations) dispatch directly on the source code (or in some cases, on the AST).

> why those techniques aren't looted for open-source languages.

If you really understand the above, the answer to this question is obvious (hint: try to think about values, not features). Until then; until you really understand the question you're asking, you're not going to find the answer very useful.

Out of curiosity I did the same in python (with numpy) simply memmap-ing the sorted binary hash list and doing a binary search (again using numpy) on that. I got around 40us lookup time using that solution. Not too bad for about 5 minutes of coding. Also tried using leveldb as datastore and got around 6us when using that (and solution is also only couple lines of code).

  import leveldb,hashlib
  db = leveldb.LevelDB('db')
  def lookup(pw):
    m = hashlib.sha1()
    m.update(pw)
    try: # the leveldb api is a bit minimalist.. 
      db.Get(m.digest())
      return True
    except:
      return False
  %timeit lookup(b'password')

The numpy solution is also quite easy (db3 is essentially your ..|cut|xxd .. file I think:

  import numpy as np
  db = np.memmap('db3', dtype=np.dtype('S20'), mode='r')
  def lookup(pw):
    m = hashlib.sha1()
    m.update(pw)
    idx = np.searchsorted(db, m.digest())
    return db[idx] == m.digest()
  %timeit lookup(b'password')

I think the author of the okon benchmarks was using a i7-6700 CPU @ 3.40GHz, 16GB RAM

How long did the import take you?

Kind of - they have best practices that are almost the opposite of the standard software dev best practices - and they still produce high quality maintainable software.

It’s interesting to see that a lot of “truths” in software development are at the very least context dependent if not outright pseudoscience.

I don’t think you can simply “loot” their best practices - although some devs do write c code the way they would write k code.

Edit: To expand on this for example even if you want to use short identifiers, multiple statements on one line, and avoid abstraction in the Java world you will be fighting the entire ecosystem to do it.

Including Arthur Whitney himself; this is one famous example of that, and is worth close inspection if you're curious about the whole concept of array languages in general:

https://code.jsoftware.com/wiki/Essays/Incunabulum

Observe how quickly it "climbs the abstraction ladder", making heavy use of macros --- if you try to read it like "normal" C and just jump into the middle it's definitely not going to be readable, but if you start at the top and "consume" the definitions first, it becomes surprisingly easy to understand. One thing that stands out is how it makes almost all the loops implicit, via the DO macro.

> if you try to read it like "normal" C and just jump into the middle it's definitely not going to be readable, but if you start at the top and "consume" the definitions first, it becomes surprisingly easy to understand.

I will take your word for this as you might know more than me about programming, but I feel like at least half of my colleagues need to get a gist of what is happening not what exactly is happening to the word. I make simulators for very complex machines. If we went around "climbing the abstraction ladder" every time something needs to be modified or added I think I will personally not get any work done.

Bugs hide in the extra chars.

On a more serious note, there are several (debatable) benefits to brevity (in programming & in general communication);

- less code to keep in your head - less chance of stupid typos (e.g. prinft instead of printf) - more "expressive" in that each char has more meaning - to someone who is fluent in the "shorthand", it is quicker to both write & read

Are simple typos that annoying? Writing a wrong function call will lead to either a compile time error or a wrong behavior, but latter happens when the function names are not chosen smartly. Also we have IDEs now, they are pretty good at handling typos.

I don't understand how it is quicker to remember printf("hello world") than P("hello world").

how sure are you? given how little we know about the brain, i'd be cautious with asserting something like that as a fact. while i don't think we store bytes in our head, i'm not certain that the length of a "symbol" doesn't matter either.

I guess the point is that I don't want to wade in definitions just to get the gist of a code. I want to jump in there, see immediately what's going on and where the problem can be.

    #define V1(f) A f(w)A w;
    #define V2(f) A f(a,w)A a,w;

For folks in the know, conventions like this allow quicker communication and expression, but they also raise the barrier somewhat to the uninitiated.

Side note for folks used to "modern" (post 1989) C code - this is old K&R style function definition. This SO post (https://stackoverflow.com/a/3092074) shows a comparison.

When I told their marketing folks that I had used kdb/q a bit, didn't think it a good fit for our use-case and knew from a brief sojourn in hedge fund-land how hard it was to hire kdb programmers their response was on the lines of "It's not hard to program in and if your guys find it hard they can just use our python wrapper".

Honestly it is an interesting thing but the extent to which it's being pushed here on a daily basis at the moment feels "inauthentic" and somewhat tiresome.

Try posting something else? The front page is populated with links that others find interesting. I started posting on HN mainly because I thought the front page was declining; it's worked out for me, really, APL is on the front page every day now. If you have any good links hanging around, people would be glad to see them, and it will probably tilt the front page into things you like a little more.

There are also 29 other spots on the front page, and some of those spots contain interesting links too! No one has time to read all of the comments on HN every day, so only reading the comments on posts you find interesting will probably be more enjoyable than reading the ones you find tiresome.

The language design is such that the programs you write can make optimal use of cache and SIMD. The interpreter can only take advantage of that if the language offers you that. It's like saying "not once have I seen an explanation of what makes c so fast and why those techniques aren't applied to python and javascript."

(Note also that the reference J interpreter is opensource and very competitive, performance-wise.)

Is there something about the primitives that APL-derivitives provide that mean that more of your program can happen inside the primitives so the interpreter is doing less work?

I'm also curious how the performance compares against something like Julia, which is JIT compiled so you avoid the distinction between the high-level and low-level code. Seems like a rather different region of the high-performance design space.

edit: also how does this relate to the "the whole interpreter fits in the cache" explanation that's also widely-invoked to explain K's speed? It seems like if you're spending most of your time in the primitives than the interpreter time doesn't matter. Maybe it's a combination of both - K and friends encourage you to organize your program in ways that support burning through data quickly, and additionally when the interpreter is operating it's a particularly fast interpreter?

I'm not an {APL,K,J} expert by any definition, but FWIW I think that's the "array-oriented" part of it: syntactically the operators don't specify the "rank" of their args so the interpreter is free to select the most efficient low-level semantics. Something like that.

Yes. Well, maybe not the primitives themselves, so much as how the language pushes you to use them.

Last I looked, execution of the J interpreter, especially on large data, tends to be a handful of very tight loops inside the built-in verbs' implementations, and those loops are glued together with a small amount of the higher-overhead dispatch code normally associated with interpretation. Implicit looping incurs a once-per-array-op cost, but the cost of the computation done on each array element dominates the total cost. The interpreter is not especially clever, but the language encourages the programmer to write code that's easy for the interpreter to handle.

Performance is a bit fragile though. If the operation you're lifting over a whole array includes array-lifted operations internally, now you're doing the heavy interpretive dispatch inside the loop. As a simple example, {.+{: means "add the first item to the last item." If you apply it to a vector, it adds the first and last scalars. If you apply it to a matrix, it adds the first and last rows (and so on). Applying it to a very wide matrix is cheap because {. and {: just have to select the right rows, and then you have a simple loop over those rows' elements. It's about 166ms on my laptop with ten million row elements:

       wide =. i. 2 10000000
       6!:2 '({.+{:) wide'
    0.166085

       tall =. i. 10000000 2
       6!:2 '({.+{:)"1 tall'
    2.99855

It's fast (for an interpreter), because instead of for example a bytecode interpreter where each instruction operates on a single value, APL/j/k interpreters execute operators over whole arrays at once with optimized C code.

,// is a complete implementation of “flatten”.

|/0(0|+)\ is a complete implementation of an efficient the maximum-subarray-sum

In both cases, each character does one very well defined thing implemented in tight C (sometimes SIMD) code, and K interpreter just orchestrates how they interact.

(Parentheses here group as a pair, but everything else is an independent operation per character)