This is still meaningless if you're testing it on someone who doesn't know APL. Having an "unless" statement in your example is exactly like testing for English comprehension. This technique would make sense if you're testing proposed design choices for changes to APL on APL programmers (here's 60 seconds to read this program; what do you think it means?), it would make sense if you have equivalent programs in different languages being tested on users of those languages ("users of Perl understood program X 10% better than users of APL given 60 seconds to examine it"), but it is meaningless in the context you provide.
> Thus, if the test-taker doesn't know English, they get the same (low) score on both tests—to the BNN, that is the same as "no new evidence."
So, of course, you wouldn't test APL vs. APL+whitespace on someone who doesn't know APL, because your test results would always, predictably, be 1:1. But you wouldn't be testing APL vs. anything—because APL is a set of design choices that have already been made, and anyone who knows APL would already have taken the time to absorb those design choices. It would be most scientifically rigorous, as I said, to start with a made-up programming language (a random combination of language features that has no overarching design principle which users could recognize to reduce cognitive load), teach it to the users, and then do the tests on that language, vs. that language w/ modifications.
Compare J, Q, and Dyalog. Three modern APL dialects, very different in feel.
You might protest that J or Q aren't APLs (they only use ASCII! Q has ragged arrays!), but that's like arguing Scheme isn't a Lisp.
Thank you for walking right into my trap.
The reason I like to use APL in arguments about programming language syntax/semantics is that it's "unusual" compared to Algol-derived languages in more than just the syntax.
Matlab/Octave are also array programming languages that in fact are very similar to APL despite having a C-like syntax. It does not make it any easier for people to figure out what Matlab programs do if they don't understand array programming. (And if you don't know linear algebra, then in practice you can't understand most typical Matlab programs at all).
You're implicitly assuming that all programming languages have similar imperative/algorithmic interpretations and differ by syntax. This is a result of a very fundamental epistemological error.
There is no such thing as "programming" that people understand. All programming languages are related to (derived from, or applied to) other domains of knowledge.
If you ignore this fact all the conclusions that you will come to will either apply only to your implicitly assumed domain (which may not fit any problems that people actually want to solve) or will be incoherent.
Your assertion is false that it's easy to perform a usability test on a subject familiar with programming concepts but not exposed to syntax. It is not easy to learn programming without exposure to any language, so finding subjects for such a test would be difficult.
It's called a comparative usability study. The same person looks at the same code sample written in two different languages, A and B (in random order), given time T to absorb each one (where T is much less than the time it would actually take to fully absorb the code), and then the references are removed and they are given tests on the properties of sample_A and sample_B (again in random order.) The score is not a measure of either A or B individually, but rather the ratio A:B, derived by the ratio of correct questions in sample_A:sample_B. A:B is then handed to a Bayesian neural network as a confidence score.
Thus, if the test-taker doesn't know English, they get the same (low) score on both tests—to the BNN, that is the same as "no new evidence."
Also, you wouldn't be comparing, say, Java to APL. You'd be comparing Java to Java + "an unless statement", or APL to APL + "enforced whitespace." You want to see the relative worth of language features, not the relative worth of our current languages, which have mostly been slapped together with no concern for UX. The result of such a study would be a set of UX suggestions, of language features that comparatively increase, or comparatively decrease comprehension.
> And you seem to be a theoretician who has never had to maintain any Lisp code.
For the last eight months, I've been writing Clojure for a living. I've written quite a bit of CL before that. I still find it harder to come back to any Lisp project after a month or two than to come back to code in any other language.