That's step 2. Step 1 is using contrived examples with clear differences in structure. You need to start small like this because in order to answer the basic questions, you need to eliminate experimental variables. That way, you can establish basic principles for how people tend to build up an understanding of code.

Then you can move on to what we're really interested in: how do people build an understanding of complex applications with many interacting parts? But you can't go there first; if you do, you won't be able to interpret your results because there will be just as many fundamental questions as experimental variables.

I agree, but isn’t this the curse of most experimental research in programming techniques?

Ideally, we’d compare like-against-like using industrial scale applications, implemented by professional practitioners, controlling for everything except what we’re trying to investigate. Finding opportunities to do that in practice is rather harder, because obviously most real software development projects don’t get implemented twice by identical teams making exactly one significant change in their approach.

What I thought was interesting here was that even though it is only a toy program, there were still some patterns to how the reader explored it that might suggest more general trends. As long as we understand the limitations of the experiment and don’t overgeneralise any conclusions, isn’t some data still better than random conjecture?

Yes, but why can't some regular code (from an open sourced project) be used in the experiments? I'm sure there should be a way to do that.

It is just, that this synthetic code is broken. It doesn't read. There is no flow in it. It just looks from the first glance, like a non-interesting, unimportant piece, that doesn't do anything, so there is no point in reading it.

Does that sound convincing enough, that there is a big difference between reading synthetic examples and actual code?

Yes, it does. There was a study of how well people could remember positions of chess pieces on a board. The general public did equally badly for both random arrangements and those that could actually appear in a game, but skilled chess players did well remembering arrangements that could appear in a game, but were no better than the general public on random configurations.

> isn’t this the curse of most experimental research in programming techniques?

Of most experimental research with humans in general, for that matter. It's a big topic of controversy in psychology, because a lot of quantitative psych results are from synthetic tasks in lab settings, which leads to argument over the extent to which those are accurate proxies for real-world behavior.

This actually smells like the same kind of problem you'd have in studying classrooms...

I’m not sure if this is what you were alluding to there, but I think there is a real problem with experimental research in this kind of field that often the test subjects are students, who tend to have much less skill and experience than working programmers with a few years of practice behind them.

It’s a specific instance of a more general problem: it seems clear that programmers often work differently depending on their familiarity both with programming generally and with the specific domain they’re working in, but we’ve only scratched the surface in identifying exactly how they work differently, and therefore what practical steps we might take to make things better for programmers in different situations.

This is often the problem with a lot of psychology research, where the only easily available subjects are university students who receive credit for participating in the research (or a nominal payment).

For some kinds of experiments, Mechanical Turk is helping to alleviate the problem: http://pps.sagepub.com/content/6/1/3.short

For an experiment like this it's important to have small, self contained code fragments. It wouldn't be fair to expect someone to decipher the whole Linux kernel source in a short amount of time, for example. Given a sufficiently small chunk of code, almost everything looks synthetic. I imagine many real life application have sections that take two arrays, find the numbers within a certain range and then take the intersection like this program does.

Actually linux kernel is very readable. And there is nothing synthetic about it. Just try opening some random page of linux kernel source code, and you'll see. Say: http://www.linux.fm/source/crypto/xor.c/

Hey, you can even listen to it ;)