Otherwise the results of that research would be biased towards comprehension of meaningless chunks of code, rather that of a real thing.
Then you can move on to what we're really interested in: how do people build an understanding of complex applications with many interacting parts? But you can't go there first; if you do, you won't be able to interpret your results because there will be just as many fundamental questions as experimental variables.
Ideally, we’d compare like-against-like using industrial scale applications, implemented by professional practitioners, controlling for everything except what we’re trying to investigate. Finding opportunities to do that in practice is rather harder, because obviously most real software development projects don’t get implemented twice by identical teams making exactly one significant change in their approach.
What I thought was interesting here was that even though it is only a toy program, there were still some patterns to how the reader explored it that might suggest more general trends. As long as we understand the limitations of the experiment and don’t overgeneralise any conclusions, isn’t some data still better than random conjecture?
It is just, that this synthetic code is broken. It doesn't read. There is no flow in it. It just looks from the first glance, like a non-interesting, unimportant piece, that doesn't do anything, so there is no point in reading it.
Does that sound convincing enough, that there is a big difference between reading synthetic examples and actual code?
Of most experimental research with humans in general, for that matter. It's a big topic of controversy in psychology, because a lot of quantitative psych results are from synthetic tasks in lab settings, which leads to argument over the extent to which those are accurate proxies for real-world behavior.
It’s a specific instance of a more general problem: it seems clear that programmers often work differently depending on their familiarity both with programming generally and with the specific domain they’re working in, but we’ve only scratched the surface in identifying exactly how they work differently, and therefore what practical steps we might take to make things better for programmers in different situations.
Hey, you can even listen to it ;)
It would also be interesting to compare with code with decent syntax highlighting. For some reason I don't quite understand, most syntax highlighting out there highlights the wrong thing: keywords. It needs to highlight the site-of-definition of functions, variables, etc. For example: http://david.rothlis.net/code_presentation/distracting_synta...
Maybe what I don't understand is all the effort people put into making pretty themes for their editors --there are millions of these themes out there-- and almost all of them insist on highlighting (drawing attention to) the language's keywords of all things. The least important part of any program.
1. Establishing familiar patterns is good, and being able to identify deviations from patterns (intentional or otherwise) is probably far more important than we usually assume.
2. You’re going to need to figure out the data flow for some code you’re reading anyway, and a lot of the time the control flow is just noise/accidental complexity, so why not put the data flow front and centre if possible?
Applying these to the example code we’ve been discussing, I might prefer to write it something like this:
x = [2, 8, 7, 9, -5, 0, 2]
y = [1, -3, 10, 0, 8, 9, 1]
print filter ( 2 < _ < 10) x
print filter (-2 < _ < 9) y
print intersection x y
I have a theory that part of the attraction/intrigue of functional programming for a lot of working programmers comes from the way it inherently emphasizes these ideas, particularly making data flow almost the default form of presentation.
I also have a theory that maybe there's another mental model to be found/learned/created, somewhere above control flow and data flow but below the general domain model, capturing the flow of observable interactions with the outside world: side effects, interactions/dependencies between concurrent operations, and so on. I suspect that part of the reason we find functional programming more difficult for some things (see the “awkward squad” paper) is that when you remove the implicit control flow, you also remove the implicit ordering of events/effects that you get with imperative programming, and there is a cost to that if you don’t replace it (or at least the parts of it that matter for that higher-level view of what your code is doing) with something else.
For anyone curious, I wrote up some of what I learned while exploring the research a few years ago: http://www.clarityincode.com/readability/ (I apologise for the less than stellar formatting; I haven’t updated that page for some time. I also apologise to the actual researchers I cited, if I’ve dumbed down their work too much in aiming for a non-expert audience.)
For the eye tracking reported here, I wonder whether the early emphasis on the top part was a combination of trying to figure out the data flow in the between() function and then its significance to the wider program.
I think it would be interesting to compare the results with a similar eye track of a program written with more emphasis on data flow rather than control flow, e.g.,
It might also be interesting to compare the results with a functional programming language that expresses those ideas more concisely and/or with tools like between() and common() as part of the standard library that programmers would probably be familiar with.Final thought: How much does the absence of a clearly marked starting point (like a main() function in C) affect how a reader approaches unknown code in Python? If this had been a C program, would the reader have aimed straight for main() and then worked down from there to functions like between() and common()?