Research is great except when its results are misinterpreted, its tests are poorly designed, etc.

I could easily redesign the horizontal whitespace test to show that horizontal whitespace matters.

Compare: spot the error

    class FruitCollection 
    {
    pubic:
      FruitCollection()
          : num_apple(0),
            num_apricot(0),
            num_avocado(0),
            num_banana(0),
            num_breadfruit(0),
            num_bilberry(0),
            num_blackberry(0),
            num_blackcurrant(0),
            num_blueberry(0),
            num_currant(0),
            num_cherry(0),
            num_cherimoya(0),
            num_clementine(0),
            num_cloudberry(0),
            num_coconut(0),
            num_damson(0),
            num_date(0),
            num_dragonfruit(0),
            num_durian(0),
            num_elderberry(0),
            num_feijoa(0),
            num_fig(0),
            num_gooseberry(0),
            num_grape(0),
            num_grapefruit(0),
            num_guava(0),
            num_huckleberry(0),
            num_honeydew(0)
            num_jackfruit(0),
            num_jettamelon(0),
            num_jambul(0),
            num_jujube(0),
            num_kiwifruit(0),
            num_kumquat(0),
            num_legume(0),
            num_lemon(0),
            num_lime(0),
            num_loquat(0),
            num_lychee(0),
            num_mandarine(0),
            num_mango(0),
            num_melon(0) 
      {
      };
    };

Now spot the error here

    class FruitCollection 
    {
    public:
      FruitCollection()
          : num_apple(0)
          , num_apricot(0)
          , num_avocado(0)
          , num_banana(0)
          , num_breadfruit(0)
          , num_bilberry(0)
          , num_blackberry(0)
          , num_blackcurrant(0)
          , num_blueberry(0)
          , num_currant(0)
          , num_cherry(0)
          , num_cherimoya(0)
          , num_clementine(0)
          , num_cloudberry(0)
          , num_coconut(0)
          , num_damson(0)
          , num_date(0)
          , num_dragonfruit(0)
          , num_durian(0)
          , num_elderberry(0)
          , num_feijoa(0)
          , num_fig(0)
          , num_gooseberry(0)
          , num_grape(0)
          , num_grapefruit(0)
          , num_guava(0)
          , num_huckleberry(0)
          , num_honeydew(0)
            num_jackfruit(0)
          , num_jettamelon(0)
          , num_jambul(0)
          , num_jujube(0)
          , num_kiwifruit(0)
          , num_kumquat(0)
          , num_legume(0)
          , num_lemon(0)
          , num_lime(0)
          , num_loquat(0)
          , num_lychee(0)
          , num_mandarine(0)
          , num_mango(0)
          , num_melon(0) 
      {
      };
    };

Similarly the rectangle with tuples. I'm only guessing on this one but I suspect if you change it to vectors then

    a = Vector(1, 2, 3, 4)
    b = Vector(3, 4, 5, 6)
    c = mul(a, b);

Is far more understandable than

    a = Vector(1, 2, 3, 4)
    b = Vector(3, 4, 5, 6)
    c = mul(a.x, a.y, a.z, a.w, b.x, b.y, b.z, b.w);

And that the more complex the math the worse it would get. Even for rectangles a larger example that did various intersections and unions I suspect the results would change.

I haven't had time to brush up on C++11 yet. Is pubic a new keyword?

In all seriousness, you make some good points, but I don't think they invalidate the findings so much as they demonstrate avenues for further inquiry.

I agree, and I was astounded when I found out they had never done an end to end study of how people use Visual Studio while I was at MSFT.

It surprises me how little we really know about the human aspects of our own field. The practical/pragmatic side of understanding how we really program, what's hard, where we stumble, what we do easily, etc is something that we're trying to address with Light Table. LT happens to be the perfect playground to do this sort of experimentation, so at the very least we'll be making it easier for researchers already in the space. But as we get further along and have a few more hands to spare, we'll start picking off research questions ourselves.

The more the merrier, right? :)

> I agree, and I was astounded when I found out they had never done an end to end study of how people use Visual Studio while I was at MSFT.

As someone who uses Visual Studio everyday, I am not surprised by this at all. Don't get me wrong, Visual Studio is one of the best IDE's out there, but sometimes it's inspite of itself.

Like removing functionality from the Text Explorer in VS 2012, which they have finally been patching back in. Dumb, dumb, dumb. If they HAD end-to-end user testing, they never would've remove the functionality in the first place.

FWIW, while I was there I ran the first end-to-end usability study. It was... enlightening :)

Wow, what did you learn?

There is tons of research like this. There are multiple conferences dedicated to research like this (e.g.: http://www.program-comprehension.org/ ). You just need to go out and find it instead of waiting for it to appear on Hacker News.

It's cool to see this research, but it really needs to become more accessible and widely applied. I'll start digging in, thanks for the pointer.

>We need to have more research like this.

There is a lot of research done in this and other areas of Software Engineering. Problem is that it's usually locked away behind a huge paywall. If you're not affiliated with some university, you essentially have no access to it.

Another problem is that the research hasn't been collected into a manager-friendly tome like the Gang of Four book. The software industry is overflowing with snake oil, and we need to replace it with science if we want to see this turn into a real engineering discipline. I'm tired of bikeshedding and holy wars -- show me the science!

If it involves studying human behaviors, it's probably not going to be science. I would prefer the holy wars over some manager telling me it's been "proven" that one side is correct.

Good point. The thought of a manager saying that sent a chill down my spine.

There's "Making Software" by Oram and Wilson: http://shop.oreilly.com/product/9780596808303.do

> Problem (as with all academic papers) is that, it's usually locked away behind a huge paywall.

This may be true for research in other domains but as a researcher in Computer Science I can assure you that the number of academic papers locked behind a paywall is negligible in CS. Only once was did I find a paper not available and I promptly received a copy by sending an e-mail to the authors of the paper.

My experience is the same. I work in NLP, and I've found that at least everything administered by ACL (probably our largest professional organization) is pretty much always available.

I love that about ACL. Not only is open-access good and nice and all that, but it also means I don't have to screw around with VPNs and such when I need to get an article while I'm at home.

The only thing I do not like about ACL is the fact that is not indexed in DBLP. I have a paper in one of the ACL affiliated conference and it does not show up. I have since then shifted from core NLP and it doesn't really matter but it would be nice to be consistent with other indexing mechanisms.

almost everything I need I find in CiteSeer or ACM Digital Library (which does have a very small pay wall)

ACM charges $200/year or $15/paper. A paywall is a paywall.

Out of curiosity - which area do you work in? Can you also post a couple of examples (from your area) of academic research papers not available for free download?

I've been out in the wilds of industry for a while now, so it's been a while since I needed to pay for something professionally... back in the halcyon days of academia, I worked in computational medicine and had (still have) interest in lingustics/neuroecon/neurosci and I still run into walls there occasionally. I can't cite specific examples for you, if I can't find it somebody I know can get me a copy.

> I worked in computational medicine and had (still have) interest in lingustics/neuroecon/neurosci and I still run into walls there occasionally.

Like I said earlier areas like computational Medicine, Biology and other such areas run into pay walls. CS research areas like theory, data mining, NLP etc. will rarely run into such problems.

Thanks! We'll be following up soon with an analysis of our eye-tracking data. I'm getting permission to post the data, so I hope to have the complete data set and videos up when the paper comes out.