News for hackers. Stuff that matters.
Re:An External Replication on the Effects of Test-driven Dev... by smallnamespace on Wednesday October 19, 2016 @01:01AM (#12741206)
The criticism was very constructive though. Increase the sample size, put it in a more realistic setting.
Re:An External Replication on the Effects of Test-driven Dev... by vacri on Wednesday October 19, 2016 @02:18AM (#12741521)
How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money? They can't be left to do the problems in their own workplace, or the next criticism will be "uncontrolled variables!". They also have to be vetted for minimum skills (there are plenty of experienced, professional devs out there who aren't worth a second look). The parent also wants more complex tasks done.

So... where is the money coming from? Who is going to pay for this multitude of professional programmers to converge to the same environment, be vetted, and spend a non-trivial amount of time coding the same thing as the others in the group?

Of course the researchers in the article would have loved to have those kind of resources and do the perfect, wide-ranging, deeply detailed study, but the OP's criticisms just show how divorced the OP is from experimenting with real-world humans in real-world situations, and with real-world resources.

Re:An External Replication on the Effects of Test-driven Dev... by smallnamespace on Wednesday October 19, 2016 @02:45AM (#12741616)
This may sound harsh, but taking the researcher's difficulties into account is not our responsibility.

The research presented here is weak. Honestly pointing that out without pulling punches is better than simply giving them a pass because 'doing good research is hard'.

Re:An External Replication on the Effects of Test-driven Dev... by vacri on Wednesday October 19, 2016 @06:36AM (#12742599)
Where did I say 'simply give them a pass'? This idea that research is either a polarised "ideal" or "trash" is moronic. Taking the nature of any study into account is part of science, and part of how you caveat the knowledge gained from that study.
Re:An External Replication on the Effects of Test-driven Dev... by smallnamespace on Wednesday October 19, 2016 @02:41PM (#12746475)
I completely agree with you. Pointing out a study's flaws is indeed part of the process of 'caveat[ing] the knowledge gained from that study'.
Re:An External Replication on the Effects of Test-driven Dev... by vidarh on Wednesday October 19, 2016 @04:03AM (#12741955)
You could start by finding a corporate sponsor with a lot of developers and a vested interest in finding out which methods will be best for them. The advantage of that being that you could even test it on a real project (the sponsor would "just" need to be willing to dedicate twice the number of developers to a suitably small project).

It'd still not be easy, and of course there'd still be issues (e.g. is there anything about the corporate culture or training in that company that would affect the result?), but it'd still be far better than a bunch of students and toy problems.

Re:An External Replication on the Effects of Test-driven Dev... by vacri on Wednesday October 19, 2016 @06:42AM (#12742625)
Corporate places do research like that all the time; they just don't publish them all that often. A corporation with a vested interest is a corporation with a competitive interest.
Re:An External Replication on the Effects of Test-driven Dev... by flukus on Wednesday October 19, 2016 @02:48AM (#12741627)
Maybe we have to accept that it's untestable.

Though I think we could derive some more realistic scenarios, like evolving requirements and switching developers mid project that would be more enlightening.

Re:An External Replication on the Effects of Test-driven Dev... by yomly on Wednesday October 19, 2016 @03:45AM (#12741868)
Whereas the cost of finding the Higgs Boson was only a mere $13.25Bn, cheap compared to all those pesky extortionate SV engineers...

As others have said, good research costs money...

Re:An External Replication on the Effects of Test-driven Dev... by mwat on Wednesday October 19, 2016 @02:51AM (#12741638)
We can't arrange a good study, so let's churn out deeply flawed ones instead.
Re:An External Replication on the Effects of Test-driven Dev... by icebraining on Wednesday October 19, 2016 @02:49AM (#12741629)
How do you propose to gather more experienced, professional developers into the same location and get them to work on a topic that isn't making them tons of money?

Hackatons seem to manage. Why not set one up?

Re:An External Replication on the Effects of Test-driven Dev... by vacri on Wednesday October 19, 2016 @06:40AM (#12742615)
Hackathons wouldn't satisfy the OP's requirements for complexity, nor demographics - you'd be looking at a self-selecting group of highly motivated people, skewing young, who would come together hackathon style. There aren't going to be many thirty- or forty-something coders with young families spending a weekend (to work on the same set problems as everyone else) at the hackathon, yet there are plenty of those in industry.