News for hackers. Stuff that matters.
Re:Ask HN: How did NASA make reliable software if they didn'... by invaliduser on Tuesday July 19, 2016 @12:01PM (#12122492)
Unit Tests are not the one and only metrics for software quality. Believe it or not, before unit tests were coined, we managed to build reliable software too. Actually a lot of the internet was actually built that way.

But your assumption is very interesting, because it states a very common misconception of quality (whatever that means, reliability, maintainability, resilience, etc):

* "Unit tests imply reliable software". Well, it does not, you'd need reliable unit tests for that, and from what I've seen in the industry, not every unit test out in the wild increases reliability.

* "Without unit tests, you can't have quality software". That is just not true either. First, not all tests are unit tests, end-to-end (integration/functional/whatever) test are extremely useful and raises the quality too. Even without any automated test, the quality of documentation, contract programming, manual testing, etc also help a lot.

Also, note that apart from testing, a lot of different practices contribute to reliability. Separation of concerns, the language level of abstraction, the overall experience and involvements of the developers, and many others.

Re:Ask HN: How did NASA make reliable software if they didn'... by jeremiep on Tuesday July 19, 2016 @12:23PM (#12122634)
I find that the people who are the most obsessed about unit testing generally aren't good programmers to begin with.

To me, keeping code simple and easy to reason about is what yields reliable software. If you can't wrap your head around something in the first place then tests are only going to give you the illusion of reliability.

Its as Rich Hickey put it: "What does every bug have in common? They all passed the type checker and the tests!"

Re:Ask HN: How did NASA make reliable software if they didn'... by pmarreck on Tuesday July 19, 2016 @02:27PM (#12123701)
> I find that the people who are the most obsessed about unit testing generally aren't good programmers to begin with.

I don't think you can generalize this, neither subjectively (my shipping code quality has gone way up since I switched to TDD) nor empirically (evidence seems to indicate that you can achieve up to a 90% production bug reduction with a 15-35% upfront extra time cost doing TDD, which most would consider "worth it"). Secondly, this doesn't mean a very good programmer on a team doesn't have to write tests, if only to prevent others from breaking the behavior expectations of his code down the line. Thirdly, tests serve to validate that behavior if you yourself ever have to go back to the code to change or refactor it, so unless you are SO good of a programmer that you have perfect memory of the mental models of all your past code (which would put you in the extreme minority of all programmers), they will still come in handy.

Re:Ask HN: How did NASA make reliable software if they didn'... by jeremiep on Tuesday July 19, 2016 @03:45PM (#12124295)
Bad programmers are going to make a mess of things no matter how many tests you have. More often than not I see tests as needlessly solidifying the architecture prematurely.

More often than not I see TDD yielding horrible, terrible software architectures and at that point you do need tests to maintain the resulting mess. TDD somehow assumes emergent architectures and I know from experience this is the worst way to architect something.

NPM for example is full of well-tested libraries with poor architecture and bad usability (could be why everything is reinvented every 6 months) but I wouldn't call that a success story.

I'm willing to bet the bug reduction from the extra time upfront doesn't come from doing TDD but actually thinking about the requirements and how they fit the larger picture.

I really like the following story to illustrate this point:

http://ravimohan.blogspot.ca/2007/04/learning-from-sudoku-so...

Re:Ask HN: How did NASA make reliable software if they didn'... by pmarreck on Wednesday July 20, 2016 @08:18AM (#12128388)
That blog post simply shows that Ron Jeffries failed to understand the core problem of "how to solve Sudoku puzzles" (or lost interest before he succeeded, which explains why he spent so much time on the data representation aka "bike shedding" or "yak shaving") and is no indictment of TDD, which is not a magic elixir that allows you to "organically" solve every possible problem.

> horrible, terrible software architectures

1) Need an example. 2) There is no evidence that the architecture would have not been even MORE horrible AND terrible without the TDD involvement

> needlessly solidifying the architecture prematurely

As all code (whether "test code" or "tested code") is wont to do. Perhaps the fallacy here is considering test code as a separate, "optional" entity in your codebase instead of as an integrated proof of it working as advertised.

> willing to bet the bug reduction ... comes from thinking about the requirements

Still not an argument against TDD. If anything, that's an argument FOR it. The difference being at the end of the day, you have a working proof of the satisfaction of your requirements.

Whiteboarding and TDD are not mutually-exclusive. I don't think that TDD at all implies "not thinking about the problem" nor does using it turn a crappy programmer (who would tend to choose terrible designs over better ones) into a good one. I think you are still constrained by the skill of the programmer no matter whether you use TDD or not. As TDD is just human-produced code, like all code.

NPM could have had lots of fresh out of college monkeys work on it for all I know. It certainly wouldn't surprise me. You're right that TDD won't fix being an inexperienced programmer, but that's not an anti-TDD argument, that's an anti-inexperienced-developer argument.

Lastly, you failed to address the other arguments I made in favor of using TDD (note: there's some conflation here between "unit testing" in general and TDD specifically unfortunately)

Re:Ask HN: How did NASA make reliable software if they didn'... by jeremiep on Wednesday July 20, 2016 @09:39AM (#12128888)
Lets just take GUI then, how can you design that using TDD? Lets say you've got a multiple steps form where the user can go back and forth between steps and change data anywhere. How are you going to test for ALL possible flows, which are counting in the millions by now.

Software reliability doesn't come from testing the obvious use-cases and calling it a day. You'd get the same result by just booting the program and having all these functions execute. Bugs are always going to be in the use-cases you didn't think to test, and there's always going to be orders of magnitude more use-cases than tests.

What is incredibly hard to get right are complex sequences of state mutations. This is where most of the bugs are going to be and this is where TDD breaks badly.

I'm not saying unit tests and TDD are bad, I'm saying they're overrated. They're one tiny tool you get to verify the reliability of your software and for a lot of application domains they are completely useless.

More often than not I see pro-TDD people not knowing concepts like property-based testing which actually can model complex user interactions. And once they learn about it their architecture is already crippled by having TDD drive the design of individual components but not the whole thing and now there's no sweet spot to hook generative testing or reason about the program as a whole.

Re:Ask HN: How did NASA make reliable software if they didn'... by pmarreck on Thursday July 21, 2016 @12:52PM (#12138162)
How to TDD a GUI:

http://stackoverflow.com/questions/382946/how-to-apply-test-...

It works.

I would have posted this response yesterday, but when I went to the stackoverflow link at the time, they were actually down :O

Re:Ask HN: How did NASA make reliable software if they didn'... by jeremiep on Friday July 22, 2016 @10:18AM (#12143634)
These answers point to dead links and reeks of Agile.

They mention frameworks and MVC variations; from the start they've already bloated whatever they're building and none of them address the real problem: complex sequences of state mutations.

So not only are they having software that's orders of magnitude more complex than it should be, they're also completely avoiding the hard problems.

You won't see a test like "enter a username, dont wait for server validation, enter an invalid email, optimistic updates, fix email, more optimistic updates, server response & state merges, advance to step 2, 3, come back to step 1, change email, advance to step 4, crash.. wat?!"

That's where the bugs are. Not "click the button and the popup opens" because that's trivial. You're also probably not testing that button while a menu is open, or a dialog already opened, or between optimistic updates and server responses, or the combinations of these, or ALL the cases you're going to forget.

You're not going back to test each new feature against every single possible state the app can be in. That's why I say TDD is overrated because the problems it fixes are trivial and it ignores the important ones.

Re:Ask HN: How did NASA make reliable software if they didn'... by pmarreck on Saturday July 23, 2016 @01:42PM (#12150262)
> complex sequences of state mutations.

Which is where immutability and functional programming win... coupled with TDD you get a major delivered-bug reduction

Re:Ask HN: How did NASA make reliable software if they didn'... by jeremiep on Monday July 25, 2016 @10:03PM (#12162925)
You have state even in functional programs. My original point was about simplifying the program so it is easier to reason about in the first place, rather than write tests for a yet-to-be-overengineered-mess :)

What if you get very tight memory/cpu constraints and FP/immutability isn't available/affordable? What if your programs runs on millions of networked nodes simultaneously?

GUIs are still very stateful even with FP for one! So if you're going to write property-based tests after you know what the properties of the system are, and these will cover way, way more cases than you'll ever think of writing tests for manually, why even bother?

What I do see is people deluding themselves into thinking their software is reliable when it definitely isn't and they're now even less careful when developing because they've got tests to protect them.

Re:Ask HN: How did NASA make reliable software if they didn'... by mathgeek on Tuesday July 19, 2016 @01:12PM (#12123058)
> I find that the people who are the most obsessed about unit testing generally aren't good programmers to begin with.

I'm inclined to think this could be generalized to those who are most obsessed about any specific practice.

Re:Ask HN: How did NASA make reliable software if they didn'... by ams6110 on Tuesday July 19, 2016 @02:06PM (#12123548)
A very non-PC joke from George Carlin springs to mind...
Re:Ask HN: How did NASA make reliable software if they didn'... by entee on Tuesday July 19, 2016 @02:30PM (#12123716)
I think unit tests are a great tool to motivate code that is simple and easy to reason about. After all, a good unit test is itself simple and easy to reason about. It also forces you to think about what is the simplest unit of software for a given task, helping design the software it is testing.

Also, I've definitely had code I thought was well tested and good to go, but when I rigorously went back and wrote unit tests, I found subtle bugs I wouldn't otherwise have known to fix.

That said, yeah it's not all it takes to get things right. Dogma is bad, but unit tests are an essential part of the toolkit for ensuring code quality.

Re:Ask HN: How did NASA make reliable software if they didn'... by kedean on Tuesday July 19, 2016 @05:00PM (#12124849)
Yeah, i've found that my heavily unit tested code is often my best code, simply because designing it with that in mind yields the simplest, smallest units. If I know I'm gonna need to write a unit test later, I'm much more inclined to split a long function out into more sensible small ones, and I'm not going to try to combine two different behaviors (because that's hell to test).
Re:Ask HN: How did NASA make reliable software if they didn'... by Cacti on Tuesday July 19, 2016 @12:37PM (#12122735)
As the saying goes, crap in, crap out. It's just as easy to write crappy unit tests as it is crappy code.
Re:Ask HN: How did NASA make reliable software if they didn'... by tmm on Tuesday July 19, 2016 @12:54PM (#12122895)
Or, as W. E. Deming wrote: "You cannot inspect quality into a product."
Re:Ask HN: How did NASA make reliable software if they didn'... by danudey on Tuesday July 19, 2016 @02:39PM (#12123786)
After three years of working on some software (basically some fancy shell scripting but in Python), when the project had grown dramatically beyond its original scope through a host of hacks and bolted-on features, I remembered that the software had unit tests and ran them.

Surprisingly, all the unit tests passed, and my first thought when I saw that was "Man, I bet these unit tests are awful."

Re:Ask HN: How did NASA make reliable software if they didn'... by finishingmove on Tuesday July 19, 2016 @01:37PM (#12123286)
And it's extremely hard to write unit tests for crappy code.
Re:Ask HN: How did NASA make reliable software if they didn'... by Cacti on Tuesday July 19, 2016 @01:51PM (#12123426)
Extremely hard to write _good_ unit tests for crappy code.
Re:Ask HN: How did NASA make reliable software if they didn'... by teej on Tuesday July 19, 2016 @12:38PM (#12122760)
Could to into more detail about what techniques NASA specifically used to achieve quality software? I'm honestly very interested. I feel like you didn't answer the spirit of the question "How did NASA build highly reliable software" while pigeon holeing on "without unit tests".
Re:Ask HN: How did NASA make reliable software if they didn'... by Cacti on Tuesday July 19, 2016 @12:46PM (#12122825)
Their hardware was essentially fixed, well defined, and limited in scope.

They reviewed every single piece of code many, many times.

They were draconian about memory handling, loops, preprocessor directives, and recursion.

All functions had to check all return types from other functions.

No more than one level of pointer dereferencing.

All code bodies had to fit on a piece of paper.

/edit should add, they used their tools thoroughly: compilers were run on max warning levels and absolutely no warnings were accepted, ever. They also used several static analysis tools on the code (under-appreciated these days!).

Re:Ask HN: How did NASA make reliable software if they didn'... by yardie on Tuesday July 19, 2016 @01:03PM (#12122979)
I'd also add they had excellent, mature coders.

None of this crunch time bullshit, though I'm sure it happened.

I've been on one of those binges. If another coder hadn't seen my errors, due to diminishing sleep, I'd probably keep banging my head against the wall. Just compounding the situation.

Re:Ask HN: How did NASA make reliable software if they didn'... by Jtsummers on Tuesday July 19, 2016 @01:40PM (#12123314)
Anecdotal generalization: I've seen better code from the 40-hour-a-week guys than the ones that do 80 hours of overtime each week.

That doesn't mean the former finish first, they usually don't. But their code tends to be more thoughtfully constructed. They have a vested interest in not spending their time in the office, and that means minimizing debugging time along with coding time.

When you're in a hot field with lots of competition, maybe those sprints make sense. But in the field of NASA-type systems, there may be a deadline, but it's rarely urgent. This gives the developers much more time to step back, breathe, and think.

Re:Ask HN: How did NASA make reliable software if they didn'... by danudey on Tuesday July 19, 2016 @02:41PM (#12123806)
I can't count how many times I stayed late at the office working on a problem for 6 hours straight, and then after a good night's sleep found and fixed the issue after 20 minutes at work the next day. Sleep is king.
Re:Ask HN: How did NASA make reliable software if they didn'... by yardie on Tuesday July 19, 2016 @03:55PM (#12124361)
I have latent dyslexia which seems to emerge after too many hours at a computer screen. I've come back from a good nights rest to find variable names that are just slightly misspelled, recursive loops that are unreachable, and functions that are incomplete because I was distracted by a bug hunt.
Re:Ask HN: How did NASA make reliable software if they didn'... by mywittyname on Tuesday July 19, 2016 @05:55PM (#12125235)
I think we all do this to some degree. When I'm working on a difficult problem, it feels like Christmas Eve, because I know I'll wake up and have the solution coded up before my coffee is gone.
Re:Ask HN: How did NASA make reliable software if they didn'... by finishingmove on Tuesday July 19, 2016 @01:39PM (#12123302)
An article was recently featured here, that said functions were limited to 60 lines. That seems like way too loose of a restriction considering the possible consequences.
Re:Ask HN: How did NASA make reliable software if they didn'... by Cacti on Tuesday July 19, 2016 @01:49PM (#12123407)
One page of text at standard size, with standard style (one statement or declaration per line). Which is roughly 60 lines.

If you follow their other guidelines (no recursion, fixed sized loops, etc.), then it can take you several lines to do much of anything, so 60 isn't very long. Those things combined means you're basically saying that each function should only be attempting to do 2-3 significant things, which seems about right to me, as most people can't fully understand more than 3 things simultaneously at one go.

Re:Ask HN: How did NASA make reliable software if they didn'... by jotux on Tuesday July 19, 2016 @01:08PM (#12123022)
Re:Ask HN: How did NASA make reliable software if they didn'... by nradov on Tuesday July 19, 2016 @03:26PM (#12124164)
Much of that is covered in "The Capability Maturity Model: Guidelines for Improving the Software Process". The book is out of print now but available fairly cheap and it's well worth reading.

https://www.amazon.com/gp/product/0201546647?ref%5F=sr%5F1%5...

And here's some further summary information and reading recommendations from Steve McConnell.

http://www.stevemcconnell.com/sgcrib.htm

Re:Ask HN: How did NASA make reliable software if they didn'... by zem on Tuesday July 19, 2016 @02:47PM (#12123838)
this article is a very good read on the topic: http://www.fastcompany.com/28121/they-write-right-stuff
Re:Ask HN: How did NASA make reliable software if they didn'... by sebastianconcpt on Tuesday July 19, 2016 @01:50PM (#12123415)
Indeed, you did catch the spirit of the question right :) Thanks
Re:Ask HN: How did NASA make reliable software if they didn'... by iweinfuld on Tuesday July 19, 2016 @02:10PM (#12123583)
Another assumption there is: Nasa had reliable software. From what stories I heard, I think that may not be true.