News for hackers. Stuff that matters.
Re:Life at a startup by kayoone on Tuesday March 08, 2011 @03:38PM (#2302506)
Great story, also looks like the guy really knows what hes doing. That said i read a post by him a while back where he talked about how useless unit tests are and how they dont help finding bugs: http://abstractstuff.livejournal.com/60388.html

I found that quite irritating from somebody who seems so experienced.

Re:Life at a startup by kenjackson on Tuesday March 08, 2011 @06:28PM (#2303046)
Are there any "hacker gods" that are really big on unit testing or use TDD? Most of the recognizable names that I see talk about it are agile book writers. Not actual devs. It was Paul Bucheit who says (from that link), "I'll make a somewhat heretical claim even: Unit test are 20% useful engineering, and 80% fad. They are great for tricky code with well defined behavior (like a parser), but wasteful for most code."

Does Sweeney, Carmack, Cutler, Joy, Thompson, Knuth, Page, Steele, Zuckerburg, Anders, etc... use unit tests in a big way?

I'm actually a fan of unit tests, although I generally dislike TDD.

Re:Life at a startup by abstractbill on Tuesday March 08, 2011 @06:58PM (#2303140)
Nobody in Peter Seibel's "Coders At Work" seemed to have very much time for unit testing as far as I remember, and they pretty much all qualify for that title in my opinion.

I'd stand by pretty much everything I wrote in my original post, for what it's worth, but I guess I don't really understand why it seemed to ruffle so many feathers to be honest.

Unit testing doesn't help me enough for the work I have to put in (compared to things like good logging and monitoring, for example, unless I'm writing something like a parser, with well-specified behavior). It doesn't seem to have helped justin.tv too much either, when we've tried it. But if it helps someone else, then good for them!

Re:Life at a startup by hello_moto on Tuesday March 08, 2011 @07:36PM (#2303251)
I agree with you: if it works for someone else, good for them. Definitely.

I think the main reason why people want to burn your article to ground was because there are way more companies out there that would benefit from their people writing unit-tests.

Now when some lazy developers who think that they're smart starting to quote "the VP of Software Development from JTV sez unit-test is useless", then we have a problem there. These passionate software developers don't want to go back to the time of Wild Wild West where coders slinging code with no reasons and start throwing incomplete shitty crap to QA and call it "DONE", get the gold medals from their team lead for being able to solve things very fast.

You know that movie... Spiderman? "With Power comes great responsibility"... since you're advertising how cool your company is and you're their "VP", then there's that responsibility attached with it.

The thing is, it requires a smart person with high level of discipline, strong principal, and the ability to say "No" to crazy requirements in order to have the luxury of not writing unit-tests. Especially when there's no safety nets (I mean QA) around that person.

I hold nothing against you though.

Re:Life at a startup by kenjackson on Tuesday March 08, 2011 @07:52PM (#2303298)
I think the main reason why people want to burn your article to ground was because there are way more companies out there that would benefit from their people writing unit-tests.

But he's not the only bright dev who tried it an ran into issues or lack of benefit in doing it. The thing that I seem to read from the more zealous of the TDD crowd is "you don't know what you're doing -- you don't understand code/software/testing/development if TDD doesn't obviously work for you".

My concern then is if a large percentage of strong devs don't seem to "get it", does it make sense to push into an organization w/o strong devs?

And the reason I asked the other question about the "hacker gods" is that outside of Josh Bloch (now), I don't really know of outstanding devs who do TDD in a big way. And this scares, because I remember when I used to have certain devs tell me that UML was the way to do everything, and if I didn't use UML throughout the product lifecycle I was no real software engineer -- and UML was the only way that in 20 years people would still be able to understand the design and code. I pushed back on UML and don't regret it for a second. I don't feel like I'm going to regret pushing back on TDD/BDD so far.

Re:Life at a startup by bluesnowmonkey on Tuesday March 08, 2011 @08:22PM (#2303379)
Maybe the confusion comes from conflating unit testing with TDD. Different but related concepts. If you write a unit test after the code, that is no longer TDD.

TDD is a major change in coding style and doesn't seem to work for a lot of people, including myself. Unit testing, on the other hand, is absolutely essential and fundamental to the task of building large, complex systems.

I haven't read Coders at Work, but if the guys in there are doing their thing without some kind of automated testing, I would be very surprised, and I'd wonder why they're making it so hard on themselves.

Re:Life at a startup by v21 on Wednesday March 09, 2011 @03:57AM (#2304347)
But on the other hand unit testing isn't the same as testing. Integration tests, regression tests, stress tests, "miscellaneous automated tests" are all valid sets of tests.
Re:Life at a startup by kmc on Wednesday March 09, 2011 @12:42PM (#2305626)
Have you used a language with strong static checking? It might change your mind about the central role of testing. I'm not saying tests are never necessary in such languages, but you can expect to spend orders of magnitude less time thinking about them.

Random guesses are a poor substitute for logical certainty.

Re:Life at a startup by hello_moto on Wednesday March 09, 2011 @01:13AM (#2304069)
You're confusing TDD and unit-test... and unit-test is not the only safety net around.

Take a look at Dr. Hipps projects. I believe almost all of them have a suite of test automation.

Keep in mind that there was no unit-test library for C/C++ back then. Even now, the unit-test libraries for C/C++ aren't as solid/easy-to-use as those in Java, C#, Python, Ruby.

The software back in the age of Ken Thompson, Dennis Ritchie are different than what we have these days as well.

I agree with nostrademons, some of these so-called "great developers" that you follow, they don't do maintenance programming. They seed a project and let someone else maintain it. Similar to the much-debated "software architect" role; when you don't feel the pain of maintenance programmer, you don't know what's wrong with the process.

Update: UML still exist today. For certain projects, there are values for using UML especially for communicating between people across different functions. Just like anything else: when you abused it, it starts to crumble down.

Re:Life at a startup by sbov on Wednesday March 09, 2011 @02:11AM (#2304182)
Re:Life at a startup by look_lookatme on Wednesday March 09, 2011 @09:35AM (#2304922)
Brad Fitzpatrick talks about unit testing as a method of easing maintainability. I haven't read all of the book but of the profiles I did read, he seemed most similar to the kind of developer I aspire to be as a startup worker.

Your article was a very nice read, btw.

Re:Life at a startup by hello_moto on Tuesday March 08, 2011 @07:25PM (#2303222)
Josh Bloch does.

Zuckerberg and Carmarck are typical geeks that just love coding and don't care about the rest (like documentation, etc). Besides, it's their own company, they can do whatever they want and don't have to deal with the consequences; they have hordes of developers to clean up.

Knuth writes something for himself. He's probably the only coders there.

Anders has thousands of qualified MSc, PhD, and a bunch of other smart people that would write automated test for him (they call them SDET at Microsoft). (Besides, he seems to be an "architecture" kind of guy, does he still write 1k LoC everyday?)

Half of the people you mentioned came from the time before unit-tests were relatively unknown and they believe that to ship means to spend countless night and sleep below their desks. Coding, Coding, Coding is definitely the mantra. They don't know the other type of software development.

If it breaks, just fix it. No punishment.

I don't mean to paint it in such a bad way. If you look at Facebook during the earlier years, if it breaks, who cares, you're not going to lose million of dollars. The situation, culture, environment, products, and market segment are totally different.

Re:Life at a startup by kenjackson on Tuesday March 08, 2011 @07:42PM (#2303266)
Kind of funny. Someone asked almost the same question as I on StackOverflow: http://stackoverflow.com/questions/1266955/are-really-great-...

And got Josh Bloch as the answer.

Not surprisingly, I find myself like Knuth. I like unit tests where I feel very uncomfortable. And I usually known when designing.

At the TDD level it does seem like religion. There doesn't seem to be strong evidence that it hurts nor strong evidence that it helps. A lot of people are successful w/o it and some are successful with it.

Re:Life at a startup by nostrademons on Tuesday March 08, 2011 @08:10PM (#2303339)
There's a bit of an implied selection bias in your question: people get to be hacker gods usually by starting something that later becomes big, and it's in the "just starting out phase" where unit tests are least useful. You never hear about the maintenance programmer that joins five years down the line and finds unit tests indispensable, because there are lots of maintenance programmers and hence little reason to think that any one of them is a hacker god.
Re:Life at a startup by hello_moto on Wednesday March 09, 2011 @01:14AM (#2304073)
This should be written in almost all arguments/discussions about unit-test, especially when people start referring to famous developers who don't write unit-tests.
Re:Life at a startup by gridspy on Tuesday March 08, 2011 @04:36PM (#2302689)
Speaking from personal experience, there are a lot of fads in software development. From the most recent language to the latest and greatest library. From design patterns through to work practices like Agile and TDD.

You must be critical in your thinking for each one of these. Just because it works for someone else doesn't mean it will work for you.

Personally, I think that most design patterns are used as an excuse to make a simple system complex. I think that unit tests are more painful to maintain than they are worth (in many cases). I think that new technology should be approached carefully and only proven tech with minimum lockin should be used.

Don't go round saying that "experienced developers always do X" because a real experienced developer knows that X is great for some things and a huge waste of time for others.

Re:Life at a startup by anamax on Tuesday March 08, 2011 @04:03PM (#2302587)
> I found that quite irritating from somebody who seems so experienced.

Okay, but is he wrong?

Let's be specific. He says "[Unit testing] just doesn't cover a large enough class of potential problems to be generally useful."

He said how he evaluates bug-finding tools and discussed his experience. What is yours?

Re:Life at a startup by nostrademons on Tuesday March 08, 2011 @05:44PM (#2302906)
The dumb part about this debate is that it's being framed as all-or-nothing. The important question isn't "Are unit tests useful?" (the answer to that is "It depends..."), it's "Under which circumstances are unit tests useful?"

If a particular unit test costs you more to write than it saves you in bugs, don't write it. And if it starts costing more in maintenance than it saves in confidence, get rid of it. There's a fair bit of cargo culting going on about how much to test - in my experience, it's not worth testing getters/setters, it's not worth testing simple "if this then set these variables" logic, and it's not worth unit testing database or external calls (you should have some integration tests for them though).

It is very much worth testing tricky cases like a parser, or a bit-level decoder, or complicated algorithms. I just recently got up to 100% code coverage on a UTF-8 decoder I'd written. I found about a dozen bugs in the process. They were silly things, like leaving a digit off a hex constant, but they rippled throughout the decoder so that the results were completely wrong. If I'd just included that in another library without testing, it'd make the dependent library subtly wrong too, and I'd never hear about it until users started complaining, and they'd waste hours trying to find bugs in their code that are really bugs in mine.

I'll also mention one other thing: the point of unit testing isn't to find bugs, it's to give you enough confidence in your code that you can freely change it. The real cost of bugs isn't that one erroneous event that customers will probably never notice, it's that the fear of that one unpredictable event makes management (which is sometimes you, if it's your startup) say "It works. Don't touch it, because you'll break it" and never improve the software. That's how you get shitty software that you can't fix.

Re:Life at a startup by petenixey on Wednesday March 09, 2011 @09:03AM (#2304831)
This is a very good point: are they useful?

I think it's fair to say that integration tests (make sure that the functionality works) are useful; period. Not knowing whether your app is going to work when you push it is not fun.

From a maintenance perspective and assuming that a problem will show up in integration testing or via a bug report, further testing is only really useful to the extent that it lets you debug a failure faster. However that acceleration is a tradeoff against the time taken to write the tests in the first place - ergo: is it really worth it?

I personally don't think the real win from tests is in saving debugging time. I have found it's in making it quicker to write code. I know that sounds crazy as most people view tests as overhead but I view them as development tools.

The key to them being useful though is writing them before you write the code. If you write them afterwards you will have already verified the code in an ad-hoc fashion and the tests are pure overhead. If you write them before they do the verification for you.

I'm coding in Rails and I've found the outside-in approach to development: write the six line Cucumber integration test and then write the RSpec expectations for the object; act as a verified to-do list and make my development far more efficient and robust.

I'm sure that other developers with better focus than I might not get the same benefit but I find it tremendously useful to immediately list down all of the edge cases and general behaviour that a piece of code should exhibit and then keep coding till growl glows green. I trade the time that I'd otherwise spend faffing around in a browser or a debugger for writing tests instead.

It's a bonus that all of those tests then ensure ongoing integrity, the real win is that tests stop me from wandering around the trail and keep me knocking things off in the order that they need to be done. [Caveat: it has taken me months and months to come up to speed on how to write tests well] (great article btw Bill)

[deleted]
Re:Life at a startup by justin on Tuesday March 08, 2011 @07:17PM (#2303192)
FWIW: With Socialcam, Eric implemented unit tests for the API (which powers the client apps), and initially for the website but later we decided against them for the site. Having them on the API definitely has been worth the initial cost as the backend has needed to be very stable for client development, and they are easily testable.

Tim, perhaps you should have made your point more vigorously when you were here :). That said, I think that many / most of our projects fall into the bus count 1, cowboy coder hacking (at least, initially), so we've been able to move fast by not having them.

Re:Life at a startup by nahname on Tuesday March 08, 2011 @05:11PM (#2302800)
He really does not come off well in those comments. Repeatedly saying I am successful, therefore what I am doing is right.

In his defense, it sounds like he has mostly worked by himself. Some practices don't have a very good ROI when you are just hacking away by yourself. I would argue that unit testing is one of them, if you are not very experienced with it anyways.

Re:Life at a startup by gridspy on Tuesday March 08, 2011 @05:43PM (#2302905)
It sounds to me like he got beaten up in his own comments section. The people commenting don't seem to realize that he is talking about a different type of programming than the commenters are doing.
Re:Life at a startup by nahname on Tuesday March 08, 2011 @06:43PM (#2303091)
What do you mean by "type of programming"?
Re:Life at a startup by gridspy on Tuesday March 08, 2011 @07:49PM (#2303289)
I probably should have said "Problem Domain"

I.e Are you an author of a library, a browser, a startup, experimental code, rapidly evolving code, space shuttle code ?

Each one has different measurements of "success" and these can effect your choices re Unit Tests.