News for hackers. Stuff that matters.
Re:Unit testing, Lean Startup, and everything in-between by kkapelon on Sunday March 12, 2017 @05:36PM (#13853263)
I can't believe that in 2017 we still need to discuss if the benefits of tests outweigh their costs.

Anyway, the article seems like a brain-storming session from the author, so I am not sure that I need to answer all the points. Those that stick out

1)As other people said already you always write a unit test against a spec. If you have no spec, then unit tests are not really helpful. When you prototype something you don't need unit tests. Most startups also would not need a vast test suite in their early stages

2) The company that had only UI tests forgot to make them resistant against ui changes. I don't know the tool they were using but most GUI tools support the Object/pages pattern. If the UI was changing so much that even that was not enough, then they did not have spec (see point 1)

3) A/B testing is not unit testing because by definition you don't have a spec (if you are still trying to find if A or B are ok)

4) Code review is not unit testing because you will catch only obvious stuff. Code review is more about the structure of the code than its correctness. It far easier to judge correctness by automation than by visual inspection.

Basically I would suggest you to remove the word "unit" from your blog post title.

Unit testing has a very clear definition and you are confusing it with code reviews, A/B tests and other unrelated stuff

google also "the testing pyramid" for a sound model of unit testing. The tax preparation company would do fine to follow this.

Re:Unit testing, Lean Startup, and everything in-between by itamarst on Sunday March 12, 2017 @06:45PM (#13853638)
The blog post isn't about defining unit testing, it's about defining testing. I'll make that more clear when revising, yes.

The testing pyramid, IIRC, starts with the presumption that the goal of testing is quality. There are other goals for testing, though, and I say in the intro quality is too hand wavy to be useful in decision making.

Re:Unit testing, Lean Startup, and everything in-between by kkapelon on Sunday March 12, 2017 @08:07PM (#13854118)
The title clearly starts with "unit testing".
Re:Unit testing, Lean Startup, and everything in-between by itamarst on Sunday March 12, 2017 @08:17PM (#13854176)
PyCon talk title (at least, what I proposed) is "Big picture software testing: unit testing, Lean Startup, and everything in between." Which seemed cumbersome, so I tried this iteration without the prefix. Will need to come up with something better if it's confusing to readers/listeners, though.
Re:Unit testing, Lean Startup, and everything in-between by dkersten on Monday March 13, 2017 @04:39AM (#13856157)
It also ends with "and everything in between". The title gave a spectrum of testing which the article then discusses.
Re:Unit testing, Lean Startup, and everything in-between by qwer on Monday March 13, 2017 @12:29AM (#13855235)
There are more words after that though.
Re:Unit testing, Lean Startup, and everything in-between by yomly on Sunday March 12, 2017 @07:10PM (#13853826)
So you've got a working prototype, which is 80% of the way towards your spec (which changes by the day, mind you) where and when do the tests then get written? Do you start from scratch when you go from prototype to product? Is that feasible in a tiny lean startup with few developers and real hard fast deadlines?
Re:Unit testing, Lean Startup, and everything in-between by itamarst on Sunday March 12, 2017 @07:23PM (#13853903)
Going from prototype to tested code is possible.

Here's how I'm doing it, for command line tool that provides local development environment for remote Kubernetes cluster (https://datawire.github.io/telepresence/):

1. Build prototype, test manually.

2. Have coworkers try it out.

3. Fix UX based on feedback, test manually.

4. Once initial UX was nailed down, write end-to-end tests.

Beyond adding features, next I will rewrite the core using test-driven development (TDD), so that the internals are maintainable. and the end-to-end tests will ensure the basic functionality continues to work.

The UI is fairly simple, and at this point I have confidence that it won't change multiple times in a day, so it's worth the effort to stop relying solely on manual testing and write some automated UI tests.

Re:Unit testing, Lean Startup, and everything in-between by kkapelon on Sunday March 12, 2017 @08:07PM (#13854113)
If you have a spec that changes by the _day_ = it means you don't have a spec. You are still prototyping.

So yes, start writing unit tests when you no longer prototype.