If you try to abstract away tests, you often just end up re-implementing the same abstractions used in the actual code, and you can end up not catching unfounded assumptions that your abstraction is making in both the tests and the code. There is a scope for having test helpers / utils to make tests easier to write, but you should be minimalist with these.

I agree, but some test helpers are reasonable. It's about balance and readability of the tests in question. Otherwise integration tests end up being a hundred lines of code.

The problem with abstractions in testing is - you should test it :)

Tests should also be treated as a form of documentation. A test should reflect the way you'd use the code in real life as closely as feasible.

I'm not sure I agree. I like to move all setup code to helper methods so that my tests are just a few lines

  // Given <setup>
  
  // When <thing happens>

  // Expect <thing> to be <such>

This allows the reader to easily see which workflows are actually tested. If the reader is interested in implementations the utility code is one click away and usually only needs to be looked at once to be completely understood. The test bodies themselves however have many flavors for many work-flows so getting rid of the repetition is critical to highlight the specific nature of individual tests.

I do think a lot of test frameworks would do well to support data-driven testing a la Spock: http://spockframework.org/spock/docs/1.0/data_driven_testing...

Terrible acronym for a good idea: DAMP

stackoverflow.com/questions/6453235/what-does-damp-not-dry-mean-when-talking-about-unit-tests

Tests rarely have bugs, I find, so generally dry isn’t critical. Also, dry is for security (see below)

Tests frequently have bugs, especially bugs that result in the test passing when it should fail.

That’s why I make the test fail before writing the code. If the code is already written, then I break it in the minimal way to test the test, and then fix it.

IME things never fail in the way you expect them to... You can build a fortification where you think you are weak only to find the enemy is already in the castle.

A green test does not equal bug free code. There may be a misimplementation / misunderstanding of the spec or your code passes beautifully under right conditions, with just this data setup.

That's a test for your test, so why only run it once transiently instead of running every time? "Mutant" testing helps with this. It's basically fuzzing your test code to make sure that every line is meaningful.

I can see how that would be useful, but I also think it's a matter of priorities.

I'm basically saying I rarely have bugs in my tests because I verify them first. In fact I can't think of a single bug in my tests over the last 4 years (or even 10 years), but I can think of dozens of bugs in my code.

For example here are some pretty exhaustive tests I've written for shell, which have exposed dozens of bugs in bash and other shells (and my own shell Oil):

https://www.oilshell.org/release/0.7.pre11/test/spec.wwz/osh...

I would rather spend time using the tests to improve the code than improving the tests themselves. But I don't doubt that technique could be useful for some projects (likely very old and mature ones)

ITYM "I rarely have bugs in my tests that I'm aware of". The amount of tests I've seen that look like they are working properly, have been "verified" and are buggy is huge. Usually we find they were buggy because someone moves some other tests around or changes functionality that should have broken the tests, but didn't.

Please, don't ever assume that your tests are beyond reproach just because you verified them. Tests are software as well and are as prone to bugs as anything.

And how do you do this in practice? I am struggling to think of a good way to keep the production code that fails the test and the production code that doesn't fail the test together. I might have my test check out an old version of the production code, compile it and test against that. But that is hard to get right.

Tests always have bugs, it's just that you don't know about the edge cases yet.

As a preventative measure, I write some tests for my tests. Also in TDD style of course. And on a very rare occasion, I have to write a test for those tests as well.

It’s time for TTDD. Start by writing tests for your tests :)

I do actually do that. I'll write some buggy code in order to learn how to test for it.

TDD for me is primarily a way to guide myself toward accomplishing a goal. So I sometimes write way more tests for myself than the business needs. I will then delete the scaffolding tests before I tag my PR for review.

Tests have fewer bugs if you write them before the system under test, and if they don't have mocks, and if you have enough of them that anything you get wrong the first time will get noticed by the results of the many similar tests.

> Tests have fewer bugs ... if they don't have mocks

100x this. I've repeatedly fail to convince my team members that mocks are unnecessary in most cases. I've reviewed code with mocks for classes like BigDecimals and built-in arrays. This is especially prevalent in Java teams/codebases.

Drives me insane.

> Tests rarely have bugs, I find, so generally dry isn’t critical

DRY is important for tests, but the areas where it is have probably (if you aren't writing a testing framework or using a language that doesn't have one that covers your use case) largely covered by your testing framework already.

Tests are there to catch issues. In a way, almost every bug that makes it to production is a test bug.

Not to mention the inconsistent tests that fail so randomly from timing issues that the team ignores them