News for hackers. Stuff that matters.
Re:Coverage is not strongly correlated with test suite effec... by jacobsenscott on Monday September 27, 2021 @08:20PM (#28677824)
It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful.

Test first also improves the quality of the tests just because it forces you to write a failing test first, and then write the code to make it pass, and you do this over and over as you build up the production code. In this way you are actually proving that the code you are adding is making tests pass.

That doesn't mean your tests are great - you are probably still mostly testing the happy path. But then you have a solid foundation to layer mutation/fuzz testing on.

I can't tell you how often I see people write tests after writing the production code - and I can go and delete most of the production code and the tests still pass. Writing quality tests for existing code is much harder than writing tests for code that doesn't exist - it is counterintuitive if you haven't done it both ways.

Re:Coverage is not strongly correlated with test suite effec... by kragen on Monday September 27, 2021 @09:01PM (#28678098)
I think it was Brian Marick who pointed out that the great benefit of a coverage report is that it tells you what you forgot to think about when you wrote the test suite. One response to code that isn't covered is to write the tests to exercise that code, but there are a couple of other possibilities he suggests might be better:

1. Can the uncovered code be removed from the system entirely? Maybe if none of the tests for other parts of the system invoked it, it's not used at all. (This is less likely if you use a lot of test stubs.)

2. Maybe if you didn't think about the scenario where the uncovered code gets invoked when you were writing the test suite, there are other things you also didn't think about—and maybe some of them aren't covered in the implementation either. Write down the missed case and put the implementation away for a while—hopefully long enough to forget how it's implemented. Once you've forgotten, refer to your notes and write a suite of tests that covers the missed case as well as anything similar.

— ⁂ —

I agree that test-first improves your tests in the way you describe: your test suite is guaranteed to have nearly complete coverage. Also, you have some evidence that the test itself works rather than vacuously passing.

But I think there are two other benefits of test-first programming that are commonly undersold.

First, it makes programming more fun, because you have immediate feedback when you make a test pass.

Second, sometimes you write a test for code you haven't written yet, and you can see by looking at the test that your design sucks: you need seven objects and six method calls to do something simple, and one of the method calls has a boolean parameter, making the code incomprehensible. This feedback allows you to improve your design, possibly several times, before writing the implementation. This allows for faster design iteration than refactoring the implementation toward a better design does. I think this is what jeffbee is saying in https://news.ycombinator.com/item?id=28677978.

Re:Coverage is not strongly correlated with test suite effec... by vitus on Monday September 27, 2021 @09:29PM (#28678255)
I find test-first most useful when debugging a hairy problem, and I think that's somewhere it's not used enough.

Reproducing the bug consistently* is the first step to understanding how it works, and how to fix it. And then you get the thrill of trying to turn that test green as you tinker.

...

* Okay, sometimes "consistently" is "fails about 1/100 executions", but that's not so bad if you can run your unit test 1000 times in the span of 30s.

Re:Coverage is not strongly correlated with test suite effec... by kragen on Monday September 27, 2021 @09:47PM (#28678380)
Oh, yeah, that's a good point; even people who don't do test-first most of the time will often do it for bug fixes.

I think the importance of performance to software quality via automated testing is profoundly underestimated.

Re:Coverage is not strongly correlated with test suite effec... by optymizer on Tuesday September 28, 2021 @12:04AM (#28679210)
I found TDD useful for a well defined problem or an agreed-upon API. For apps for example, especially those not well defined and designed as-you-go, where the designer and PM might change their minds frequently after toying around with the app or getting user feedback, TDD is a lot of overhead and tests after writing the code are primarily useful for preventing regressions when somebody else changes your code.
Re:Coverage is not strongly correlated with test suite effec... by jacobsenscott on Tuesday September 28, 2021 @01:18AM (#28679591)
I hear this a lot, but the result is usually an untested, and usually nearly untestable (because it was written without tests), prototype with a few characterization tests that gets pushed to production and haunts you for the rest of the life of the product.

Pototyping to define the problem or API is fine, but most people don't have the discipline to tear it out and start over when they finally do have a well defined problem.

It is somewhat unfathomable to me that you can have an idea if what production code to write, but no idea what test to write. Certainly you can "expect this page has a button" if you are about to add a button and write the test first. Or "expect this method to add a record to the database", etc. Certainly you are about to write code that does something, so just write a test to expect that thing to happen.

Especially in the case where someone is changing their mind all the time, you are dead in the water without tests to support your changes. You need to manually re-test everything every time they change their mind.

Testing after the fact is not that useful for catching regressions because it is unlikely enough tests will be written due to pressure to release. Also a much greater percentage of the tests written after the production code tend to be vacuous tests.

Re:Coverage is not strongly correlated with test suite effec... by brabel on Tuesday September 28, 2021 @02:40AM (#28679936)
> the result is usually an untested, and usually nearly untestable

No way... if you ever wrote any tests, you can easily know how to write code that will be testable even if you do it later. Doing it before is just going to be a big waste of time if you throw the code away later, which happens a lot when you need to experiment with things before actually choosing what works best. Yes, you can do that with TDD as well but it will absolutely slow you down. If you end up not writing tests later, when the implementation has been chosen, it's just because you don't care about the code quality... I find it hard to believe TDD will fix you in that case.

Re:Coverage is not strongly correlated with test suite effec... by berkes on Tuesday September 28, 2021 @03:10AM (#28680116)
I've always experienced that writing the actual code is but a minor part of building features (or fixing bugs). Especially when prototyping, which most often is merely ducttaping libs together.

Everyone remembers a bugfix of one line, that took hours, or days to find.

I'd stringly encourage you to check your commitlogs. You'll probably find you commit at most hundreds of lines a day, quite probably, as is my case too, av era aging under ten lines a day.

Please stop thinking that writing code which does not make it into commits, is waste. It is learning. And tests are by far the best place to learn about your, and Libs' code, APIs and behaviour.

Re:Coverage is not strongly correlated with test suite effec... by regularfry on Tuesday September 28, 2021 @03:24AM (#28680185)
Swings and roundabouts: what you lose on writing tests you gain on not bothering to implement things you don't need. If the test passes, you stop.

Without the tests to guide you it's very easy to waste time over-engineering, even if what you're building is well-structured.

Re:Coverage is not strongly correlated with test suite effec... by brabel on Tuesday September 28, 2021 @07:39AM (#28681612)
I often write code without implementing anything, just the "surface API". That's when I find whether things will work or not. Tests are a hindrance to that. Once I figure out the design, then I will test all that I think is important.

> what you lose on writing tests you gain on not bothering to implement things you don't need.

I am not sure where to start... I've written so much code, applications, libraries, algorithms... and I am pretty sure the opposite of what you say is true: with TDD, I would've spent hours trying to get something working that later I would find, by exploration, that I didn't need at all.

Re:Coverage is not strongly correlated with test suite effec... by regularfry on Tuesday September 28, 2021 @11:12AM (#28683748)
> I often write code without implementing anything, just the "surface API". That's when I find whether things will work or not.

There's an entire school of TDD which works this way. That's not incompatible at all.

> with TDD, I would've spent hours trying to get something working that later I would find, by exploration, that I didn't need at all.

If you're writing tests for something you don't need, the problem isn't with TDD as I understand it.

Re:Coverage is not strongly correlated with test suite effec... by 4140tm on Tuesday September 28, 2021 @02:31AM (#28679893)
You don't need precise specs to practice TDD. If you have idea of the code you need to write, you can write the test for it beforehand - no matter how often someone might change their mind about the app.

Doing so would actually make your life a lot easier when it's time to alter functionality, because now you have well tested and testable code. Code that is written to be tested is usually a lot easier to reason about, to change and extend.

If you do it in concise manner and test behavior rather than implementation, what you previously thought of overhead will actually speed you up.

Re:Coverage is not strongly correlated with test suite effec... by berkes on Tuesday September 28, 2021 @03:12AM (#28680126)
Tests are also meant to be continously refactored, though.
Re:Coverage is not strongly correlated with test suite effec... by porker on Tuesday September 28, 2021 @01:54AM (#28679751)
This is my experience too (business/customer facing CRUD systems). Perhaps we don't spend long enough thinking about the problem before we start or planning enough, but most software I write starts with a business problem that needs a solution and data, inputs and processes will evolve as it's written. Normally because what comes out in business analysis and stakeholder interviews is close to but not quite what is wanted, but also because of user feedback from user testing (sometimes of a mockup, sometimes prototype software) which is so valuable but now the inputs change again.

I enjoy writing tests when developing libraries where the inputs and outputs are well defined. It's relaxing to do this kind of programming.

Re:Coverage is not strongly correlated with test suite effec... by jbergens on Wednesday September 29, 2021 @02:47AM (#28690912)
For prototyping TDD may be waste. If you are just trying different ux and different logic it may not help to test. For almost everything else I think it helps.
Re:Coverage is not strongly correlated with test suite effec... by taeric on Monday September 27, 2021 @08:38PM (#28677953)
I was going to be worried if mutation testing wasn't mentioned here. Is a great way to test the effectiveness of your tests at catching the common mistakes people make in code.

That is, a mutation suit doesn't test your code, per se. It tests your test suit.

Re:Coverage is not strongly correlated with test suite effec... by sverhagen on Monday September 27, 2021 @08:49PM (#28678019)
I've worked a lot on Java, and while there is great mutation testing tools (well, PIT specifically), I find them hard to "scale" practically, i.e. to run them in an automated fashion "every time". You either get long running builds with them in it, or you have to deal with the logistics of moving caches around (so that the mutation testing can be incremental). And as just another tool that a developer MAY use if they so choose to, it's... well... nice... but no longer front of mind, for most people on the team. I'm very interested if you're experiences have been different?
Re:Coverage is not strongly correlated with test suite effec... by taeric on Monday September 27, 2021 @09:42PM (#28678343)
No, I found the same. Team disabled them on every release build, as they felt it was too slow. I may bring them back in the release servers, but it is honestly trying to argue for these when nobody else cares.
Re:Coverage is not strongly correlated with test suite effec... by masklinn on Tuesday September 28, 2021 @01:06AM (#28679546)
It needs someone who cares which us a thankless job but it’s useful as smote tests run nightly / weekly.

It tends to be better for broad concerns which trigger somewhat rarely, tend not to be otherwise caught, but can be quite concerning. Crater is a good example of that, it’s infeasible to build & test every public crate on every CI check, but it’s a useful sanity check when release arrives or when messing with the more subtle aspects of the compiler.

Re:Coverage is not strongly correlated with test suite effec... by SilverRed on Tuesday September 28, 2021 @12:01AM (#28679192)
My favorite peer review trick is reverting the code change and showing that the test still passes. No tests will cover everything but its not hard to apply some simple tricks that raise the bar of actual coverage.
Re:Coverage is not strongly correlated with test suite effec... by SV_BubbleTime on Monday September 27, 2021 @11:29PM (#28678995)
> Writing quality tests for existing code is much harder than writing tests for code that doesn't exist

I’ve been trying this and the biggest place it’s helped me is parsing. Working out all the bad formats, types, unexpected stuff before I write the code was worth it.

It’s still probably low quality compared to what it will eventually be, but a pretty good way to start.

Re:Coverage is not strongly correlated with test suite effec... by thaumasiotes on Monday September 27, 2021 @08:43PM (#28677979)
> But once it is "covered" the coverage report is useless.

Well, yes, any report that is guaranteed to say the same thing every time is useless.