News for hackers. Stuff that matters.
Re:Fear makes you a worse programmer by bluefirebrand on Monday April 08, 2024 @05:28PM (#39973918)
> Even with good design, w/o tests you will fear change and so the code will rot. With good tests, there’s no fear, so you’ll clean the code

Has anyone ever actually found this to be true?

I work in a place with 99% test coverage requirements and it's honestly still a super brittle system that everyone is afraid to make big changes to

In my experience, automated tests don't promote quality engineering, they just calcify whatever quality exists

And before you get into it, TDD is not a panacea for this problem

Re:Fear makes you a worse programmer by Izkata on Monday April 08, 2024 @05:46PM (#39974061)
Test-a-single-function style, what most people default to when they hear "unit test", is probably the thing that most leads into that situation.

I've been occasionally pushing coworkers to restructure code and tests into a higher level style, defining an API boundary and calling it from the rest of the code (as if you're writing a library except it's inside your codebase instead of installed), then writing tests to that API. That does make for some easily refactorable code where the tests don't need to change at all.

That pseudo-library style is kind of what "unit test" originally meant: a business unit, not a code unit. Examples simplified it too much and people copied the style instead of the substance, and the original meaning was lost.

For another example, some of our recent projects have been on updating daemons, those have also been really good candidates for this since there are clear entry and exit points.

Re:Fear makes you a worse programmer by maw on Monday April 08, 2024 @07:07PM (#39974621)
> That pseudo-library style is kind of what "unit test" originally meant: a business unit, not a code unit. Examples simplified it too much and people copied the style instead of the substance, and the original meaning was lost.

Very interesting. It sounds a bit like what happened with the term "Hungarian notation".

Re:Fear makes you a worse programmer by GuB-42 on Monday April 08, 2024 @07:31PM (#39974766)
But you probably won't get 99% coverage doing that. Some conditions are just too difficult to get from a higher level, and you still need to take them in consideration: hardware-related problems, race conditions, etc...

I don't consider it a problem, but it is to those who want to see big numbers.

Re:Fear makes you a worse programmer by yellowapple on Monday April 08, 2024 @09:45PM (#39975574)
I feel like if you're unable to get 99% coverage with that then there's probably some dead code that needs pruned.

As for hardware-related problems and race conditions, testing at a higher level of abstraction seems like it'd help more than it'd hurt - in the former case ensuring graceful handling as part of the tests, and in the latter case making the race conditions more likely to hit (and hopefully fix).

Re:Fear makes you a worse programmer by lll-o-lll on Tuesday April 09, 2024 @12:35AM (#39976220)
I see the OP as talking about “integration” style tests (say testing against an api) and not being able to capture all of the OS and similar edge case errors in the test. I do think this is a hard problem, particularly if you don’t design for it from the outset.

It’s a bit of the old classic “reading from the file never returns an ‘access denied’ error, until it does”. There’s ways and means, but let’s not pretend this is simple.

Re:Fear makes you a worse programmer by jval43 on Tuesday April 09, 2024 @01:56AM (#39976513)
You don't need 99% coverage. You just need enough coverage to be able to refactor the thing without breaking it.
Re:Fear makes you a worse programmer by shrimp_emoji on Tuesday April 09, 2024 @09:05AM (#39978996)
I bet the reason they aim for 99% coverage is because they're using Python on JavaScript for some complex software, which means they've roped themselves into being their own compiler. Without all that test coverage, all the bugs are surprise runtime bugs, and the tests take the "surprise" part out.
Re:Fear makes you a worse programmer by mythhabit on Tuesday April 09, 2024 @03:00AM (#39976810)
I believe that the original definition, way back before most of us were born, was a test of the smallest possible unit, i.e a single function ect. At least according to Wikipedia, in 1969, it was defined at unit tests, component tests and integration tests.

In todays development world, the unittest is primarily a developer tool to help speed up development. One should not be afraid to delete unittests if they are doing refactorings. But the long term value is always in the integration tests.

Re:Fear makes you a worse programmer by bazoom42 on Tuesday April 09, 2024 @05:47AM (#39977781)
Not the smallest unit, just an internal unit or component smaller than the whole of the system.

The important thing is not the size of the code affected by the test, the important thing is that a test should verify a single requirement isolated from other requirements.

I believe the original distinction between unit test and integration test was that integration test was for when parts developed in isolation by seperate teams was finally integrated. But this tended to be a disaster, which is why continous integration was adopted.

Re:Fear makes you a worse programmer by mythhabit on Tuesday April 09, 2024 @08:06AM (#39978540)
Just going of the Wikipedia page on unit testing, in the 1956 paper on programming for the SAGE project, refers to testing as: Parameter testing, assembly testing, shake down testing. In the context, parameter testing is testing of individual sub routines (functions/methods). From there the term unit test was used in the Mercury System as unit testing of "individual programs outside and inside the system". I suppose here, the ethos is the same.

Obviously, when reading the rest of the papers, they are clear on the fact, that it is the specifications that the programmer have developed for, that should be tested. That was at the time, synonyms with individual sub routines, and as such it was both the smallest unit and a logical testable unit at the same time. Since then, we've come a long way with programming smaller units with better composability.

I'm not sure I agree that the original meaning of a unit test was lost. Perhaps, only one part of the definition was carried over to modern development practices. In any case, I always stand by the fact that long term value is in tests of the "API". Everything else is implementation details.

Re:Fear makes you a worse programmer by vineyardmike on Monday April 08, 2024 @06:17PM (#39974317)
Just to re-iterate another commenter, I had a positive experience before.

When I was at BigCompanyA, we had 95% coverage requirements as a management fiat. The company generally was very management-decree-driven. People would unit test individual methods and helper functions and each “unit” of code. If you wanted to change an API, you had to dig through a sea of broken tests because each little bit of code was individually tested. We literally had unit tests that validated one line methods for string concatenation of a prefix (in Java). Management said add tests with each commit so everyone added tests without thinking about what is valuable.

At BigCompanyB, our testing was engineering driven, not some management metric being tracked. The goal was to test “public interfaces” and ensure that these tests captured all the helper methods along the way. This helped catch dead or extraneous code if you couldn’t write a test to exercise a particular path. Changing an API didn’t require changing a bunch of silly tests. We still had equal >95% coverage, and it was very useful.

Basically you need to actually think about what makes sense to test and write logical tests.

Re:Fear makes you a worse programmer by nickjj on Monday April 08, 2024 @05:38PM (#39974004)
> Has anyone ever actually found this to be true?

It depends.

I've written a number of apps where I'm fearless with making changes because I trust the tests I wrote. One of the apps has been running for 7 years and it's had a lot of big updates, refactors, etc.. I'm breaking every rule there is on jinxing things but there hasn't been a single bug introduced due to those updates and there's ~85% coverage. It's only a ~3k line Flask app though, but it does get deployed straight to production with no staging environment and gets hundreds of requests per day. It writes to a DB, Redis, interacts with multiple 3rd party APIs and sends out webhooks so it does quite a few "external" things.

I've never been a fan of TDD and personally I think tests that really hit your DB, Redis, etc. help a lot more than mocked out unit tests or a billion unit tests and nothing else. I tend to write more tests that really test things together. Not full blown Selenium style tests, but I do really write to the DB and other data stores in tests and often use a framework's built-in test client for making HTTP calls. Everything can still be really fast too (~100 tests in 2 seconds is my usual rough guide for having an assortment of "real" tests with Flask).

Re:Fear makes you a worse programmer by solraph on Monday April 08, 2024 @05:53PM (#39974132)
> I've never been a fan of TDD and personally I think tests that really hit your DB, Redis, etc. help a lot more than mocked out unit tests or a billion unit tests and nothing else... ...but I do really write to the DB and other data stores in tests

I've come to the conclusion that the idea that "unit tests" should test functions/objects in isolation with completely mocked dependencies is based more on the slow speed of those dependencies in the past than what actually makes for good tests. Now that we have faster computers and storage devices, and easy/fast store creation, we should move past this.

Obviously, this is very dependant (no pun intended) on the dependency in question, but as a minimum, anything with SQL should have a test that hits a real SQL DB (PG in docker for example) at some point.

Re:Fear makes you a worse programmer by arp242 on Monday April 08, 2024 @07:17PM (#39974689)
It's not even about faster computers and storage, but just about better test techniques, and better written tests. You don't actually need very fast hardware for fast tests; you just need to spend some time on it. And roughly know what you're doing.

External dependencies (such as PostgreSQL, Redis, what-have-you) are a pain though. I feel pretty strongly that just a single command ("go test", "cargo test", "npm test", etc.) should run all the tests on all platforms, reliably, with a minimal of fuss and messing about. Things like docker-compose or whatnot quality as "a fuss and messing about".

Re:Fear makes you a worse programmer by solraph on Tuesday April 09, 2024 @06:40AM (#39978035)
They can be a bit of a pain, but if you have a lot of logic tied up in your external dependencies (SQL usually bring main offender here) it's not possible to properly test without using said dependency.

That said, in the current job, we have a dedicated Postgres container that comes up with with a just file command. The setup, test, and teardown of schema all happen within the standard test system of the tested platform (pytest or go test specifically).

Re:Fear makes you a worse programmer by jasonpeacock on Monday April 08, 2024 @05:43PM (#39974045)
I have, and it was amazing.

You could quickly make the changes you want not worry about what else breaks, then run the tests. The tests that you expect to fail would fail, and you fix them. Then you find other tests that you didn't expect also failed; you'd review the impact of your changes on those parts of the system and make appropriate changes until the tests were happy.

Because the testing was so thorough (and quality), you had high confidence that nothing unexpected was broken.

It was "move fast & break [tests], then fix them and deploy" :)

Re:Fear makes you a worse programmer by furstenheim on Monday April 08, 2024 @05:49PM (#39974085)
Take a look at hexagon pattern from Spotify. Once you start testing the user contract of your services against real databases (testcontainners is a good option), then you can change all the internals and be sure that the externals will work.

https://engineering.atspotify.com/2018/01/testing-of-microse...

Re:Fear makes you a worse programmer by solraph on Monday April 08, 2024 @05:47PM (#39974073)
> > Even with good design, w/o tests you will fear change and so the code will rot. With good tests, there’s no fear, so you’ll clean the code

> Has anyone ever actually found this to be true?

Yes, but you have to have the right kind of test coverage, and that's the tricky part.

Yesterday, I refactored a bunch of functions that changed a bunch of unit tests. However, since we also have integration/system tests on that code, I'm confident that I haven't broken the code as a whole. Without those system tests, I would not have confidence that the change would be successful, and probably would not have refactored.

In another codebase that hadn't been touched for a year, as part of a feature change I refactored an SQL statement to what I thought was a more optimal design and immediately broke a bunch of tests. Based on that, I was able to understand the original intent of the SQL, and updated it in line with the feature change. I added test scenarios for the new feature, but left the existing scenarios as is.

Without those tests, would have broken the system in a subtle way.

Re:Fear makes you a worse programmer by dimal on Monday April 08, 2024 @07:24PM (#39974724)
Only for limited areas of a server codebase on a small team who understood what tests were meant to accomplish and wrote excellent tests. Basically, about a year and a half out of a twenty five year career. I’ve never seen them useful on front end code except in extremely limited circumstances.

In most other cases, we had tests that only covered the happy path and did little more than slow down builds and make refactoring more difficult. In other words, they made things worse. E2E tests, in particular, are the Afghanistan of web development. I’ve seen more time wasted getting useless tests green than I’ve seen wasted on any other programming exercise.

Re:Fear makes you a worse programmer by vidugavia on Monday April 08, 2024 @05:39PM (#39974013)
99% coverage does not mean you have good tests. Yes, I have experienced that good tests, combined with a good understanding which parts are critical and which are not, create a great comfort about introducing changes, whether it's Friday afternoon or Monday morning.
Re:Fear makes you a worse programmer by sealeck on Monday April 08, 2024 @05:45PM (#39974053)
I think if you evolve your testing system to feed random numbers into your program and ensure that various correctness properties hold for all these random inputs you are likely to get a pretty robust program. You can make this slightly efficient by using a fuzzer (essentially uses feedback from running previous inputs to see how running new inputs works). See for example https://propertesting.com/ or https://en.wikipedia.org/wiki/American_Fuzzy_Lop_(software)

Of course if you need a higher standard of guarantee you can try model checking or other formal verification techniques.

Re:Fear makes you a worse programmer by lukev on Monday April 08, 2024 @05:41PM (#39974031)
This is why I'm neutral on the value of unit tests. Sometimes they are helpful, other times less so.

I am always in favor of tests around the boundaries between system components (regardless of what you call them; integration tests, API tests, etc.) The more robust the better.

If I'm working on a system with robust (ideally generative) integration tests covering the interface of a component, I feel I have near-complete freedom to rewrite any aspect of the component I want with high confidence.

Re:Fear makes you a worse programmer by adonovan on Monday April 08, 2024 @05:53PM (#39974125)
Unit tests are good when there are clear and durable units, but what many people seem to mean by a unit test is one that exercises the internal functions of a package, or the API of an internal package that evolves as fast as your product.

Stable interfaces that the user cares about (e.g. CLI, public API, RPCs) make for the highest-value tests.

Re:Fear makes you a worse programmer by nomel on Monday April 08, 2024 @06:03PM (#39974205)
> ideally generative

How have you seen this done well? In my experience, this usually ends up being some hello-world type simplicity that doesn't really represent the real world use corner cases.

Re:Fear makes you a worse programmer by lukev on Monday April 08, 2024 @06:20PM (#39974338)
I have, on multiple occasions. Unfortunately mostly on contracting gigs so I can't cite specifics but I've worked on a major ecommerce platform's api, and two separate databases using this approach.

For public examples, I'd point you to (e.g.) Jepsen.

I'm not going to deny it requires a pretty high level of time/money to implement. But done well it's super powerful.

Re:Fear makes you a worse programmer by randomdata on Monday April 08, 2024 @11:48PM (#39976055)
> I am always in favor of tests around the boundaries between system components (regardless of what you call them; integration tests, API tests, etc.)

Funnily enough, "unit tests" is what this was called originally. The boundary is what "unit" referred to. In my experience, these days most people just call them "tests". That they happen at the boundary is implied as by this point it is generally agreed upon that testing anything else is a waste of time at best and sometimes even detrimental.

Which is why nobody can settle on what "unit", "integration", etc. mean in the modern age. There is nothing related to testing in need of additional qualification to communicate to other developers what isn't already communicated with "tests" alone.

Re:Fear makes you a worse programmer by zelphirkalt on Monday April 08, 2024 @06:24PM (#39974366)
At some point you start to fear changing too much, because of all the tests you will have to rewrite, which can be tedious, especially, when the code mutates things all over the place and you have to mock 5 things per unit test.
Re:Fear makes you a worse programmer by randomdata on Tuesday April 09, 2024 @02:11AM (#39976574)
> because of all the tests you will have to rewrite

If you have to rewrite tests that means you've changed the user experience in ways that are not backwards compatible.

Which is sometimes valid, but not exactly what is being talked about here. The discussion here is more about changing the code in ways that makes the code better, but still delivers the same user experience – possibly with new features added, but not where anything is taken away.

Re:Fear makes you a worse programmer by Jensson on Tuesday April 09, 2024 @06:06AM (#39977872)
> If you have to rewrite tests that means you've changed the user experience in ways that are not backwards compatible.

That is only true for integration tests. You can rewrite a set of local functions without changing user behavior, and then you need to change tests, such refactors becomes a pain when you have too many unit tests but are really easy when you have many integration tests.

Re:Fear makes you a worse programmer by randomdata on Tuesday April 09, 2024 @09:19AM (#39979148)
I don't get it. Beck was quite explicit when coining the term "unit tests" that the unit refers to the set of functionality found at the integration point – which seems to be what you refer to as "integration tests", and what everyone else these days call "tests". It's all the same.

If changes to "local functions" calls for tests to be rewritten, that means you've exposed "local functionality", even if by accident, to the outside user. Which means it is not actually local functionality, but something you have exported and are committed to maintaining. Rewriting the tests is not the correct course of action. You need to fix the code that you just broke as the functional contract was violated. With any luck that hard lesson will teach you to be more careful next time.

Re:Fear makes you a worse programmer by zelphirkalt on Tuesday April 09, 2024 @05:28PM (#39984476)
Unless of course, someone before you has written lots of tests for all the functions separately, regardless of whether they represent functionality at the integration point and there is a culture of not wanting to delete tests, because of coverage reasons. Then what you have are suddenly broken tests, even if you change only procedures at a lower level.
Re:Fear makes you a worse programmer by randomdata on Tuesday April 09, 2024 @05:36PM (#39984548)
Deleting the tests will not impact coverage as the tests at the integration points will necessarily already cover any local functions. That is unless said local functions are unused, but in that case you would remove the unused local functions anyway, still not impacting coverage.

But you are right that deleting the tests isn't an option as it will break the contract that was entered into with the users of the code. Of course, you can't modify the tests for the same reason, so...

Re:Fear makes you a worse programmer by agumonkey on Monday April 08, 2024 @06:17PM (#39974319)
You're both right. TDD is no panacea, and having no test is being blind. There's a black art into estimating what angles to test and how deep to avoid calcification.
Re:Fear makes you a worse programmer by randomdata on Tuesday April 09, 2024 @03:15AM (#39976899)
> There's a black art into estimating what angles to test and how deep to avoid calcification.

There may be a black art to recognizing what you overlooked, but otherwise it is pretty straightforward. Tests are your documentation. The angles needed to be covered are exactly what the user needs to know to use your software – how to use it and, when using it, what happens both in the expected case and when failure occurs (especially what happens when failure occurs!).

Calcification is of little concern as the inverse is breaking changes, and the user does not want to deal with your breaking changes. You can put your mind at ease knowing that once you commit to a documented feature, it should remain there until the end of time.

Re:Fear makes you a worse programmer by bluefirebrand on Tuesday April 09, 2024 @08:55AM (#39978913)
> Calcification is of little concern as the inverse is breaking changes, and the user does not want to deal with your breaking changes

Please explain this to Product Managers. :)

Re:Fear makes you a worse programmer by hinkley on Monday April 08, 2024 @06:09PM (#39974249)
I’ve experienced it.

But only at companies that don’t give a shit about test coverage %. The percent kills the purpose of the tests, which is certainty.

Re:Fear makes you a worse programmer by elliottkember on Monday April 08, 2024 @06:25PM (#39974369)
I think you missed a word. "Good" tests is a crucial part of the statement. It's totally possible to write bad tests, the same way it's possible to write bad code.
Re:Fear makes you a worse programmer by barfard on Monday April 08, 2024 @05:56PM (#39974149)
could still get 99% test coverage with useless tests
Re:Fear makes you a worse programmer by Aeolun on Monday April 08, 2024 @08:42PM (#39975211)
> Has anyone ever actually found this to be true?

We have a 300kloc monster, and I find that going up from 0 to 60% test coverage has given me appreciably more confidence that the system still works after any change.

Sure, the test code is almost twice that size, and breaks almost as often as the code, but my confidence in the system itself is definitely higher.

To prevent people from writing tests that need too many mocks we now have explicit dependency injection. So much easier to reason about stuff, and prevent’s people from not testing the important bits.

Re:Fear makes you a worse programmer by bazoom42 on Tuesday April 09, 2024 @02:06AM (#39976560)
It is easy to create bad tests. I have seen an over-reliance on mocking giving high code coverage but still quite useless tests.
Re:Fear makes you a worse programmer by onion2k on Tuesday April 09, 2024 @02:15AM (#39976589)
I work in a place with 99% test coverage requirements and it's honestly still a super brittle system that everyone is afraid to make big changes to.

Obviously tests are going to break when you change the code's APIs and functionality. That's expected, and does nothing to boost your confidence in the part of the code you're working in. The point of tests is to improve your confidence that you're not doing things that have an unexpected impact somewhere else (hence the Bob Martin quote in the article about tests being useful even if you have good design).

Tests aren't an alternative to thinking. You still have to consider what changes you're making, and what tests should break as a result. That's just part of your code change. When tests that shouldn't have broken start failing that's when they show their value.

Re:Fear makes you a worse programmer by PaulStatezny on Tuesday April 09, 2024 @02:19AM (#39976613)
You need mostly integration tests, not unit tests.

Unit tests which dependency-inject mocks of other parts of your codebase are 99% worthless.

Source: Spent many years writing the latter and they caught close to 0 bugs. Moved to Elixir ecosystem where integration tests are the norm, and they catch bugs on a regular basis.

Re:Fear makes you a worse programmer by pjerem on Tuesday April 09, 2024 @03:42AM (#39977022)
In my experience, Unit tests are interesting when they are inlined with the function they test, i.e. in the same file (like python doctests). Of course it’s harder to set the testing environment but that may in fact be encouraging you to write stateless code so there is that.
Re:Fear makes you a worse programmer by rramadass on Tuesday April 09, 2024 @03:19AM (#39976915)
Bingo!

I never understood the fascination with Unit Tests. For Testing to be useful the code being tested should have a certain degree of Complexity (algorithmic/behavioural/state-transition/etc.). But what i see from most unit tests is mere "busy work" as if mock-testing a trivial class/api/etc. would somehow make your code better. A similar criticism is also applicable to TDD based programming.

Re:Fear makes you a worse programmer by hitchstory on Tuesday April 09, 2024 @04:33AM (#39977313)
The "busy work" of low level unit tests is perfectly valid, but TDD still works extremely well with high level integration tests so I don't think a similar criticism necessarily applies.
Re:Fear makes you a worse programmer by randomdata on Tuesday April 09, 2024 @02:16PM (#39982378)
> I never understood the fascination with Unit Tests.

Me neither. The legacy forms of testing that "unit" was trying to differentiate from have died out. All tests are unit tests today, which is why most developers just say "tests". "Unit" is already implied and adds no additional information not already understood by "test" alone.

Same goes for integration tests. Integration tests, as it was historically defined, died out. It seems anyone still trying to use the term today is using it to also just mean "testing", or what was historically known as unit testing. Like unit, a pointless qualification.

Re:Fear makes you a worse programmer by lijok on Tuesday April 09, 2024 @03:59AM (#39977105)
> With good tests

> I work in a place with 99% test coverage requirements

Sounds like your problem is bad tests. In fact, I guarantee they’re bad tests because that’s exactly what coverage requirements lead to

Re:Fear makes you a worse programmer by okeuro49 on Tuesday April 09, 2024 @05:19AM (#39977642)
> Has anyone ever actually found this to be true?

Yes, always.