News for hackers. Stuff that matters.
Re:C++ Dev Ecosystem 2021 by NavinF on Monday July 19, 2021 @06:36PM (#27888079)
Is it really? Most unit tests I've encountered are garbage. They only test for bugs that are incredibly unlikely to happen.

Integration tests and fuzz testing is where it's at. Anyone can add or improve an integration test within an hour and find several bugs that real users would encounter.

Re:C++ Dev Ecosystem 2021 by titzer on Monday July 19, 2021 @07:21PM (#27888400)
Strong disagree. Effective unit tests accelerate development by allowing fast turnaround and refactoring with confidence. There are bad unit tests, of course, that are highly coupled with the code that they test.

I have written tens of thousands of unit tests in my career, and by far the most useful ones are the ones that point you to the exact subsystem in which a bug you've just introduced during refactoring/re-implementing lies. Bad unit tests are brittle "change detectors" and make you constantly adjust them for no real reason. I find mocks and interaction-based tests (rather than simple input/output tests) are this way. I flat-out do not use mocks for this very reason. They are intricate, involved, and take huge amounts of time to load context when they fail. Don't write such tests.

If your unit tests are garbage, it's because they were not thought out very much and commit sins.

Re:C++ Dev Ecosystem 2021 by bcrosby95 on Monday July 19, 2021 @07:46PM (#27888573)
> I flat-out do not use mocks for this very reason. They are intricate, involved, and take huge amounts of time to load context when they fail. Don't write such tests.

This is why I prefer integration tests. Most of my projects interact heavily with a database. Making it an integration test unless I mock it out. Which I don't do, because in my experience, doing so tends to hide lots of bugs.

Re:C++ Dev Ecosystem 2021 by titzer on Monday July 19, 2021 @08:36PM (#27888921)
> This is why I prefer integration tests.

IMO integration tests are great for making sure that your system actually functions with all its parts together. Nothing but an integration test can do that. But unit tests serve a different role: reducing the scope of the system under test so that you can pinpoint exactly what part failed when it does fail. They help you iterate quickly in refactoring, and when something goes wrong in production (or a bug report from the field), they narrow the cone of uncertainty by eliminating from consideration the parts of the system that are working correctly. In my experience, this rapidly cuts down on the amount of wild goose chases one goes on and overall makes a team more efficient. Then once a bug in the wild is found, the system responsible for the failure gets new regression tests and unit tests. Sometimes, I wrote unit tests for units I am suspicious of as I am debugging, and even if those tests don't find a bug, I might keep them.

I worked for 6+ years on V8 and the parts of the system that were the least unit tested seemed to be home to far more production issues than others.

It also helps a lot to keep your unit tests running super fast. Under 1 second is ideal.

Re:C++ Dev Ecosystem 2021 by thom on Monday July 19, 2021 @08:38PM (#27888939)
I think I agree with this, although I also think a lot of people would still call this integration testing. Basically a lot of code units are implementation details and you want to be free to refactor them fearlessly without having to write double the code each time tightly unit testing each part. Much better to narrow your system to a few well defined boundaries that deliver some clear value. The code behind that interface can be as deep as you like but you don’t really need to push tests down into the depths.

As I say, some people will still think of this as integration testing just because there’s a lot of moving parts working together. But often you’re just testing fairly pure code with no external or IO dependencies so I’m happy thinking of these as unit tests still. Where those types of dependencies exist I do still mock though.

Re:C++ Dev Ecosystem 2021 by wutbrodo on Tuesday July 20, 2021 @01:23AM (#27890351)
> I find mocks and interaction-based tests (rather than simple input/output tests) are this way

It's fascinating, I agree with the rest of your comment wholeheartedly but am a heavy mock user. Much of the code I interact with has complex logic with well-defined and discrete interacting objects. Without mocks, the combinatorial explosion of input/output behavior means tests are extremely difficult to write, or more likely, are written incompletely, without properly enumerating the invariants expected of the interfaces under test.

I'm currently working on a complex system right that's safety-critical but also SOTA, and full of domain experts who are decisively not engineering experts (among other things, engineering expertise is part of what I bring to the table). Well-defined invariants and narrow unit tests that enforce them are a _god-send_, and I'm literally not sure we could function without them: as we're constantly refactoring the system to work in different ways, automated testing of invariants enables a whole universe of experimental changes that would otherwise carry a great risk of introducing bugs into other parts of the system. Our system is far too complex for integration tests to cover a fraction of the completeness that unit testing enables (though we also have simulation/integration tests, of course).

Re:C++ Dev Ecosystem 2021 by Rexxar on Monday July 19, 2021 @11:35PM (#27889809)
You have just described integration test, not unit test. So maybe you are not disagreeing at all ?
Re:C++ Dev Ecosystem 2021 by Rd6n6 on Monday July 19, 2021 @07:43PM (#27888553)
Except for unusually complex logic, almost every bug I’ve ever seen has been configuration or integration related. I do unit tests in safety critical software, but if I actually need to catch a bug I usually worry more about integration or end to end tests
Re:C++ Dev Ecosystem 2021 by Jtsummers on Monday July 19, 2021 @06:45PM (#27888156)
> They only test for bugs that are incredibly unlikely to happen.

Well, that might be the problem. Testing "for bugs" isn't a terribly helpful activity. Testing for behavior is much more helpful, and unit tests can definitely be applied to that aspect of the system. Of course, it requires you to properly decompose the system into subsystems (modules) with loose coupling that can be individually (or in small sets) loaded into test harnesses for testing.

Re:C++ Dev Ecosystem 2021 by ddlutz on Monday July 19, 2021 @06:57PM (#27888249)
Maybe you are not working on the right projects or with the right people, because good unit tests are extremely valuable. At a minimum, ensuring that making a change to some existing code doesn't break existing behavior is valuable.
Re:C++ Dev Ecosystem 2021 by ternaryoperator on Monday July 19, 2021 @06:41PM (#27888121)
All that could be true and it still would not suggest that no unit tests at all is a valid choice.