And yet, more or less nobody does.
So either it is next to impossible to begin doing it (seems like a bizarre conclusion), or, perhaps more likely, nobody wants to do it, and the few dev teams that do manage to do this have not managed to turn that into a competitive advantage. Which makes the value of TDD rather questionable based on simple evidence.
To explain this observed behaviour that TDD is clearly not a competitive advantage[2], I can name a million pet theories. But without going into any of those, the sheer fact that it's __this__ rare in practice says a lot, no?
[1] I mostly go to java related ones, maybe it's less well known amongst other communities.
[2] What other explanation is there? Clearly not 'ah, but, TDD is brand new and you have to give it some time for teams to get familiar with it, and for the concept to percolate through, maybe wait for tooling support to catch up' - TDD's quite an old concept!
When you're doing it, you feel great, or at least you convince yourself that you feel great because you know you're supposed to. But ultimately it's hard to keep it up. It gets tedious, it requires a lot of willpower, and the benefits are a little too abstract or far removed.
At the end of the day, most people will be just as healthy with a more moderate approach. Just eat some vegetables, not too much sugar, and keep a reasonable limit on total calories. In the programming analogy - just write some tests for important functionality, and be careful that the tests you write are actually working, but you don't need to be fanatical about writing the tests before the functionality or having 100% coverage.
And then as with diet, it's also possible that if you're aiming for the moderate approach you end up slipping too much to the other extreme and writing almost no tests. For some people a structured diet is necessary to keep them on track, and I suspect for some people TDD can be a more effective tool than it is for others if their default coding style tends toward more sloppy/unstructured.
One of the biggest problems with TDD is that it kind of relies on having clearly defined specifications. I don't know about you or other people here, but I've worked in a lot of places (many even called themselves Agile) where the work was not properly scoped at all. If you start doing TDD and the scope isn't clear, the goal posts keep on moving and things just perpetually take longer.
I think it's all about cost in the end. It's cheaper for companies to ship buggy code and then iteratively patch the bugs. Unless they're massive showstopper bugs (which normal tests should be catching anyway), it probably still comparatively works out cheaper to fix bugs as you find them.
That's very true. I think it's an easier sell when you have a relatively stable set of specifications (accounting software core logic) where the rate of change is low but the cost of regressions is high. But I think you can tackle the same problem space with good unit tests instead of enforcing a test-first mindset.
In situations where work is vague, spending more upfront time thinking about architecture is much more useful (design docs?) imo. Having a set of units for a crappy, entangled system is pretty costly.
I can see this argument for large-scoped scenario based integration tests, definitely. But TDD unit testing operates on a much smaller scale, that of a single patch, or a Jira ticket.
It's an incredibly easy sale. The whole basis of TDD is that it's an approach that makes your development efforts faster, with higher quality. A million graphics of the amount of time that development spends fixing errors are what sold TDD to the masses.
The theory of TDD is exactly what sells to the suits and the money counters.
The theory doesn't mesh with reality, though, and it's that engagement with the enemy (reality) where TDD falls down.
As an aside, in my own career I've seldom been able to incorporate TDD because each project has been novel enough that trying to define tests up front was just not possible. Yes, if I was implementing the re-invent the wheel "sum two numbers" type example, it's trivial. But most of the time it's a vague API for a vague need on an uncertain technical foundation, and until the clay has taken form we really weren't sure what we were dealing with.
Short answer: TDD is hard
Long answer: It takes a very experienced engineer to start and layout a piece of software so that it's easy to write and maintain tests. IMO, this is one of the advantages of server-side rendered web applications; it's much easier to write a test that verifies HTML than a test that verifies the state of a Windows / Mac application; or even the state of the DOM in a rich in-browser application.
But, continuing on that train of thought: It's much easier, as a developer, to write a test that tests an API, (either programmatic API or web service) as a unit than a giant monolith.
I did exactly this for a hosting company back when Windows Server 2003 was getting EOL'd. They setup on 2012, migrated the sites over, and then my script would pull from both versions and see how much variation there was. Even then it was mostly just a diff.
Otherwise that's the equivalent of testing implementation details.
1: Stick an entity in the database 2: Try to get the page for the entity without logging in. Verify the error page 3: Log in as someone who doesn't have the entity. Try to get the page for the entity. Verify the error page 4: Log in as someone who can get the entity. Verify some basics of the page contents
Write a test that:
1: Posts the contents of the form without logging in. Verify the error. ... 4: Log in as someone who can post the form. Post the contents of the form. Verify the contents are in the database.
Tests like above meet the higher-level post's definition of a "unit test" and are rather easy to write. You don't even have to be in the same language as application. They're simple enough that developers can write them as part of their workflow, and simple enough that developers can run them to ensure no regressions.
Trying to do something like that with a thick UI application is much, much harder. Possible, but not easy.
It implies to me that the absence of an error is a success. In other words, if you don't find the HTML you're looking for that represents an error, then it's considered a success.
The problem is that at some point it's possible for the page to be redesigned (or the entire site) such that the HTML you're looking for that represents an error is no longer there for a legitimate error.
IOW, the tests are not future proof. But this is the general problem with looking at HTML, it's fragile.
That doesn't mean I don't see the need, I understand the goal, I just don't agree with this approach.
I think if it were me I'd do 1 of 2 things.
1. rely on HTTP codes (403 is a reasonable response for what you've described), or
2. rely on HTTP headers. And I would argue it's better to set them on success and consider it an error anytime the header is missing.
Obviously redesigns don't happen that often so it's not a super large issue, probably not worth rewriting anything but I do think if I were about to implement that functionality I would try and use HTTP codes or HTTP headers.
Implies that you're looking for the contents that are supposed to be returned.
When there's an error, the contents you're looking for won't be there, therefore the test will fail.
Furthermore, HTTP testing libraries typically require you to specify required HTTP status codes.
- "that sounds super fragile, there's only a few circumstances in which I would that"
- "Here's the circumstances in which I'm doing that"
- "Here are the problems with what you're doing, and here are alternatives that don't involve examining the HTML"
- "I'm not examining the HTML!?!? Whatever gave you that impression!"
---
Not worth my time.
The market may be rewarding the TDD skill, it makes sense. But consider that causality makes sense the other way around too. Anyone with the patience, eye for detail, and organizational skills demanded by TDD could be just an above-average programmer.
Personally I like a “don’t forget the tests” approach for larger units of business logic, but I’m against exhaustive micro-test suites that just turn into bespoke, buggy, and useless-at-compile-time type systems.
And I don’t really follow TDD practices myself, so this would say nothing about me.
When the big "TDD is dead" debate hit the scene, I felt that DHH and the detractors won that debate big time - even though I use TDD and consider it to be a valuable tool.
Why would I think this - after all, if I like TDD and often employ it, why would I agree that the people who claim it is "dead" have won?
Probably because the debate had become relatively extreme:
https://blog.cleancoder.com/uncle-bob/2014/05/02/Professiona...
"If I am right… If TDD is as significant to software as hand-washing was to medicine and is instrumental in pulling us back from the brink of that looming catastrophe, then Kent Beck will be hailed a hero, and TDD will carry the full weight of professionalism. After that, those who refuse to practice TDD will be excused from the ranks of professional programmers. It would not surprise me if, one day, TDD had the force of law behind it."
Whoah! And keep in mind, people were acting on this. I read a blog (sorry no cite) from a highly regarded developer who worked as a contractor to restructure software dev teams, and he came flat out and said that if people don't adopt TDD, they won't be employed for much longer wherever he works. I got the feeling that even questioning it would be considered grounds for "no hire".
That mentality is what I think is "dead" - and this is evidenced in the fact that you and I (who do quite a bit of TDD) acknowledge that it is not something we always do and that it isn't always possible or desirable. I got in wicked arguments about this at work, and I'm almost certain I was dropped from consideration for jobs during interviews where I argued the now more moderate position which, ironically, makes me one of TDD's defenders in 2020!.
It's hard to communicate as a developer at times. We're all learning different things at different times and taking things to an extreme can cause bad outcomes for the rest of the team. One of the reasons I try to be accurate during online discussions. "I do TDD 85% of the time" is a better thing to add to the conversation than "I do TDD" since it gives people the idea of nuance in software development methods.
I personally find TDD paradoxical in that it's often used as a way to gather your thoughts, yet the more exploratory the work the more TDD hurts.
Tests are very very useful when wrapped around the core logic and data structures of your application. But that doesn't mean they need to come first.
Curious - did you hold this opinion around 5-10 years ago, and did you voice it to any TDD advocates at the time? My experience was that a statement like this (which I consider immensely reasonable) could kick off really severe arguments, and could get you no-hired in interviews.
There may have been a bit of a Motte-and-Bailey argument going on, where TDD advocates would retreat to the value of tests when defending TDD, and then go back to insisting on TDD once the coast was clear.
But as far as I'm concerned that's a good thing. I write tests where it makes sense, and I can think of 1 project where I eventually regretted the decision not to write tests (before or after), but even then it didn't really hurt it just would have been a lot less scary with a good suite of tests around some specific functionality (related to pulling emails, doing the wrong thing there could mean permanently losing emails without processing).
Or the competitive advantages it offers do not align with the competitive advantages our society currently selects for. These days it seems that short term profits are the primary factor selected for and that users tend to go with the first to market until it has a significantly worse experience.
>But without going into any of those, the sheer fact that it's __this__ rare in practice says a lot, no?
It's a bit like how good security is a rare practice, as good enough security is much cheaper and the cost of security incidents is generally not as large as they should be (for example, data breaches have most of their costs pushed onto the victims of the data being leaked and not the collecting company that was breached).
Or it is an incredibly good procedure for a tiny minority of the projects, but sucks for the vast majority of them.
I just finished a quite large collection of parsers and did it entirely using TDD, got 100% coverage and it was much easier than when I did it without it in the past. I kinda HAD to get 100% coverage because how else would I test my code? I only wrote the part that read from files at the end of the project.
Also, when I was writing code that communicated with banks or telcos in arcane COBOL-era protocols, I didn't really have a way to "test" my code other than in production, so I relied on TDD for my day-to-day coding. It worked fine.
For GUI stuff, or web development? I used TDD in the past and didn't gain anything for it.
This should be obvious for everyone, but TDD only works when it works... it's not a silver bullet.
I wrote a pdf parsing library and it's a joy to test and the tests give you a lot of certainty the code works because the unit is the library itself, the input is the file and the output is well defined.
But I really am beginning to dislike testing most code in a web app. If you have a pure calculation or some business logic, test away. For everything else some higher level tests give far more assurance, when you're moving fast those might even be primarily manual. A good rule of thumb I think is that a test using mocks is a bit of an anti pattern, they make you feel like you're testing code when generally you're testing your test.
The typical test pyramid is upside-down.
Libraries usually have specification. Web gui? Not so much.
Yes, I completely agree!
It took me a while to get used to write code like this, but having the business logic completely unbraided from the interface/database code is probably the biggest productivity boon I ever had, because my tests run blazingly fast. People don't think it matters, but instant feedback makes a lot of difference.
This is also how things like DDD, Hexagonal Architecture, Functional-Core-Imperative-Shell are structured, so we're not alone. It doesn't have to be as complex as some of those: it just has to be easily testable...
-
> A good rule of thumb I think is that a test using mocks is a bit of an anti pattern, they make you feel like you're testing code when generally you're testing your test.
Oh, I couldn't agree more.
My personal pet peeve is having to write unit tests for thin-controllers or Spring-style services. I take an hour to carefully mock 10 deep dependencies, and in the end all I'm testing is if the language is able to call methods in other classes.
Instead, if I just write the silliest integration test, I'll probably get better coverage and it will be able to uncover more bugs.
That's what I've heard called "trading external quality" for time to market. That is exactly how you're supposed to do it, at least according to the dev coach that I've been listening to (@GeePawHill)
Why you write tests and refactor is to keep your code's internal quality up. Those are the things that customers can't see, the things that make it harder for you to change your code when you come back to do more work later.
The Correlation Principal states that ISQ and developer productivity are directly correlated and that you cannot trade internal quality for development time to market without this trade-off quickly coming back to bite you, in the form of lost productivity.
A feature that is only partially implemented, or an edge case that isn't tested properly and causes something to fail at runtime, are both examples of external quality. You can indeed come back and fix that later, and you can make this trade cautiously to get your work to production faster.
But if you write a long method for example that "just works" and ship it without further introspection, refactoring, or complete test coverage, you may never recover from that. It might require devs to spend an extra hour or more each time that method should need to change in the future. In some future state it has become so long and complicated that nobody can fully understand it, the time to stop digging is whenever you see yourself inside of this hole.
And according to Sandi Metz, those big long methods are almost always guaranteed to be the ones that will need to change again, they are long and complicated because those are the business objects that you care about. They should be well factored as early as possible to facilitate future changes, (unless you have a crystal ball and can say for sure that won't ever need to happen!)
> You can indeed come back and fix that later, and you can make this trade cautiously to get your work to production faster.
I think that is only true in certain companies of a certain size. If you're a small startup trying to launch an MVP, yes, time to market will be prioritised over everything.
If you're in a regulated sector, or your company is any larger than e.g. 30 people, "going back and fixing later" gets harder and costly than shipping correct software in the first place; either because you'll damage your reputation, you'll have a deadline to implement the fix (unnecessarily adding pressure on your own team), or other teams start relying on the wrong behaviour and now your fix is not trivial anymore because it doesn't affect only a local scope.
I want to highlight this because it's a trend to think any company and any project can (and should!) be managed as if anything being done is a prototype, w/ a perception that this is "agile", and picking the worst set of trade-offs for the task at hand.
Some tasks are suited better for mad scientists, ... some for town planners. Some regulated environments are only suited to the type of work that town planners can do (6-sigma people, that's how it's also explained in the class.)
But the settlers won't have much to do if pioneers haven't done some heavy lifting first, and so on (thanks a lot, mad scientist,) and the town planners can't do the type of work that you need from them either, unless there have been many layers of groundwork laid before them.
Glad you enjoyed checking it out!
For future changes, factoring also requires a crystal ball since you have no idea how requirements will change or if your factoring will actually be useful for them. Better to keep things simple and contained until you actually need to make them more complex.
I've been doing TDD a long time (18+ years), and I'm definitely pragmatic in the sense that I don't test everything, definitely don't aim for 100% coverage, and don't use mocks often, etc. I aim to extract most of the value from the tests without paying an absurdly high tax on them. Still learning in that regard, but I have had some real successes in testing.
I find this is key in our broader team's ability to keep the internal quality high. It's obviously easy to mess up a code base, but (high quality!) tests have been an important part of that, in addition to refactoring, code review, etc.
For me the biggest takeaway was the way I was organizing my code. I work on a 15 year old web application with lots of legacy stuff, so it isn't easy to make changes fast. New code that I write depends on old code, so I am constrained a bit. Even still, I found marked improvement in my code structure after trying TDD. I do not know whether I'll stick with TDD, but I definitely will remember the code organizing lessons it taught me
The most aggressive shop I've ever seen set their bar at 95% coverage.
Sure, it might take a bit longer to ship a simple feature, but bugs also take away from feature development, and in a complex enough application small changes might result large bugs. I guess that's true about software testing in general, but I think most developers would say there's value in having tests.
Often, the tradeoff in shipping quickly is technical debt that impacts your ability to ship quickly in the long run. I think TDD can help manage that, especially if you're planning to write tests for a feature anyway.
It does depend on what you're working on, of course, but I think most companies over-estimate their need to rapidly ship features. In the grand scheme of things adding a week to a 6 week project isn't that big a deal and might even save time in the long run.
mmm maybe once. But that's a compounding delay. If every project is delayed by ~15% for TDD then over the course of years you can fall well behind your competitors.
> complex enough application small changes might result large bugs
Totally agree - but also on really tightly TDD-tests code bases small changes can result in huge test refactorings...many times to the point of just not doing something because the time to update the tests is prohibitively expensive.
There's a balance to all of this, and (empirically) TDD seems to be on the extreme end of the balance.
That said, I've found the most boost from the full XP menu -- pair programming, TDD etc. I can get a lot done with halved daily ritalin intake vs working alone.
Only this way does TDD not interfere with development velocity.
Of course, if all you have is a basic CRUD app, then there's not much testable "core" code that you can separate from your framework, so TDD and unit tests are probably not the best idea. IMO end-to-end and integration tests are the way to go anyway.
I like running my tests almost as often as I compile my code. If they run fast enough, it creates the most addictive feedback loop for TDD.
Besides, we should be measuring the time to write tests against the time it takes to manually test. If you don't execute tests every time you make a change, you're not really sure if your code works. And manual testing just means we have no idea whether or not our code works.
If I was to be an architect only for a system though, I'd likely appreciate TDD to ensure implementors do what I think I spec'd
On my own personal projects I have dabbled with TDD and I've seen the benefits it can provide, but it does make even simple programming tasks take a lot longer. Sadly, companies these days (especially during the pandemic) can no longer afford the luxury of development taking longer, even if it does mean the end result will most likely be cleaner and have less bugs. The company I work for see shipping potentially buggy code and fixing it as bugs are reported as an acceptable development practice.
With the advent of automated builds and deployment processes, it is way too easy to quickly ship code and roll back bad releases or push out emergency patches. Things don't have to be perfect the first or second time around. The optics to non-technical executives seeing code go out and features released are a lot better than seeing things take longer to develop.