This works remarkable well in practice and allows for large scale refactorings under the hood with little to no impact on the tests. We can also mock databases, memcached, redis and graylog on their respective http/tcp/udp level. This in turn means no custom build mocks which could break when refactoring. The tests itself also contain no logic, they are pretty much just chained method calls with data that should go in and an expected response that should come out, along with a specification of all external resource our API fetches during the request and their responses etc. Any unexpected outgoing HTTP request from our server will actually result in a test failure.
As for scaling this approach, from our experience it works quite well, especially when you have lots of complicated interactions with customer APIs during your requests since the flows are super quick to set up.
I took this approach on several different projects, but I figured that a lot of the boilerplate/infrastructural code that you need to actually write these kinds of tests is poor or simply not available.
For example, declaratively starting and running multiple services together, in parallel and at the right time (with service dependencies) and printing their logs out.
Or, mocking the forward passage of time. (click something -> move forward a week -> click something else).
Or, asynchronously 'listening' using epoll for emails on a mock SMTP server that logs out the emails it receives.
Selenium's good for the web interaction stuff, but you need much much more than that to be able to effectively test at this level.
I think this kind of testing would be much more widely used and effective if the tools available were up to scratch.
In our special case we have about 100 different endpoints all versioned and all dependent on multiple endpoints from the (rather badly documented) APIs of our customer. Most of the work our API does is spent combining / enriching the customers data and performing integration across the subresources. Setting up individual mocks for every single on of these complex requests flows manually is pretty much impossible at this scale.
So doing black box testing and enforcing a 100% test coverage (best for avoiding dead code) helps keeping us sane. In the end we don't care so much about how the implementation behind our HTTP response looks as long as we return the correct data in the end. The code itself still has to look good though :)
We've always found that writing your own smoke tests for the other guy's code saves a lot of head scratching and the game I like to call Blame Tennis, when each side insists that any new problem must be in the other side's stuff because surely WE haven't broken anything.
Typically run as a pre-build sanity check when you have remote integrations.
On some Labs projects we've written extensive request tests and stub services based on documentation, then handed those to the upstream service. Usually there is some angst at this point, being the first time that any kind of TDD suite has turned up to ask awkward questions.
Sadly, despite not working, they were still about 4 years ahead of their competitors.
(Disclaimer: I work on Newman as a part of my day job)
In the face of grossly changing requirements, I've never had much luck keeping E2E tests up and functioning properly. And people have a bad habit of investing more time and energy than a particular test is worth in trying to keep it working or porting it to the new requirements.
Unit tests are cheap. If the requirements change invalidates twenty of them, you just delete them and write new ones. Easy.
For our use case we have some pretty "fancy" deep-equals logic for nested structures and allow to specific fields as "to be ignored" in our test expectations.
When it comes to speed, the most important part is to cut out everything you don't need, for us this means that our testing framework completely throws away the internals of Node.js HTTP layer. We never create a single TCP socket in our tests, which gives an incredible speed up. As a bonus this also allows to test timeouts and low level HTTP errors in < 1ms, since we can just return the timeout directly from the low level APIs and Node.js will invoke the timeout event on the http client handle.
> Running "noir:mock" (noir) task > ...................................................... > 1048 passing (10s)
And yes, that really reads 10 seconds and yes I always get a bit bored when I have to run the tests for one of our Django backends, feels like an eternity... :)
...sadly that's still only for a single project x config x arch combination. I still need to play around with incremental linking options that appear off by default...
Gotta be careful there chief. In practice is usually refers to "your experience" (but might not be mine).
My experience is on the flip side: end-to-end tests are super slow due various reasons...
Those kinds of "atrophied" tests can make a refactor considerably more painful than it should be.
But TDD, in my experience, gives a slightly different dynamic. Because you generated the code being motivated by the tests, there are a lot of unit tests that don't test functional units - tests that are essentially testing implementation decisions.
I've seen situations where that big refactoring would involve scrapping or rewriting tests, and that causes its own kind of architectural conservatism.
I ended up in a compromise position. I use a lot of tests, but I only do TDD in niche cases where it suits the problem I'm solving.
I've seen this happen to myself numerous times as well, but I don't think this is good argument against TDD.
If a tests fails because the implementation details change, that tells us that we wrote a bad test, not that TDD is bad. As I get better at writing tests, I am better able to catch these kinds of errors ahead of time. There are a ton of little techniques you can use and questions you can ask yourself about your code that can make your tests better.
Furthermore, test code is just as important as production code, and we should be just as ruthless in refactoring and peer-reviewing each others tests (within reason).
TDD is NOT suitable for every problem, this is true. But I think too many people give up on it because they see "the problems of TDD" which are actually just "the problems of writing bad tests".
Note: I'm not trying to target you sago, but merely making a general observation. I'm talking more about people who don't write tests even close to the time of writing the code, or don't write any tests at all because the their tests kept breaking for no good reason and were slowing them down.
I strongly disagree. The only thing that matters to how software works for an end user is its boundary. You simply can't do TDD by only testing the boundary.
You have to write a test before each unit of code, so you have to test those internal interfaces, the way units of code work. But that is exactly the stuff that should be allowed to change. It's a mistake to assume 'implementation details' only exist within a function. That stuff you can avoid testing, but the behavior of the function is itself an implementation detail for whatever is using that function.
I simply cannot imagine TDD for anything other than a toy example where you don't end up writing tests that depend on the software's implementation, and are therefore inertia to changing that.
> we should be just as ruthless in refactoring and peer-reviewing each others tests
I strongly agree. But this is rarely part of the TDD ethos. Saying "I refactored the code and threw away 100 tests." is likely to send TDD folks twitching, in my experience.
You're ignoring the fact that tests aren't only to validate "end user functionality". They're also there to make life easier for the developer; to make both initial development and maintenance easier.
If I need a custom sort algorithm for my product, I'm going to write unit tests for it's implementation. I'm also going write functional tests to make sure the end user is seeing things sorted. However, those tests serve different purposes.
The unit tests allow me to write clean, simple tests that show exactly what the implementation is supposed to be doing for different inputs. That allows me to change the implementation of the sorting algorithm later and worry less about breaking the functionality.
The functional tests are (generally) not going to be as clean and obvious, nor cover every possible partition of inputs that the implementation could get.
Erm, no, that was my entire point!
I think my points align pretty well with what you were saying instead of against it.
I would start with writing an assertion about the end state of the sort. Then using syntactic substitution of the predicate calculi derive the program by assertions.
I'd then have a specification of the algorithm as a mathematical model from which I can check every possible execution for the entire domain on the sort. If I'm so inclined I would have a good basis for a proof.
The unit tests then become assertions on the implementation and don't have to verify correctness.
It would be awesome though if I could just generate the code directly from the specification!
It's easy to dream up algorithms and traditional unit tests for this (and easy to see how the large input space makes the approach suboptimal). I'd love to see how your approach would work.
I'm familiar with Coq and Prolog(one of which I think you're hinting at), but I'd reach for Perl or C over either one if I needed to write a trading program or generate a statement from my accounting software or something.
I think of it like this: nobody is going to care if I screw up the script to balance my accounts. If I write some software that messes up the balance of everyone's account in some bank then someone is going to care.
I don't think formal methods and rigor are the sole domain of government institutions, avionics control software, and industrial robotics. The public relies on the cloud and the applications that run on it and they have no recourse if their data is exposed or disappears.
Imagine if instead of crossing over a bridge you were stopped at a booth and asked to sign a wager stating that nobody is liable if the bridge collapses underneath you while you cross it. Rich or poor, regardless of how much you pay in taxes, you should be able to rely on that bridge being built to the highest standards we know how and someone should be liable if it falls down. You can rely on this in real life. Why do we not have this for software?
I'm not saying I write all of my software using formal methods and rigorous proofs. It's often good enough to sketch my idea on the back of a napkin when I'm building a tool shed. But nobody built a sky-scraper without doing a little math and being fairly rigorous in their thinking.
I'd be interested in knowing how, and what kind of thing you build. I can almost imagine it for an API or framework, where almost every implementation choice is on the boundary. But even then, you have to test private functions to do TDD, surely.
I work on real time stuff, and in that stuff, the user-facing functionality is hard to test, and usually not tested in a TDD kind of way. The TDD stuff is all internal.
Now, safety equipment in a race car is something else entirely. Half of it keeps you from dying immediately on impact, sure, but the other half actually lets you go faster. The 5 point harness, for example, helps keep you from losing control of the vehicle during high G maneuvers, by keeping your torso in the seat and therefore your legs and arms in (roughly) the correct orientation to the steering wheel and the pedals.
There are a couple things to consider that sometimes become taboo in orgs--but shouldn't.
The first is that you don't need to test everything. I recommend having a great set of those API-level black box tests as an acceptance suite, but at a component or unit level you should pick your battles. If you'd architect for DI -only- to test, that testing had better be pretty necessary, otherwise you introduced complication (and risk) without a lot of benefit. That risk can actually overwhelm any benefit of testing.
The second is that you should absolutely scrap tests of implementation details at the point that the implementation becomes a black box. Tests are like scaffolding during construction, and we don't leave the scaffolding up when the building is done--we do inspections of the building instead. The same mindset should be applied to testing. Different phases need different tests, and too many tests at too low a level of abstraction absolutely cause maintenance and flexibility issues if you're not willing to let them go.
But that's no reason to back off entirely. SOME things need to be unit- or component-tested SOMETIMES. Those things generally benefit from TDD.
It's all about making intelligent decisions. There's a sometimes-controversial movement in QA/test called "context-driven testing" which comes down to there are very few always-best practices, and strategy should be tailored to your problem and situation--not done by rote, book, or prior (read, your boss's or CTO's) assumptions of what's always right. Otherwise you tend to get something more onerous than useful.
The considerations around that approach apply at the low-level test tier as well.
I call this test concreting:
https://hitchtest.readthedocs.org/en/latest/glossary/test_co...
The way to avoid it is to either be damn sure that the API you've surrounded is tight, clean and unlikely to need to change much or to test at a higher level.
There shouldn't be any. Those are not useful at all.
Additional point is that invariant-based testing can easily be transformed into automated fuzzy tests using Quickcheck and its analogues, and this can find immense amount of otherwise hidden bugs.
That makes no sense at all.
> Your example is a perfect illustration
I didn't give an example.
> invariant-based testing can easily be transformed into automated fuzzy tests
'fuzz' not 'fuzzy'. True, but irrelevant to the point being argued.
Yes, rewriting tests is part of a refactor. You always have to be willing to do that or just scrapping outdated tests.
I too don't have the disposition to patiently wade through fixing tons of tons of broken tests over and over again. IF I have to I can do it, but I know better than to unreservedly trust my judgement about how things are progressing.
But what this has taught me is to stop trying to push water uphill. If testing is hard because of questionably coupled code, refactor it NOW instead of waiting for things to get bad. The refactoring almost always suggests new features we could add to the code (or makes me backpedal on pronouncements that certain things were 'impossible'), and the number and kind of tests that break is reduced.
I came across a quote recently from Bertrand Meyer (the Design By Contract guy), where he suggested that code for making decisions and code for acting on those decisions should be separated. I found myself nodding along to this because it was something I knew intuitively but had never articulated: Decisions without actions don't need mocks, and actions without decisions need at most one or two (often none). Decisions can be tested (possibly in parallel) with unit tests, and actions tested with functional tests. Then all your integration testing is that decisions lead to actions (basically that event dispatch works) and that catastrophic failures are handled gracefully. A proper testing pyramid.
Now I want to figure out which of his books or interviews this was in because I want to read the rest of what he had to say on the subject.
Think of it this way: pre-testing makes you afraid to change you mind because you throw out your tests, not post-testing makes you afraid to change your mind because you might break something. If you think changing your mind is good the path forward is clear.
An even bigger problem is that large scale refactorings can result in the tests themselves no longer being correct (which isn't the same thing as “not passing”), if you test at too fine a level of granularity. However, when you wrote these tests, you couldn't have possibly foreseen a large scale refactoring 18 months into the future, so how do you tell in advance what the right level of granularity for your tests is?
I find tests to be extremely helpful for designing APIs. I write tests to enforce the contract my API is making with its consumers. Therefore, TDD forces me to consider the API from the user's perspective, rather than designing an API based on implementation details. That also helps to cut down on the cost of refactoring—unless I'm making an API-incompatible change, my tests should keep passing. This gives me more confidence as I'm refactoring, allowing me to make more sweeping changes.
I tend to write tests when I'm not confident in the implementation or when the tests are trivial to write. At least, that's how I do it for new developments. If you're releasing code that people are relying on, you should have a whole suite of user tests and regression tests.
If you changed that functionality intentionally, then the other things it interacts with will require a change regardless. Furthermore, if it's that small of a change(as usually, the smallest changes are the ones that impact the most tests), it usually doesn't require any change to the test, aside from adding/subtracting a parameter or something.
Major functionality changes should carry weight. It's like saying that a motorcycle runs so much faster without a person on it- it defeats the purpose.
I greatly-prefer the warm fuzziness of knowing that I have a large number of unit tests as a "safety net" to detect if my own thoughts on what is "correct" are in some way wrong.
In my observation and feeling, the statement: " Because you want to ensure that you always pass the majority of tests, you tend to think about this when you change and extend the program. You therefore are more reluctant to make large-scale changes that will lead to the failure of lots of tests. Psychologically, you become conservative to avoid breaking lots of tests."
is most likely due to the low quality and low test coverage of the exiting code base. So that the tests are fragile, seemingly hinders changes.
If things are started with TDD, I would be really surprised that tests hinders change.
Just because there are unit tests, doesn't mean they are covering all the code foremost, or that they are written properly to truly detect when things have changed for the worst.
Tests are not a replacement for actually testing and verifying your changes in whatever application gradient you're working in.
I agree. If I'm working on some part of the app that doesn't have any test coverage, I might add some high level tests before I start.
Interesting. I've often found that the lack of tests leaves me absolutely terrified of making changes to large Ruby and JavaScript applications. I mean, if a test breaks, then I get a nice message right on the spot. But if I don't have tests, then the application itself breaks, and I may not find out until it's been put into production on a high-volume site or shipped to users.
Once an application crosses, say, 25,000 lines of code, it's hard to keep an entire program in my head, especially in a dynamic language and with multiple authors working on the code base. Under these conditions, large scale refactorings or framework upgrades can cause massive test failures, but the only alternative is to cause massive, unknown breakage.