I took this approach on several different projects, but I figured that a lot of the boilerplate/infrastructural code that you need to actually write these kinds of tests is poor or simply not available.
For example, declaratively starting and running multiple services together, in parallel and at the right time (with service dependencies) and printing their logs out.
Or, mocking the forward passage of time. (click something -> move forward a week -> click something else).
Or, asynchronously 'listening' using epoll for emails on a mock SMTP server that logs out the emails it receives.
Selenium's good for the web interaction stuff, but you need much much more than that to be able to effectively test at this level.
I think this kind of testing would be much more widely used and effective if the tools available were up to scratch.
In our special case we have about 100 different endpoints all versioned and all dependent on multiple endpoints from the (rather badly documented) APIs of our customer. Most of the work our API does is spent combining / enriching the customers data and performing integration across the subresources. Setting up individual mocks for every single on of these complex requests flows manually is pretty much impossible at this scale.
So doing black box testing and enforcing a 100% test coverage (best for avoiding dead code) helps keeping us sane. In the end we don't care so much about how the implementation behind our HTTP response looks as long as we return the correct data in the end. The code itself still has to look good though :)
We've always found that writing your own smoke tests for the other guy's code saves a lot of head scratching and the game I like to call Blame Tennis, when each side insists that any new problem must be in the other side's stuff because surely WE haven't broken anything.
Typically run as a pre-build sanity check when you have remote integrations.
On some Labs projects we've written extensive request tests and stub services based on documentation, then handed those to the upstream service. Usually there is some angst at this point, being the first time that any kind of TDD suite has turned up to ask awkward questions.
Sadly, despite not working, they were still about 4 years ahead of their competitors.
(Disclaimer: I work on Newman as a part of my day job)
In the face of grossly changing requirements, I've never had much luck keeping E2E tests up and functioning properly. And people have a bad habit of investing more time and energy than a particular test is worth in trying to keep it working or porting it to the new requirements.
Unit tests are cheap. If the requirements change invalidates twenty of them, you just delete them and write new ones. Easy.
For our use case we have some pretty "fancy" deep-equals logic for nested structures and allow to specific fields as "to be ignored" in our test expectations.
When it comes to speed, the most important part is to cut out everything you don't need, for us this means that our testing framework completely throws away the internals of Node.js HTTP layer. We never create a single TCP socket in our tests, which gives an incredible speed up. As a bonus this also allows to test timeouts and low level HTTP errors in < 1ms, since we can just return the timeout directly from the low level APIs and Node.js will invoke the timeout event on the http client handle.
> Running "noir:mock" (noir) task > ...................................................... > 1048 passing (10s)
And yes, that really reads 10 seconds and yes I always get a bit bored when I have to run the tests for one of our Django backends, feels like an eternity... :)
...sadly that's still only for a single project x config x arch combination. I still need to play around with incremental linking options that appear off by default...
Gotta be careful there chief. In practice is usually refers to "your experience" (but might not be mine).
My experience is on the flip side: end-to-end tests are super slow due various reasons...
Those kinds of "atrophied" tests can make a refactor considerably more painful than it should be.
This works remarkable well in practice and allows for large scale refactorings under the hood with little to no impact on the tests. We can also mock databases, memcached, redis and graylog on their respective http/tcp/udp level. This in turn means no custom build mocks which could break when refactoring. The tests itself also contain no logic, they are pretty much just chained method calls with data that should go in and an expected response that should come out, along with a specification of all external resource our API fetches during the request and their responses etc. Any unexpected outgoing HTTP request from our server will actually result in a test failure.
As for scaling this approach, from our experience it works quite well, especially when you have lots of complicated interactions with customer APIs during your requests since the flows are super quick to set up.