FYI Docker already has a RESTful API, and programming container start/stop is trivial to do in any language. I haven't used Testcontainers before, and can kinda see the utility, but IMO it really isn't worth it in the long term to take on a new external dependency for a bit of code that (1) is a critical part of the team's development and release process and (2) can be written in-house in maybe an hour.

Conveniently as of a few days ago, you don’t need to be worried about this anymore. TestContainers is no longer a third-party!

https://www.docker.com/blog/docker-whale-comes-atomicjar-mak...

If only Docker used a Sperm Whale as their mascot….

> it really isn't worth it in the long term to take on a new external dependency for a bit of code that (1) is a critical part of the team's development and release process and (2) can be written in-house in maybe an hour.

This seems to be quite a contradiction. If it's so easy to just write from scratch, then why would it be scary to depend on? Of course, it's not that easy to write from scratch. You could make a proof-of-concept in maybe an hour... Maybe. But they already took the proof of concept to a complete phase. Made it work with Podman. Added tons of integration code to make it easy to use with many common services. Ported it to several different languages. And, built a community around it.

If you do this from scratch, you have to go through most of the effort and problems they already did, except if you write your own solution, you have to maintain it right from the git-go, whereas if you choose Testcontainers, you'll only wind up having to maintain it if the project is left for dead and starts to bitrot. The Docker API is pretty stable though, so honestly, this doesn't seem likely to be a huge issue.

Testcontainers is exactly the sort of thing open source is great for; it's something where everyone gets to benefit from the wisdom and battle-testing of everyone else. For most of the problems you might run into, there is a pretty decent chance someone already did, so there's a pretty decent chance it's already been fixed.

Most people have GitHub and Dockerhub dependencies in their critical dependency path for builds and deployment. Services go down, change their policies, deprecate APIs, and go under, but code continues to work if you replicate the environment it originally worked in. The biggest risk with code dependencies (for non-production code like test code) is usually that it blocks you from updating some other software. The biggest risk with services is that they completely disappear and you are completely blocked until you fully remove the dependency.

I think people depending on Testcontainers are fine and doing very well with their risk analysis.

> This seems to be quite a contradiction. If it's so easy to just write from scratch, then why would it be scary to depend on?

It's easy to write an implementation specific to your existing project's development environment and workflow correctly.

It's hard to write a generic version that works in any environment. It's hard to write a version that lets you build a company and make money.

It's scary to depend on this generic version because it's too generic, and it's built by a for-profit company now who wants to upsell you to some "testcontainer cloud" crap, which doesn't exactly incentivize them to make the OSS version perfect.

For example, we were already using bazel, so writing a version using bazel to create a correct bare rootfs for a dependency + using runc to execute it in tests resulted in something with a roughly 3ms warm startup time that fulfilled all our needs, and cached correctly (since bazel has good caching).

A junior engineer, starry-eyed at test-containers, switched some code over to it, and the warm startup time went up by 1000x from 3ms to 3s, as did the flake-rate, and docker's caching is far worse than bazel's so the several minute cold-starts also happened even more often.

> Testcontainers is exactly the sort of thing open source is great for; it's something where everyone gets to benefit from the wisdom and battle-testing of everyone else.

You get to have a mismatched mess of solutions to everyone's problems, including solution's to problems you don't have. You get code of the quality of the average OSS programmer which, while higher than the average programmer, is still pretty crap.

Free Software is great when it's run by a small group of smart opinionated people. As soon as you try to build a company around that OSS and, god forbid, hire some enterprise sales people and product managers, it quickly becomes worse at the actual small developer problem than what even a mediocre developer could hack out in an hour.

Depending on testcontainers is fine, but if you know what you're doing, writing something purpose-built is fine too, and probably gives you something much nicer in the end.

To be honest, it doesn't sound like you really had a good use case for Testcontainers anyways. Where it excels the most is in just pulling in some external containers, especially databases, e.g. PostgreSQL and Redis, directly in your test harness. In those cases, it works well.

Our use-case was redis servers, one of your supposedly good use-cases for testcontainers.

I didn't mention, but the testcontainer code also preferred to add that network io to the actual test runtime, which made measuring the test's performance harder, and meant we couldn't as safely cache the test's results. The bazel version made it easy to build the test's dependency as part of the test compilation process (as it should be) so the test runtime didn't have to do external network IO.

"Excels" is also a stretch; we had a few hundred tests launching dedicated redis servers, and with hand-rolled code, that worked fine with zero flakes. With testcontainers, it regularly flaked with some opaque docker network error or sometimes just plain a timeout because apparently launching 100s of containers in parallel is a hard problem for docker or something.

I'm sure it works well for some people, but if those people wanted to build out their own version without docker and testcontainers, specific to their development tooling and environment, it would probably work better in most cases I think.

Yeah, I couldn't tell you: I didn't have serious issues with performance or reliability when I used this. It didn't really seem like it was doing anything special, it seemed like it is just starting a Docker container like you'd expect, at least on Linux. I sincerely doubt I would've had any better experience handrolling it since my solution to handroll it would also be to use the Docker API; the reason for this is for portability.

When writing something like this for yourself you would only support one database, one programming language, one unit testing framework, much less documentation and none of the edge cases that someone else requires.

The effort would not be equal to replicating the whole project. It wouldn’t be the same quality of course. Question is do you need all of it or not.

Yeah, but that's kind of the thing, you get less battle testing most likely for also less surface area. If you pull in a project like this, it makes it vastly easier when you need more, because if it happens to already have built-in support, you don't need to spend additional engineer time on it.

To me it seems straightforwardly a win to start with Testcontainer and move to something else when it proves insufficient.

This thought process taken to the extreme is what results in half the internet depending on leftPad and isOdd to function. Open source is great, but there is a difference between providing a utility vs adding needless layers of abstraction. It is the responsibility of the developer to figure out which is which rather than reach for the package manager for every task in front of them.

In this case, like I said earlier Docker already has a RESTful API. You don't need to replicate the entire feature set of Testcontainers, just make a couple of HTTP calls in your language of choice to achieve the exact same outcome.

That's just a slippery slope argument. I am not arguing for people to depend on leftPad, I'm arguing that depending on a library like this which is probably tens of thousands of lines of tested code that does roughly what you want already is a damn good idea. Again, I don't want to handroll a solution and have to test it out across Windows, Mac, Linux and then check again to make sure it works with both Docker and Podman, and then again to check AArch64 Windows, AArch64 Mac and AArch64 Linux, and so on. I'd prefer to use a library. And hell, even if any of those things don't work, I can report it to the maintainers and possibly contribute a fix, and then everyone else gets to benefit too.

Can you explain more in more detail why this is a game changer if i already have an inhouse framework that is similiar in using docker for integration tests? Does it start docker up faster then you could do normally? Is it just the out of the box apis it provides?

I dont know why integration testing like this is considered a gamechanger. the testing pyramid is a testing pyramid for a reason and its always considered them important. Sometimes starting with integration tests in your project is right because your dont waste time doing manual point and clicks. Instead you design your system around being able to integration test, this includes when you choose dependancies. You think to yourself "how easily will that be able to be stood up on its own from a command?" If the answer is "not very good" then you move on.

If you have an existing in-house framework for anything, maybe it's not worth switching over. It does help though when a best practice bubbles to the top and makes this in reach for those who don't have an existing in-house framework and who wouldn't know how to get started on one. It also helps for more people to have a shared understanding about a subject like this thanks to a popular implementation.

Meanwhile, Testcontainers is done quite well. It's not perfect, but it's sure better than the in-house stuff I built in the past (for the same basic concept).

No, it does not start faster than other Docker containers.

I do challenge the testing pyramid, though. At the risk of repeating my other comment on a different branch of the discussion: the value of integration tests is high, as the cost of integration tests has decreased, it makes sense to do more integration testing, at the expense of unit testing. The cost has decreased exactly due to Docker and mature application frameworks (like in Java: Spring). (See: Testing Trophy.)

one thing i have seen with testcontainers (been a user for a few years) is the ergonomic SDKs that they have especially in languages like golang, it makes spinning containers up/down, accessing the ports (eg: a mongodb container for some e2e test flow) super trivial - its like a nicety layer on top of vanilla docker (w/ the cost of including their sdk in your test build process)

yes, 100% can be done using docker directly or docker rest api (and def doesn't make sense to migrate if you have already made an investment in an in-house framework that doesn't require much upkeep)

thanks for the responses, i just wanted to cut through the marketing. taking on standardised tools is a win for me, i just wanted to know about real world experience and use-case. Indeed taking on deps is not something i do lightly.

> value of test pyramid

I mean more from the perspective of covering your bases, you never just want one kind of testing pattern in your project. Each codebase is different and i agree that taking on high value test styles/cases is a project by project challange that should be tailored by many variables. The shape of your testing pyramid may be different to others. If your inheriting a legacy system, maybe its top heavy because the effort/reward ratio just isnt there. In this circumstances i usually take on the approach of "add more layers when bugs are found" to hone in on places that could use more or less test coverage.

Our inhouse framework is really just a wrapper around certain tools that fill different gaps (think docker/selenium etc) in order for different projects to build suites that are compatible with our ci/cd pipelines that do things like generate environments on demand to run test suites against. So dropping in testcontainers to replace the home-grown docker will be trivial. Keeping test frameworks fresh and compatible with the cloud vendors that agreesively upgrade is a challange just like keeping the API bleed of other programming deps is. Our test suites essentially have a domain language that is consistant. We can upgrade selenium, swap functions for different operations, without having to change any tests. Same goes for unit or integration tests - they are exactly the same in terms of assertions, syntax etc, they may just just have different environment setup logic. CI/CD can inject and overrride logic as it needs. Sometimes its suitable, in some cases, to mock certain external hard deps in integration tests for instance to having all the unit testing tools availible a plus. Or in other cases, we may take a unit test written against mocks, and inject real deps into it for certain CI/CD scenarios.

I think they make the biggest difference when testing data pipelines (which have historically been difficult to test). You can now easily test out compatibility between different versions of databases, verify data types, embed as part of your build, etc.

I believe the next step, once using test containers, would be automating data generation and validation. Then you will have an automated pipeline of integration tests that are independent, fast and reliable.

You can automate data validation with snapshot tests. I do it this way with a data pipeline and have a function that queries the destination DBs and puts them unto json to be written validated with a snapshot

If you are testing a microservices "ball of mud", you can (and probably should) setup a testing environment and do your integration tests right there, against real dependencies. The tool seems nice for simple dependencies and local testing but I fail to see it as a game changer.

I agree with this. At work we use both approaches but at different levels of the test pyramid.

To test integration with 1 dependency at class level we can use test containers.

But to test the integration of the whole microservice with other microservices + dependencies we use a test environment and some test code. It's a bit like an E2E test for an API.

I would argue that the test environment is more useful if I had to choose between the two as it can test the service contract fully, unlike lower type testing which requires a lot of mocking.

This is useful too but expensive.

Test containers provide a middle ground.

For example we have pure unit tests. But also some tests that boot up Postgres. Test the db migration and gives you a db to play with for your specific “unit” test test case.

No need for a complete environment with Kafka etc. It provides a cost effective stepping stone to what you describe.

What would be nice if test containers could create a complete environment, on the test machine and delete it again.

Still a deploy with some smoke tests on a real env are nice.

It's not really a middle ground if you're not testing your service in the same conditions as in production environment.

If you're not testing integration with Kafka, and the producer, your service is still lacking integration tests.

Testing classes in isolation with testcontainer is fine. But I observed that with microservice architecture the line between E2E tests and integration tests are blurred.

Microservices can and should be tested from the client perspective.

True but if you want to test every merge-request it becomes expensive.

We had a customer k8s cluster per feature branch with e2e testing.

A middle ground is testcontainers for feature branches, and the trunk branch a full e2e suite deployed to a live cluster...

In my last project we used https://java.testcontainers.org/modules/kafka/ to start a small Kafka container. It's not the exactly like a production installation, but it goes a long way.

Agreed, this is the best way, testcontainers are over hyped.

Yeah, I prefer setting up docker-compose.yml myself, so I can startup services once, and also do manual testing.

The only thing I would maybe use testcontainers for is to deploy my own service into docker as part of integration test so I can test a more realistic deployment scenario instead of running it locally outside docker.

> and local testing

You mention this as an afterthought but that's the critical feature. Giving developers the ability to run integration tests locally is a massive win in a "ball of mud" environment. There are other ways to accomplish this locally, but the test-infrastructure-as-test-code approach is a powerful and conceptually elegant abstraction, especially when used as a tool to design testcontainers for your own services that can be imported as packages into dependent services.

I very strongly disagree. Having "integration" tests are super powerful, you should be able to test against your interfaces/contracts and not have to spin up an entire environment with all of your hundreds of microservces, building "integrated" tests that are then dependant on the current correctness of the other microservices.

I advocate for not having any integrated environment for automated testing at all. The aim should be to be able to run all tests locally and get a quicker feedback loop.