Yeah, docker compose is great for running distributed apps locally when everything “just works”. The problem is that it also impedes development when the compose setup isn’t fully optimized.
I work on dockerized rails apps, and whenever a new dependency is added, it can take 15 minutes to rebuild the image, which completely breaks my flow. Docker also creates a ton of data bloat locally, requiring me to run docker system prune pretty regularly. Setting up a dependency cache isn’t completely straightforward either.
I still think docker compose is worth it, but it’s a deceptively complex beast to get right, and most companies I’ve worked for don’t take the time to get it absolutely right. There’s definitely a cost to using it that people tend to overlook.
Just remember to have a process to rebuild your base image at least once a month so you pick up latest security fixes etc.
You could try making your own base image that installs your gems. Eg, copy your Dockerfile to Dockerfile.base. Remove everything after the bundle install, and build to say app-base:latest.
Then change the FROM in your Dockerfile to app-base, and remove any OS package installs. Keep the bundle install; it will use the layer from app-base if unchanged, or do an incremental install if not.
This will reduce the amount of data bloat too.
You could try using guard-bundler or something similar to further streamline the process when adding gems
Personally, I found having multiple intermediate dockerfiles more confusing than helpful, so having an easy to follow, centralised build made life a lot easier for me, especially if you produce multiple images that are pretty similar at the base.
Long term I'm planning on moving these builds to bazel and taking advantage of incremental builds for my entire monorepo. I'm in the final stages and just validating some stuff is in place before I do that. In the short term I've mitigated this by adding in heavy unit test coverage. Backend devs can basically run the unit tests for most services and are confident the service is working as expected and also enforces any/all security/auth checks. When developing I don't even start up the containers. I boot them up once at the end just to check my work.
Then just have a different Docker-compose config for prod
1. My applications have no dynamic "linking"
2. The containers are distroless
3. Services can be built independently
4. I run the same exact sha in dev/staging/prod
The better solution for me is looking into a build system that can support my different language and tooling choices which happens to be things like Blaze (Bazel/Pants/Buck/Please).
I assume this is due to docker downloading the related image and running the container.
If this is the case, isn't this orders of magnitude faster than adding the dependency manually. And aren't the gains are even more significant when we factor the fact that most of the cases dev and prod environments are identical? Or am I missing something?
Yes
> If this is the case, isn't this orders of magnitude faster than adding the dependency manually.
No. When developing a rails server running locally, you just need to install the new dependency via bundle install (usually takes just a few seconds) and then your server is ready to run again. With docker, the image needs to be entirely rebuilt, which is a much slower process, even with good caching. The difference is that docker treats the build as ephemeral and willing to be thrown away and rebuilt, as opposed to running directly, which simply augments the existing environment. It’s a much, much slower process to rebuild via docker.
It's worth noting that bundle uses a Gemfile, so it's not installing each Gem as a seperate step.
I wasn’t involved in the setup but adding a new gem or upgrading one is very quick. We use a mounted volume to hold the gems as seen here: https://github.com/concord-consortium/rigse/blob/master/rail...
1. sets up automatic port forwards to my host system
2. handles ingresses by binding port :80 on my system
3. provisions PVs for you when you create PVCs and respects retention policies
4. Allows you to select from a monitoring stack and only require that you supply configurations for dashboards/alerts that can be 1:1 applicable to production so devs can debug dashboards/alerts locally.
`k3d` is targeting itself as being Rancher's kubernetes implementation so if you buy into all of the Rancher tooling it'd be a great option.
This means that you need to build your app using Docker, and then import it to k3d/kind environment, thus needing local registry (not enabled by default), which means that images are duplicated.
We do a lot of integration with third parties which is essentially pull down some data and write it into a microservice responsible for keeping that data. Each of these integrations is owned by a separate engineer because it takes up a lot of time to deal with integration calls, planning, etc and we need to be language flexible (some people only give you a .dll or .jar, no choices).
I'm assuming people building these tools don't use a local development workflow because: their systems are too big to run on a single machine (Googlers) or don't work on large projects (open source side projects).
The clostest thing to what you and I are looking for, though, is skaffold and tilt.dev.
The command is supposed to be: 'k3d image import --cluster myCluster image:tag', but it fails with "Failed to start exec process in node 'k3d-myCluster-tools'"
On kind it works fine with 'kind load docker-image --name myCluster image:tag'.
I'm using WSL2.
[1] Yes, our Makefiles sometimes call docker/npm/ant/maven/gradle/cargo under the hood. Its been worth it, in my experience. One syntax on all projects to set up the workspace, refresh dependencies, deploy to dev env, clean, etc.
1. The syntax is very complex
2. It's just a wrapper around shell command execution
3. There's no single standard everyone fully implements
How is this any more complex than any programming language? Take javascript. Do I want to use =, ==, or ===?
> I cannot test if my Makefile will run on a developer's mac if I don't have a mac myself and even then it's a crap shoot (do they have homebrew? is their PATH correct? etc).
Containers suffer from these kinds of problems as well. For example, if your ip tables are not set up just so, you get no network access from inside your container.
Just learn it then? Running make commands is trivial. Writing basic make targets is trivial too.
Or you can just make everyone wrap their build tools in make.
Make is not a "wrapper" it's a "translucent layer on top of" since it doesn't hide the deps on the underlying tools.
When I do `docker build -t ... something` or `bazel build //something` I don't need to care if it's node, C++, Java, etc. I just need to know that it is a thing that I want to build. Make cannot deliver that to you as you need all of your system toolchains installed and you need to make sure your makefile works on every OS/system config.
For me, that's kinda the point. If (when!) the underlying layers break, I can open up the Makefile and see exactly what was called, without too much abstraction. It's not there to provide complex build options, it's to provide aliases to the commonly used commands in a platform agnostic way.
Most of my make commands are simply aliases for one or more commands, which are often something like `docker run --rm -it project_runner_image some_command`
There's some real problems with Makefiles in the form of tabs as the prefix, running everything as a shell, weird platform specific versions, etc - but I've yet to see anything else which is as small, as universally available, with such a low barrier to entry. Most of the other options I've seen so far (and I'll be the first to admit I haven't looked very hard) seemed to either be overly complex for simple dev setups, or bound to a particular language.
I agree that blaze is the better option, but all the benefits it provides comes with a greater up-front cost, where its not necessarily possible to drop-in on a project and run with. Also notable is that blaze is inspired by and is a replacement for make. So in a lot of ways, make is a poor mans blaze now.
Actually I find most answers either in official documentation or on StackExchange sites. And I much prefer digging in mailing lists over digging in GitHub issues. You access both over web anyway.
We've got multiple apps deployed various ways, everything from pseudo-automated CI/CD (auto build but manual deploy) to fully manual deployment directly out of our IDE. Nothing set up within Docker containers yet except for a few proofs of concept.
That said, a self-contained system in the context of a container orchestration framework, docker-compose, kubernetes, etc, facilitates an number of benefits, not the least of which is the ease of scaling engineering staff.
For example, in typical organizations that claim to have modern development practices, there are static dev, test, and staging environments for a given service. Obviously these are subject to config skew, etc. and must be time-shared between folks. When it's a simple task to create an environment per engineer for each purpose or any other, that can accelerate development tremendously.
There are other benefits around resilience in production, but that's another topic entirely.
Most of the issues seem to be with Postgres DB connections, but it makes starting up our backend painfully slow...
Edit: Some tuning options also exist - https://docs.docker.com/docker-for-mac/osxfs-caching/
I've also done deploys just using managed services like ECS/Fargate or Heroku where we just build an artifact and push it to the host with a script. A stack that's a load balancer, stateless app server then a DB and/or file store can be provisioned manually once then not really worried about for a long time.
Ops gives devs "disposable environments" to do their testing in. Basically, GitHub + Jenkins + Terraform + AWS. I've used this to stand up real infrastructure every time a PR is opened, and pipelines run against the infra. Code and infra match each other because they both exist in the same branch/repo (monorepo; you can of course do multi-repo, just takes more coordination). It's all destroyed as soon as the PR is closed, but you can also keep it up to do dev work against. You can also keep one copy of the latest master branch up at all times (the "dev" or "cert" or "test" infrastructure) as a shared test environment. Downside: you have to have access to the network (which is in AWS, so that's not so difficult). Upside: the dev environment always mimics production, devs don't need to do anything to stand it up.
Option 2: A bunch of mocks, and scripts to stand up stuff in Docker, using docker-compose or something else. This becomes a bit of a problem to manage with lots of devs, though, and docker-compose is not a good model for production deployments.
1. S3/Azure Blob Store
2. DocumentDB/CosmoDB/Cloud Atlas
3. Aurora/A for PG DB
4. ElastiCache/some complex name in azure
If there was ever a case that I needed to ever use a real - full vendor lockin - solution I would hide it behind an API of my own and configure that API to store data locally. For example, the Azure Blob Store local container is completely broken for any sensible usage and I've been on an github issue thread for ~3 months waiting for them to resolve the issue (you're forced to link your netns to that container to talk to it as the application clients only works over 127.0.0.1). So essentially I have the following settings:
- SERVICE_STORAGE_DRIVER=azure/fs
- SERVICE_FS_LOCATION=/data
- SERVICE_AZURE_CONTAINER=....
- SERVICE_AZURE_....=....
NixOS, however, I will not recommend as your dev environment. Use a regular distro and use NixOS for your containers.
We used to be able clone 400 VMs in 1 hour, sanitize all customer data and refresh a full dev environment from m prod in an hour. TBs of data.
But let's all stand around and claim k8s is not your flavor of the day solution to a very old problem, but one that caries a huge amount of complexity and so much network effry that it scares the pants off most people.
HN uses * for formatting so it's been replaced with {STAR}
Another approach, from a previous startup, who mainly/only used PHP was to volume mount the code you were editing into the containers that were running and to expose an xdebug port for our IDEs to connect to. This was very hacky and error prone but when it worked it worked.
Something that I plan to do in the future is to setup some incremental build system that can build all of my software and use that to build my containers. Whenever these outputs change I'll have them redeploy the container in docker-compose or minikube. This way I get very fast builds, very small containers, and much fewer hacks.
I think this is the key take away for many startups. Get it so you:
Once you have that in a docker-compose.yml file migrating to something like kube when you need health checks, autoscaling, etc is easy.The only thing you must be critical of is bashing people over the head to make everything you run in your company run in this environment you've created. No special deployments. Establish 1 method with 1 tool and go from there.
Every company I've worked at I've brought a single command `docker-compose up -d` workflow and people were amazed by the productivity gains. No more remembering ports (container_name.local.company.com), no more chasing down configuration files and examples, no more talking to shared DBs, etc.