News for hackers. Stuff that matters.
Re:Ask HN: Has anyone migrated off containerized infrastruct... by physicles on Thursday August 13, 2020 @07:18AM (#24142118)
You’re comparing cowboy sysadmin (mutable servers, ssh in and live-edit stuff) to process-heavy devops with CI. These are orthogonal to containers/not-containers.

If you don’t use CI, it’s easy to get fast deploys with containers. Just build the images on your dev box, tag with the branch and commit hash, and push directly to a docker registry (docker push is smart enough to only push the layers the registry doesn’t already have). Code is running 30 seconds after compiling finishes.

(Don’t want to pay for a registry? It’s trivial to run one yourself)

These aren’t foolproof, fully reproducible builds, but practically they’re pretty close if your tools require your repo to be clean and pushed before building the image, and if your build system is sane. Besides, if you’re used to editing code as it’s running on servers, you don’t care about reproducible builds.

Also, if you’re starting containers manually on the command line, you’re doing it wrong. At least use compose so your setup is declarative and lifetime-managed.

(Edit: s/swarm/compose/)

Re:Ask HN: Has anyone migrated off containerized infrastruct... by murkt on Thursday August 13, 2020 @07:44AM (#24142305)
> Also, if you’re starting containers manually on the command line, you’re doing it wrong. At least use compose so your setup is declarative and lifetime-managed.

As I wrote in a nearby comment, I'm not starting containers manually - we have compose, swarm, it's declarative and lifetime-managed.

However, we often need to do some bespoke data analysis, so we often ssh into a server, type `make shell` to launch a REPL and type/paste some stuff into it.

Re:Ask HN: Has anyone migrated off containerized infrastruct... by akvadrako on Thursday August 13, 2020 @10:33AM (#24143722)
You can do all that with containers quickly.

I develop my stuff with k8s with all it's lifetime management and have 10 second deploys from my dev box.

Re:Ask HN: Has anyone migrated off containerized infrastruct... by tluyben2 on Friday August 14, 2020 @10:53PM (#24165483)
Still 10 seconds. I have 0 seconds deploy for some ‘cowboy development’ I do; it is now a competitive advantage :) 10 sec (and for almost all setups I have seen it is vastly more that than) is a lot while devving and deploying for test. Each their own, but just fixing bugs live with my client (Zoom + me live on the test server fixing many issues in an afternoon) is vastly more efficient for me. Obviously committing from the test(now dev) server to github will result in ci/cd to the staging server, but workflows where I work on my local, commit, ci/cd to test and then the client tests is vastly slower and I do not like it; it feels like a waste of my time.

In my main business I am forced (regulatory) to do it all by the book; vastly less enjoyable for me.

Re:Ask HN: Has anyone migrated off containerized infrastruct... by murkt on Thursday August 13, 2020 @10:41AM (#24143804)
10 seconds deploys? How is this possible :) Do you have any links that explain your workflow?
Re:Ask HN: Has anyone migrated off containerized infrastruct... by akvadrako on Thursday August 13, 2020 @11:53AM (#24144683)
It's nothing fancy - I just use a typical k8s Service/Deployment object on GKE. A deploy is:

  1. docker build (most layers cached) - 2s
  2. docker push - 2s
  3. update deploy.yaml
  4. kubectl apply -f deploy.yaml
  5. kubectl rollout status deployments {name} - 6s
Re:Ask HN: Has anyone migrated off containerized infrastruct... by lovehashbrowns on Friday August 14, 2020 @04:51AM (#24155497)
Lucky!! With our infrastructure, deploys can take an hour or so. 10 minutes for the build, 10 minutes for the image to get built, plus the rest of the time for terraform to apply infrastructure changes across dev, staging, and prod. Only thing we do have is automated testing after every deploy so issues tend to get caught. But that's still so long for a deploy! I don't know of a good way to get it down faster.
Re:Ask HN: Has anyone migrated off containerized infrastruct... by jonfw on Friday August 14, 2020 @09:13AM (#24157459)
Why is terraform deploying infrastructure for every container deployment? Can't you just rollout onto the existing infrastructure?

Also sounds like there is some lay hanging fruit available by adding some caching/layering in the build process

Re:Ask HN: Has anyone migrated off containerized infrastruct... by lovehashbrowns on Friday August 14, 2020 @11:08AM (#24158890)
Ah, I should have explained more. That's an hour for our ASG + EC2 deployments. The only benefit we get is easy roll backs because it always deploys a new ASG. We're switching over to Spinnaker and started with our EC2 infrastructure. I think container deployments will be faster but still, an hour for EC2 deployments!
Re:Ask HN: Has anyone migrated off containerized infrastruct... by physicles on Thursday August 13, 2020 @08:34PM (#24151777)
Yeah this is exactly what I do too, works just fine. You probably already have something like this, but I hacked a bash+yq script that automatically updates all relevant yaml files with the latest image tag. So getting new code running is two lines:

make image push deploy

kubectl -f somewhere/deploy.yaml

Re:Ask HN: Has anyone migrated off containerized infrastruct... by shaklee3 on Friday August 14, 2020 @09:24AM (#24157606)
Take a look at skaffold. Ours is under 10s
Re:Ask HN: Has anyone migrated off containerized infrastruct... by jrockway on Friday August 14, 2020 @05:42AM (#24155804)
I think people kind of boiled themselves alive with Docker and don't step back to think if they're where they want to be often enough.

Docker first started getting traction when people were building their software with make. Make never quite got caching right (not really its fault), so nobody was really sure that their changes were going to show up in their release unless they ran "make clean" first. And, you had to have 700 helper utilities installed on your workstation for a build to work after "make clean". Docker to the rescue! It gives you a fresh Linux distribution, apt-get installs all the utilities needed for your build, and then builds your thing. It works 100% of the time! Celebrate!

At the same time, programming languages started moving towards being build systems. They want "mylang run foo.ml" to be fast, so they implemented their own caching. But this time they did it right; the "mylang" compiler knows about the effect every input file has on every output file, so it's guaranteed to give you the right answer with or without the cache. Some languages are so confident these days that you can't even disable the cache! They know it works perfectly every time. The result is extremely fast incremental builds that are just as reliable as clean builds, if you have access to that cache.

This, unfortunately, is not something that Docker supports -- layers have one input filesystem and one output filesystem, but now languages are producing two outputs -- one binary for the next layer, one cache directory that should be used as input (opportunistically) to the next build. The result is, to work around people writing makefiles like "foo.o: foo.c" when they actually meant "foo.o: foo.c foo.h helper.h /usr/include/third-party-library.h /usr/lib/libthird-party.so.42", EVERYONE suffers. "mylang run foo.mt" takes a few milliseconds on your workstation, but 5 minutes in Docker. Your critical fix is hamstrung by your CI infrastructure.

There are a number of solutions to this problem. You could have language-specific builders that integrate with your CI system, take the source code and a recent cache as inputs, and produce a Docker container as output. (Systems like Bazel even have a protocol for sharing cache between machines, so you don't have to copy gigabytes of data around that you probably don't need.)

But instead of doing that, people are writing articles about how their CI takes 3 hours when a build on their workstation takes 3 seconds, and it's because containers suck! But they don't actually suck -- the underlying problem is not saying "in production, we will only run .tar.gz files that contain everything the application needs". That is actually wonderful. The underlying problem is "the first build step is 'rm -rf /' and the second is 'make world'".

Re:Ask HN: Has anyone migrated off containerized infrastruct... by twic on Friday August 14, 2020 @09:04AM (#24157347)
> Docker first started getting traction when people were building their software with make.

A result of this is that containerisation took off hardest in ecosystems without good build and deploy tools. Getting exactly the right version of the application, every library, and the runtime has traditionally been a struggle in Ruby, a nightmare in Python, torture in C, etc, but pretty easy in Java. As a result, most of the Java shops i know are still deploying by copying a zip file or some such onto a server.

Re:Ask HN: Has anyone migrated off containerized infrastruct... by yrro on Friday August 14, 2020 @08:44AM (#24157142)
I'm astounded that docker still doesn't have the concept of 'build volumes' that can be used for streaming artifacts and caches into build steps.

That said, 'docker build' is not the only game in town. For a long time, Red Hat's source-to-image tool has been able to do incremental builds:

    s2i build --incremental /path/to/mycode mybuilderimage myapp
This creates a new container from the mybuilderimage image, copies the source code from /path/to/mycode into the container at /tmp/src, and runs mybuilderimage's 'assemble' script (which knows how to invoke "mylang run foo.ml" and install the build result into the right place so that the 'run' script will later find it). The result is committed as the myapp container image, which typically uses mybuilderimage's 'run' script as its entry point.

The next time the command is run, the first thing s2i will do is create a new container based on myapp, and invoke its 'save-artifacts' script, which has the job of copying out any build caches and other artifacts to be used in an incremental build. The container is then discarded.

Now, the build runs as before, but with the addition of s2i copying the saved build artifacts into /tmp/artifacts, so that the 'assemble' script can use them to make the build faster.

This isn't perfect: you pay for speedier builds with larger container images, since you can't delete build caches, etc. like you'd normally do at the end of a Dockerfile. But it's a good first step, and you can always have another step in your pipeline that starts with the myapp container, deletes unwanted files and then squashes the result into a single layer above the original base image that mybuilderimage was itself built from.

Re:Ask HN: Has anyone migrated off containerized infrastruct... by thewanisdown on Friday August 14, 2020 @11:23AM (#24159077)
It does, now. The whole build system was uprooted with the change from legacy build to buildkit. One of the current "experimental" features is build volumes.
Re:Ask HN: Has anyone migrated off containerized infrastruct... by tpetry on Saturday August 15, 2020 @04:02AM (#24166952)
Do you have a link with a description? I can only find years old open docker github issues, 3rd party software or some very hacky solutions
Re:Ask HN: Has anyone migrated off containerized infrastruct... by shoo_pl on Friday August 14, 2020 @06:03AM (#24155953)
This is a great comment - especially the the last 2 paragraphs.

Docker is not the same as containers - docker is just one way to build containers. I've not seen one yet that would be easier to use (and as as result faster than docker) but it does not mean we cannot create one!

Re:Ask HN: Has anyone migrated off containerized infrastruct... by otabdeveloper4 on Friday August 14, 2020 @07:36AM (#24156572)
I personally use Nix to build Docker-compatible containers in production.