My background is physics, and the philosophy there is "If you can't reproduce it, it doesn't exist."
In my experience with software, sharing opaque binary blobs is a Bad Thing. It doesn't matter if the blob is a proprietary firmware image, or a "open source" docker image. It's a magic unreproducible image. And (to me, at least) therefore suspect.
> Docker package management is like that: You make some changes, you commit often, fork your instance and try lots of different things. Eventually you find something that works and you tag it.
Again, the physics philosophy is "If you don't know what you did, you didn't do it".
For software, programming by randomly trying things is one of the worst methods available. You should understand what you're doing, and understand the system you're building.
> You share your tags with other people. This is an extremely productive way to get work done.
Sure. It's a way to quickly create, test, roll back, and share opaque binary blobs. It's "productive" in that you get things done. But you have no real idea what you got done. You just have a magic image that "works".
I would not hire a programmer with that kind of attitude. I've been burnt by that method, and those kind of programmers, too many times.
I believe this assumption needs to be challenged.
Aside from merely believing that.
I can't even tell what you mean. If I want to rebuild my machine same as before, or deploy build 100s of servers at different points in time with the same setup, reproducibility is necessary.
There's not even one argument against that. If you have any, please oblige us.
Should deployments be a surprise?
No it isn't.
Reproducible is talking about the process or steps needed to make a machine. You do not need to repeat the steps to produce the first machine to produce the second machine. You can simply clone the first machine, and Docker makes this very easy to do.
Keats begins his article with the statement offered, without justification:
In this post I'm going to assume that reproducibility is good and necessary.
Since I don't believe that, and in fact believe the opposite, I take issue with the entire article. Indeed, he goes on to construct a straw man comparing the exact opposite of what I propose -- a non-interactive "dockerfile" and not even a particularly good one, with a very well thought-out "purely functional" build system.
> Any significant contribution you might have for software engineering in regards to that?
I don't know of any big technology company that doesn't design their live infrastructure around ephemeral machines cloned from a template.
They don't all use Docker, and they don't all use interactive development, however, but these two tools can make it easier for people who (for example) don't have Google's economy of scale.
> Should deployments be a surprise?
No they should not.
Deployments take less time with my interactive method because you do not have to wait to "rebuild everything" which can take hours.
Remember heartbleed? People using nixos had to rebuild every component that used OpenSSL, then test them, then find something else broke and repeat the process. Only once they had finished building their new process could they roll out this process to all machines, doing one more final rebuild. Even if done perfectly this has a minimum of two iterations.
I could simply create a new clone, get it working, then tag it as the next release. New instances can then be cloned from the release, monitored, and the old instances destroyed. The entire process took under an hour for the entire fleet, including reading about heartbleed on HN. This is clearly almost as fast as running `apt-get dist-upgrade` on each of the live systems, but without any of the risk.
The organisational complexity, however is admittedly much higher than "wash, wait, repeat", which is why most system administrators could not even begin to use live clones until virtualization and tooling had become much more commonplace.
> I can't even tell what you mean.
And yet you feel qualified to tell me I'm wrong.
Cloning is not the answer since it's all or nothing -- where reproducibility can be achieved piecemeal.
Only offering cloning means that starting with the same initial Dockerfile and making some needed updates to it, one can't be sure that the NON updated parts will be producible. So you don't get flexible reuse.
Only offering cloning also means that when you need to update only a part (e.g. because of security reasons) you can't just use the same dockerfile with the changed component and be sure that you'll get an otherwise identical system. So you don't get flexible updating.
Cloning also required an already set-in-stone configured system to be the prototype.
But until one reaches that, they have to experiment with various configurations (to end with the specific Docker image they want). Without reproducibility, they can't be guaranteed that their various test configurations under test are the same and only their latest changes differ. Perhaps another thing they left as is broke too.
>I don't know of any big technology company that doesn't design their live infrastructure around ephemeral machines cloned from a template.
That's just an argument from popularity. And not necessarily a popularity they opted for, while also being provided with the alternative.
That they use "ephemeral machines cloned from a template" is just what they have to do -- not what would be ideal.
Besides, reproducibility doesn't preclude cloning as part of the process -- it's a superset of it, offering way more flexibility.
>I could simply create a new clone, get it working, then tag it as the next release.
But if you mess with the clone it's not a clone anymore. And if a whole team has to mess with the clone over some period of time to update or fix it, there's hell to keep track what went on.
Nobody argues that you can't simple install cloned images as they are.
What exactly do you mean by flexible reuse, and why is it good? What exactly do you mean by NON updated parts and what does it mean that they are "producible"?
I don't understand your words. Speak plainly!
I don't use a Dockerfile, and the interactive development approach means normally you start up your clone, work on it for a bit, then commit the clone.
Here's an example:
$ docker run -t -i debian:jessie bash
root@264b51d62d0e:/# apt-get install nvi
E: Unable to locate package nvi
root@264b51d62d0e:/# apt-get update
root@264b51d62d0e:/# apt-get install nvi
$ v=`docker commit 264b51d62d0e`
$ docker tag $v whatever
> But until one reaches that, they have to experiment with various configurations (to end with the specific Docker image they want). Without reproducibility, they can't be guaranteed that their various test configurations under test are the same and only their latest changes differ. Perhaps another thing they left as is broke too.
This is a danger that requires discipline. Committing instances is very cheap, so it is good practice to commit often, and keep notes (commit messages). I think version control is common enough that most sysadmins know how to do this, and other packaging systems suffer from the exact same problem.
However because it's interactive, that need to experiment is satisfied fully and the sysadmin/user gets to use all of their tools to develop the working prototypes.
> But if you mess with the clone it's not a clone anymore. And if a whole team has to mess with the clone over some period of time to update or fix it, there's hell to keep track what went on.
This is not true.
With Docker you are encouraged to make many clones and branches and try different things out. The clones that are useful get tagged and forwarded to others.
A sysadmin who isn't using version control has other problems!
> Only offering cloning also means that when you need to update only a part (e.g. because of security reasons) you can't just use the same dockerfile with the changed component and be sure that you'll get an otherwise identical system. So you don't get flexible updating.
I don't understand this complaint, but it mentions dockerfiles again. Let me be clear: I never use dockerfiles.
If I want to update for security reasons (as I mentioned in my heartbleed example), then I will find it easier and faster to implement the fixes interactively. Having to write my recipe for nix-os and run it and wait while it runs is insecure because you are vulnerable longer.
> Besides, reproducibility doesn't preclude cloning as part of the process -- it's a superset of it, offering way more flexibility.
The non-interactive (and "reproducible") approach is slower, less secure, more prone to errors. I don't believe it can possibly be "more flexible" since the interactive approach can do everything the non-interactive approach can do and faster.
I do admit that Docker's build artefacts use more disk space and more Internet bandwidth than plain text nix configuration artefacts, but this is not fundamental to interactive development, which is specifically what I'm advocating.
> Any significant contribution you might have for software engineering in regards to that?
> That's just an argument from popularity. And not necessarily a popularity they opted for, while also being provided with the alternative.
Arguing that "nobody does it so it must be wrong" isn't any better than "just because everyone does it doesn't make it right".
I really think you need to learn about something and make up your own mind, instead of repeating "Docker is Bad" because you read something like that on a blog.
I also think you should really try to understand what I am suggesting before you go arguing against it.
> What exactly do you mean by flexible reuse, and why is it good?
I already explained. What exactly don't you understand?
It's about being able to reuse a recipe (dockerfile, etc.) for creating your final artifact while ALSO being able to change it -- as opposed to the "take it or leave it" case with cloning some binary blob.
> What exactly do you mean by NON updated parts
I wrote "Only offering cloning means that starting with the same initial Dockerfile and making some needed updates to it, one can't be sure that the NON updated parts will be (re)producible".
My typo aside, the meaning is clear: with cloning available but no reproducibility, when you want to only change part of a container (e.g. update a specific piece of software there), you don't know (when you re-create the container) that the other parts you didn't change are as they were before.
Basically that's the very definition of non-reproducibility.
> I don't use a Dockerfile, and the interactive development approach means normally you start up your clone, work on it for a bit, then commit the clone.
That's neither a good, nor a new process. That's like 90's sysadmin work. For one, you're doing all this work manually. Second, if you want to revert/change something you made, you have to mess with your clone, potentially putting it in some weirdo state -- you basically remove the scriptability part and just mess with your clone manually. Or you write some custom scripts or use some provisioning software -- which brings you right back to the reproducibility discussion.
With reproducibility you wouldn't have to commit the clone -- just the "recipe" to make. And you would still COULD commit the clone if you wanted (as I said, reproducibility is a superset of merely working with clones).
> The mistake with the first line didn't mean I needed to start over. In a realistic example that could save me an hour of rebuilding.
Doesn't save you anything over reproducibility -- since the latter does not prevent cloning. But having only cloning (which is what the parent lamented) does deprive you of the benefits of reproducibility.
"In a realistic example", for example, things could get much more hairy (and many more tests and changes could be needed), beyond merely forgetting to update apt before installing nvi.
And all of these changes and process would be totally opaque to your clonable blob.
> I can push my tag to other developers I'm working with, and they can use `docker diff` to find out what the differences are.
The can use docker diff and try to guess what the differences are -- and what the intention was, etc. Because docker diff is just a record of file changes, not of procedures followed and the intentions behind them.
> This is a danger that requires discipline.
And that's (part) of the whole problem. Anything that requires discipline on the human part and that could be automated is busy-work -- and error prone.
> With Docker you are encouraged to make many clones and branches and try different things out.
Yes, but that's meant for safe-keeping, like vm-snapshots. It doesn't replacing actually having a written record of what you did, what's supposed to be istalled inside, and why.
> I don't understand this complaint, but it mentions dockerfiles again. Let me be clear: I never use dockerfiles.
So again, a throwback to the manual devops age.
> Arguing that "nobody does it so it must be wrong" isn't any better than "just because everyone does it doesn't make it right".
Perhaps. But I fail to see where I did that.
> repeating "Docker is Bad"
Who said Docker is bad? Docker is great. It's non reproducibility that's the issue.
You must be confused. I can create and delete files on my filesystem -- changing it -- even though it is a binary blob.
Perhaps you mean something else?
> with cloning available but no reproducibility, when you want to only change part of a container (e.g. update a specific piece of software there), you don't know (when you re-create the container) that the other parts you didn't change are as they were before
I keep hearing you say when I want to re-create the container, but I don't ever hear you say why I would want to re-create the container.
> For one, you're doing all this work manually
Wrong. It's less manual work.
Writing a nixfile is manual work. It's hard manual work because none of the tools are interactive, and it's very distracting waiting for the computer to reply the nixfile between interations.
Interactive development is superior.
> if you want to revert/change something you made, you have to mess with your clone
No I don't. I discard it.
Do you use git rebase on your own history? Or do you create a new branch with a cleaned up history?
I do the latter.
> potentially putting it in some weirdo state
I don't need to reinstall my operating system every day because I get confused about what is on my computer.
> The can use docker diff and try to guess what the differences are -- and what the intention was, etc. Because docker diff is just a record of file changes, not of procedures followed and the intentions behind them.
The intentions are recorded in the commit log. That's why docker commit allows commit messages.
Procedures are only recorded if someone records them. Why would I do that if I only have to do it once?
> "In a realistic example", for example, things could get much more hairy (and many more tests and changes could be needed), beyond merely forgetting to update apt before installing nvi.
That's what I said, but you're missing the part where I don't have to wait hours and hours and hours while nix rebuilds my system over and over again.
"Reproducibility" has enormous costs, and the value proffered can be had with better tools.
> Anything that requires discipline on the human part and that could be automated is busy-work -- and error prone.
You're not automating the nixscript-writing, so you haven't saved any work. You've created busy-work by reinstalling your operating system over and over again.
> Yes, but that's meant for safe-keeping, like vm-snapshots. It doesn't replacing actually having a written record of what you did, what's supposed to be istalled inside, and why.
No, it's exactly a written record because that's what the commit messages are for.
If I want to know what's inside, there are other tools (dpkg) for taking inventory.
> So again, a throwback to the manual devops age.
Nonsense.
For some reason you only want to count the time that your script is running from when it is finished, instead of the time and expertise needed to develop the script in the first place, that you only need to run once.
If you can't understand that a sysadmin's job, whether they are writing a nixfile or they are directly interacting with a machine, is manual work, then I just can't imagine how I can be understood by you.
That's insane.
That's nice for cloning, it's not reproducability.
How do you know that the docker image was created with the right set of software? How do you know that there wasn't anything else added to it?
Copying the final image means you're just copying an opaque binary blob. You can't do the work yourself to build an image, and get the same docker image.
Something like "nix" makes that simple. Everyone can agree on the steps required to get from A to Z. Everyone can independently agree that each step looks the same for everyone.
For me, it's not about ease of use. It's about security and trust.
Define "the right set". If I ask someone to prepare something I might use
docker export/diff
> How do you know that there wasn't anything else added to it?
docker diff
So what?
Why exactly do you think "copying an opaque binary blob" is bad?
Don't you ever copy files on your hard disk?
Have you ever used `dd` to copy a filesystem as a binary blob to a disk or a USB key?
Did you know "git" copies "opaque binary blobs" around? Do you really prefer RCS because the version control is stored in plain text files?
What exactly is your complaint here?
My position is simple, but apparently radical: I recommend interactive development of systems because it is better, and I define better as faster and more secure. "Reproducible" simply is not a goal.
> For me, it's not about ease of use. It's about security and trust.
You should rethink your position.
Nix packages from sources you don't trust aren't more secure than docker images (even if you use nix.useChroot=true), which, barring bugs in docker and Linux, at least get virtual memory, disk, and networking.
Nix packages built by yourself are harder to write than docker tags you build yourself.
Or to get out of computers, you have a drug where they say it cures cancer 100% of the time and no one can reproduce the results, it's fine for you?
Reproducible in this case is talking about the process or steps needed to make a machine. You do not need to repeat the steps to produce the first machine to produce the second machine. You can simply clone the first machine, and Docker makes this very easy to do.
That I can clone a machine does not mean that I can clone the interaction between a patient and their drugs, but if I could it would most certainly be preferable to trying to repeat the experiments with larger and larger populations.
I can't reproduce me, so do I not exist?
And yet you don't know why.
This is called dogma, and dogma is bad: It leads people to make stupid decisions over and over, all the while repeating that the alternative is worse.
Then you don't know what the word "dogma" means.
What I said was "In my experience." i.e. practice learned over time.
This cannot be interpreted by any reasonable person as "blobs are bad, just 'cause."
That kind of straw man argument leads me to conclude that similar issues are being your inability to understand the use of reproducability.
You aren't offering any evidence or justification besides the statement. I might as well say, "in my experience, ``reproducability (sic) is a waste of time''" -- what's the point of that?
Instead, I'm providing concrete examples of why it's a waste of time, and rather than anyone arguing the contrary, they're doing exactly what you're doing: Repeating that there's some point of "reproducability" (sic). There isn't, you're wrong. End of.
But whatever, my background isn't physics, so what could I possibly know? Running engineering teams for the last two decades that have to develop and deal with high volume, high uptime application servers only means my systems have to work and make money, not that they be "reproducable" (sic).
If you put your hand in a fire 5 times and it burns you 5 times, you don't have to understand anything about fire to know that you should probably stop putting your hand in. And if you saw other people about to put their hand in, you would mention to them that everytime you put your hand in, you end up regretting it.
We don't need religion to keep from getting burned when we have tools.
It's also a surprisingly convenient way to build tests.
If we design systems out of a collection of processes, that we can replace at will, then what we actually have is a collection of checkpoints and branches that we can try simultaneously. Replacing individual branches allows us to try out changes there, without having to rebuild (and wait) the entire system.
Another way to think about this, is that Clojure is more like subversion than like git: It might be better than RCS, but you're still stuck with a linear revision history, and new changes can't completely conflict with old objects and old changes that are created.
And yet another way, is to imagine all of those old class definitions that old objects hold onto are actually global variables, because they have global (time) extent, and yet are unreachable, so it's difficult to find out whether your new code is being affected by those old settings.
Smalltalk, and even other (esp. Commercial) Lisps don't have this problem. It has nothing to do with interactivity, but with hidden state.
I mean, it was a win, not having to untangle the hairball of rpm/deb dependencies. I like that part too. But not understanding the downside was what shocked me. It was like everyone drank the magic kool-aid potion.
Really? This surprises you?
The gross majority of the world runs software this way.
The real issues are quality control and release process.
That's exactly how I recommend applying patches.
If it breaks things you can fix it.
You don't do this on a live system: If you're smart and your live systems are ephemeral you can copy your image, get it working, verify it interactively, and once you have a working image, you spin up new instances using the new system, and start shutting down the old systems.
We've been down this road before. Hacking something until it works, then imaging it and passing it on to your coworkers is a maintenance nightmare.
For example, did you read that article?
You know Docker has none of those problems
> The first problem with a snowflake server is that it's difficult to reproduce. Should your hardware start having problems, this means that it's difficult to fire up another server to support the same functions.
docker tag/push/pull
docker start tag
docker commit/export/import
docker push/pull
docker history/diff
He spent over 500 words to say that's probably not a good idea.
No kidding.
Now.
What I'm actually advocating is interactive development: The exact same way we develop our APL workspaces and our Smalltalk images, can be used to design and build server infrastructure.
Instead of a development cycle that goes:
* Edit nixfile/dockerfile/vagrantfile
* Wait (sometimes hours)
* Test things
* Repeat
You get a very short:
* Do stuff
* Get happy
* Tag the results
Then when you want to run "in production", you publish your tags/image, review the history with your team, not dissimilar to how Smalltalk people will clean up their changes file, and then introduce the new version into the cluster. Once you're satisfied the world hasn't ended, you can decommission the old images.
You can even make scripts that do these many of these steps automatically (divert x% of traffic to new instances, collect results, and so on). If you have a well-designed infrastructure, you can have junior developers use those scripts to make changes on their first day.
Now.
I get it, you have realised that if you start over often enough you get good at it.
I get that you've seen people who treat systems development as a completely linear descent into unmaintainable bat barf that can only be recovered by a bad day reinstalling everything and "starting over",
I also get that you think amortising that cost by taking those hours to rebuild your entire system when glibc changes is necessary to avoid that bad day.
I also get you like static external configuration like Dockerfiles or Vagrantfiles, and view them as instrumental in the audit of that process.
However while you understood that I disagree with that, you didn't understand why: If you had a decade or so of Smalltalk or Lisp experience instead of just ruby experience, you might have known exactly what I meant, but instead of asking, you just ignored the part you didn't understand, making a straw man argument that you did understand.
After all, what's more likely, someone knows something important that you don't? Or someone knows something unimportant and just isn't as smart as you?
The "build from source" camp provides solutions to a couple problems that at first glance seem to be difficult in a "principled mutation of binary image" model:
- Moving to a different architecture, e.g. x86 to ARM or ppcle. In build from source, I essentially change some parameters and components in the lowest level and rebuild.
- Other big changes to base components, e.g. compiling for Linux/libc vs. building atop a unikernel (obviously heavily dependent on what the upper layers are doing, but I can imagine wanting to run similar code atop multiple bases).
- Sharing common components. This is not really a "from source" thing, but is an important part of Nix. If two images include components in different orders, in Nix the common components will be shared (at the arguably coarse granularity of a Nix package), while the linear structure of Docker layers means once two images diverge, everything further down the chain will be its own copy, even if the same bits.
However there's another, perhaps simpler solution: I can have a separate "wu-ftpd" container, and a separate "hylafax" container. This admittedly might not have been as easy when nix started, but it is very very easy now.
So what?
You don't need to reproduce that working state: The work has already been done!
Think about how the interactive approach Lisp and Smalltalk programmers use is superior to the compile/run/rerun approach used by C++ and Java programmers -- by exploring the problem in a nonlinear way, you can find your way to a solution more quickly.
Docker package management is like that: You make some changes, you commit often, fork your instance and try lots of different things. Eventually you find something that works and you tag it. You share your tags with other people. This is an extremely productive way to get work done.