This is actually why we all need to support systemd. It is already an init system, and a sophisticated one; now that it has integrated container support it's only a matter of time before systemd's pid 1 assumes zombie reaping duties for the host and all container images.

People who criticize systemd fail to realize that it provides solutoons for many problems under Linux, and even problems with the Unix model itself, and has widespread support from companies, distributions, and other stakeholders.

Yes, its provides a supervisord type mechanism, thats great, so is dependencies but outweighed by the following:

The horrid shite that is the journald. whoever thought it was a brilliant idea to smoosh grep tail and vi together is someone who's never had to debug anything hung over.

seriously why in living fuck do I have to use your shitty tool to look at system logs? The whole point of linux that you are supposed to be able to use anything to open anything. Sure its a stupid idea to use most tools, but thats not your decision.

Seriously do I have to learn a completely new syntax to find all things that happened at 10:23:04 three days ago? Everywhere else I can use vim/nano/emacs/awk/grep/sed.

but here? no I'm to stupid to understand those tools, here use this one. We've thought of every use case, if you're having trouble its because you're either a hater, idiot, sysadmin, or too old.

you can tell that the people who write this shit are not the poor fuckers who have to tune/support/debug/migrate or use it.

So no, I'm not going to "get behind systemd" until journal is fit for purpose.

If you're running a syslogd then the logs also get stored there, so you can continue to use the tools you are familiar with if that is more important to you than possibly considering different tools that may work better.

Much as I like the inference that I've not tried journal, or that I'm immune to new ideas, thanks, thanks very much.

Look, my job is to implement new ideas. I run a datacenter that has 15pb on tier 1 storage, and 28,000 CPUs.

The one thing I care about most is this: getting home to see my children.

The only time I look at logs is if something terrible has gone wrong. Everything else is done by sourcing metrics directly from the components directly (collectd + graphite)

We do ship logs, and yes we have elastic search, but thats far to heavy and far to higher latency to be useful.

So the only real time that I look at logs is when something is horrifically broken, or we cant figure out from the graphs what has gone wrong.

At that time, I do not want to have to remember new syntax to get at what should be text files. (yes signed logs, but lets be honest thats why you ship logs.)

If I got more functionality, then yes I'll be onboard.

However I can't use bash to talk to journal, It only has a C API, which smacks of it being designed for phones and super integrated HPC, not the 95% use case.

Because it uses binary blobs, (And yes, I do love binary, just not for logs) Its a pain in the arse to inspect using a scripting language.

if journalctl _COMM=sshd | tail | grep "something" then;

really nice. also what the fuck is with starting a switch with an _? underscore indicates hidden function. seriously, why is it different to every over log inspection tool?

It's not a switch, it's a match argument, and the underscore prefix indicates that the field is "trusted": the kernel retrieved the value and the logging application is unable to provide "fake" data for the field in case it has been compromised.

And people who blindly push systemd fail to realize that we've alreqady solved these problems. A more complex system doesn't solve it.

So instead of a 350 line Python script, you think systemd is better? I disagree.

A more complex system? How is solving this problem universally a bad thing?

Is having _one_ way to do things that works everywhere so bad? Having a container manager do things I wouldn't expect, like have issues with zombies or dropping syslog messages, makes me never want to touch it.

My dear chap, one way does not work everywhere. if that were the case we'd all have agreed to just use vim, or emacs, or sublime, or nedit, or nano, or ed......

Vim has largely won vs. emacs, and gone on to the next bracket against ST. Emacs is out of the tourney. I can't thibk of a single hacker under 30 who isn't a total Lisphead, who wants to even touch emacs.

Docker does different things than VM's, but they work together perfectly. I run my all my Docker hosts on a VMWare infrastructure, which gives me best of both worlds. Yes - for some things, VMWare can't be beaten - but that's the nice thing: they are not mutually exclusive. Both are very handy tools in my toolbox, and neither are a one-stop solution. Docker sure adds complexity - but also a huge amount of flexibility. Once you understand the power of it all, and how to think with Docker (which isn't that obvious) - it's pretty damn good. Not that there are no downsides or things that could be improved, but nothing is perfect.

And then:

1) The problem is hugely exaggerated. It only applies to badly written software. And make sure we don't lose syslog? Seriously? just mount /dev/log into your container (it's a unix socket after all) and you're done.

2) Arguably faster? Try booting up 15 instances of app X. With docker, that's achieved in less than a second. On VMWare this has a huge memory and CPU overhead, not to mention boot-up times.

3) Try versioning VM's. Good luck. I have 100% reproducible, versioned images generated and tested in our CI system, that I can easily deploy in a VM on my laptop where I will have 100% the same setup as it will in test/qa/prod. To deploy the new version, start new instance, point load balancer/proxy/whatever to the new instance (happens automagically btw). Everything goes well? Stop old instance. Something goes wrong? Point proxy to old instance, and kill the badly-behaving. If everything went well: zero downtime. If you detect something goes wrong, it's nearly instant to switch back to the previous version. Also, testing this is a lot easier than with VM's. Sure you could do that with VM's too - but the overhead is massive, and takes a lot more time.

My ideal setup would be a hypervisor infrastructure with a bunch of VM's running only Docker applications.

> My ideal setup would be a hypervisor infrastructure with a bunch of VM's running only Docker applications.

You should take a look at https://github.com/docker/machine and https://github.com/docker/swarm , they were designed with exactly that goal in mind!

Sadly, it still lags behind even KVM, it can't hot migrate, which means zero downtime maintenance is impossible, along with transparent HA.

Care to share your CI tech stack? Specially the deployment bit automagically updating the balancer. We are internally building that kind of setup at the moment and some insight from experienced people like you would be very helpful. Thanks.

CI is Jenkins, which uses the Docker plugin for creating slaves on-the-fly. The plugin has it's share of problems, so I don't use it to build the images itself - I do this using a generalized bash script, which also pushes it to our private repo.

For automatically updating the load-balancer, check out docker-gen (https://github.com/jwilder/docker-gen). I hacked together setup that detects the latest version and points to this-one when a container comes up or goes down. It assumes a lot of things specific to my setup though (versioning of the containers, env vars, ...)

This is not talking about writing a new init in the sense of Systemd. All the init program needs to do is spawn the one process that you were going to start anyway, and adopt orphan processes.

You could literally use bash as the init process if you do not care about shutting down cleanly (or are willing to have a more complicated shutdown procedure that 'docker stop').

FWIW, and I say this as a big Docker fan, I had the exact same reaction. Docker, to me, is for immutable stuff where I don't care about the end results of that particular instance of the software. Turning containers into thinner VMs doesn't do much for me.

A hypervisor has a ~150-250MB memory overhead over linux containers + namespaces because it boots a full kernel.

There are other differences but if all you want from your system is reproducibility then VM images work fine of course.