News for hackers. Stuff that matters.
Re:Understanding and Hardening Linux Containers [pdf] by X86BSD on Wednesday April 20, 2016 @11:55PM (#11539576)
Obviously the author put a lot of effort into this paper. Hard work shows throughout. Kudos to you sir!

Granted I realize the title is "Understanding and Hardening Linux Containers".

However, personally, this just illustrates my frustration and the frustration of others with the Linux world.

They exist in their own little echo chamber. Linux was not the first to create "containers" that are secure. There is no inclusion of other solutions outside Linux.

The reason other solutions do not have these problems is because the authors actually thought about the problem. And if the Linux camp would simply look outside their echo chamber long enough to see how others solved these problems before them this paper might not have been written.

It's just simply frustrating to watch Linux reinvent some wheels, poorly, time after time.

Flame on!

Re:Understanding and Hardening Linux Containers [pdf] by Annatar on Thursday April 21, 2016 @05:26AM (#11540627)
Totally agree with you. I fail to see the point of attempting to harden Linux "containers" (Linux actually does not have container technology, since cgroups are not containers). Why bother with that, when illumos, and SmartOS on top of it, provide free, open source Solaris zones technology? Not even installation is required, since the thing runs from random access memory, and can run Linux at the speed of bare metal?

Everybody is just flapping on Linux containers. Meanwhile illumos and SmartOS are eating Linux's lunch with techology which has been working reliably in Swiss banks since 2006, and it is free open source software. What's not to love?

There are T-shirts which say "Go away or I will replace you with a very small shell script." Now it's "Go away or I will virtualize your Linux inside of SmartOS." The damn thing runs like a bandit, while Linux containers keep on flapping trying to find a working solution with papers like these. Why bother, the problem has already been solved, go run SmartOS and enjoy a working solution!

Re:Understanding and Hardening Linux Containers [pdf] by z3t4 on Thursday April 21, 2016 @09:19AM (#11541855)
SmartOS if very nice in theory. I tried to get SmartOS working on a new Supermicro server and failed, even though I got a lot of help from their awesome community and people at Joyent.

Then I tried Linux (Ubuntu) and everything just worked! It was also very easy to find guides on the web for most things Linux.

Re:Understanding and Hardening Linux Containers [pdf] by Annatar on Thursday April 21, 2016 @09:47AM (#11542053)
If you want the JustWorks(SM), you have to get hardware which is on the Joyent engineering bill of materials' list, or research and buy hardware which illumos has drivers for.

http://eng.joyent.com/manufacturing/bom.html

SmartOS being descended from Solaris has detailed manual pages built into the operating system, because when one runs a real UNIX, one is entitled to comprehensive documentation with lots of examples.

A lot of the Oracle Solaris documentation, which is also very comprehensive, applies to and can be used on illumos derivatives. The SmartOS wiki and the smartos-discuss mailing list are good sources of SmartOS specific enhancements. Did I mention the built-in man pages yet?

Re:Understanding and Hardening Linux Containers [pdf] by RantyDave on Thursday April 21, 2016 @05:58PM (#11545622)
I've had a whole bunch of stuff just work. But if it doesn't, you're stuffed.
Re:Understanding and Hardening Linux Containers [pdf] by dozzie on Friday April 22, 2016 @04:58AM (#11547875)
> Meanwhile illumos and SmartOS are eating Linux's lunch with techology which has been working reliably in Swiss banks since 2006

They would be eating Linux's lunch if they were used in any wider scale.

A primary objection to Solaris zones and a reason they're not used widely is that they run on Solaris and not on Linux.

Re:Understanding and Hardening Linux Containers [pdf] by dmpk2k on Friday April 22, 2016 @09:21AM (#11548890)
SmartOS has the LX brand; in other words, SmartOS can look like a Linux kernel if it wants to. I've used Ubuntu on SmartOS, and it worked flawlessly.
Re:Understanding and Hardening Linux Containers [pdf] by Annatar on Friday April 22, 2016 @12:19PM (#11550278)
> They would be eating Linux's lunch if they were used in any wider scale.

Linux took 20 years to make its way into the enterprises, and well, SmartOS has to start somewhere. We are only at the beginning of the journey, but with one major difference: unlike Linux, which is still maturing, an illumos based operating environment has 37 years of enterprise abuse and hardening behind it.

> A primary objection to Solaris zones and a reason they're not used widely is that they run on Solaris and not on Linux.

What exactly is objectionable about running a Solaris-like operating environment? If it is software, SmartOS provides ~14,000 packages:

https://www.perkin.org.uk/posts/building-packages-at-scale.h...

Re:Understanding and Hardening Linux Containers [pdf] by dozzie on Friday April 22, 2016 @06:30PM (#11552987)
> [...] well, SmartOS has to start somewhere.

When SmartOS or generally Solaris-descendants' zones start being anywhere near popular, then we'll talk about eating Linux's lunch. For now the situation doesn't look like that even remotely.

> What exactly is objectionable about running a Solaris-like operating environment?

To me -- not much at first sight. But there is the factor of familiarity, which is very, very important. There are plenty of sysadmins, you know, who can recover Linux from almost any failure, and gaining similar knowledge in different OS takes quite long time. Solaris derivatives (or BSDs, for that matter) are at huge disadvantage to those people.

Re:Understanding and Hardening Linux Containers [pdf] by Annatar on Friday April 22, 2016 @10:21PM (#11553823)
> When SmartOS or generally Solaris-descendants' zones start being anywhere near popular, then we'll talk about eating Linux's lunch. For now the situation doesn't look like that even remotely.

By eating Linux's lunch I meant how good, reliable and capable the zones and lx-branded zones technology is. It is very reliable and very capable. It works very well while Linux is still trying to find itself within the cloud container paradigm: look at how many competing solutions there are on Linux in that space, and every single one of them exists because the other one lacked something, or did not do it in a satisfctory way. That is what I meant by everybody's flapping on Linux with containers.

Meanwhile, there are no such issues or problems in the illumos and SmartOS communities: the tech used there has been designed from the ground up to be secure and work for large enterprises, and unlike the Linux cloud technology, it has been in heavy use for ten years now. illumos and SmartOS are not trying to still find themsselves, because zones, ZFS, and Crossbow network virtualization already worked for years. Back in 2006 for instance, at a place where I worked, we were already packing eight Oracle databases per 32 GB RAM system, in production. Now it's 2016, and zones have only gotten better with the addition of KVM and Crossbow network virtualization.

> But there is the factor of familiarity, which is very, very important. There are plenty of sysadmins, you know, who can recover Linux from almost any failure, and gaining similar knowledge in different OS takes quite long time. Solaris derivatives (or BSDs, for that matter) are at huge disadvantage to those people.

As for those people, they had to learn Linux too; they can go back to the roots and learn a real UNIX the correct way now, and be glad that they have the opportunity to do so.

If one truly knows UNIX, one understands Linux on a much deeper level.

Re:Understanding and Hardening Linux Containers [pdf] by dozzie on Saturday April 23, 2016 @08:50AM (#11555337)
> By eating Linux's lunch I meant how good, reliable and capable the zones and lx-branded zones technology is.

Could you then use idioms in their proper meaning, please? Because I only protested against claim that zones have taken a market share of Linux containers, which is very untrue, and at the same time not the claim you have made.

> As for those people, they had to learn Linux too; they can go back to the roots and learn a real UNIX the correct way now, and be glad that they have the opportunity to do so.

So, zones provide better platform because people can go back and learn Solaris? Or what?

Zones need to provide something tangible over containers for people to invest in learning different platform. Having just another tool, which merely does what one already can do with containers, only slightly slicker, but severly limited in applicability because parent platform is not familiar to all the colleagues, doesn't look like a good trade.

And note that we were only talking about familiarity. There are still other reasons, like little developers working on a given Solaris derivative, which doesn't give much confidence that any bugs that occur would be fixed, or a concern that the supported hardware list will be long enough.

Re:Understanding and Hardening Linux Containers [pdf] by Annatar on Sunday April 24, 2016 @04:58AM (#11559036)
> Could you then use idioms in their proper meaning, please?

eat lunch

eat someone's lunch

Sl. to best someone; to defeat, outwit, or win against someone.

http://idioms.thefreedictionary.com/eat+lunch

...It can refer to market share like you argue, but in a broader sense, this is at least how I understood the term eating someone's lunch.

> So, zones provide better platform because people can go back and learn Solaris? Or what?

No, Solaris (well, illumos, since nobody cares about Solaris any more) provides a better platform for learning and understanding UNIX. And with that comes the insight of writing better code in general, especially on Linux. For example, one is able to write completely portable shell code and deliver cleanly packaged software on Linux if one understands UNIX (in /opt, /etc/opt/, and /var/opt instead of /usr[/local] -- and that is just one trivial example).

zones are a fully working solution because they provide full security isolation, while running as regular processes with their own init(1M), instead of being forced to run Docker inside of KVM or XEN virtual machines to achieve that. And because processes inside of zones are just regular processes in the global zone, they only use as much resources as they need, and no more. With zones, one does not have to hard partition and preallocate memory and CPU to achieve full isolation, as is very often the case with running Docker inside of a virtual machine.

https://www.youtube.com/watch?v=0T2XFSALOaU

And because zones' storage is backed by ZFS, the data is guaranteed end-to-end integrity, and self-healing if the zpool is redundant (RAID1+0, RAIDZ, RAIDZ2).

Writing filesystems is hard: https://danluu.com/file-consistency/

For its part, illumos is the reference system as it is fully POSIX, XPG4, and SUS / UNIX03 compliant, so learning it means learning and mastering the standards.

And, once spoiled by all the nice features in illumos / SmartOS, going back to Linux feels like going back to skins and stone knives.

Re:Understanding and Hardening Linux Containers [pdf] by dozzie on Monday April 25, 2016 @05:17AM (#11563202)
> Sl. to best someone; to defeat, outwit, or win against someone.

> ...It can refer to market share like you argue, but in a broader sense, this is at least how I understood the term eating someone's lunch.

It's still a far stretch to claim that zones win against containers. They could if we limited the field to just technical level, and then again, I would need to see some arguments why they actually are better.

>> So, zones provide better platform because people can go back and learn Solaris? Or what?

> No, Solaris [...] provides a better platform for learning and understanding UNIX.

You're confusing here learning unix with using it.

If one was at the stage of learning, Solaris could be fine, except there's much less material about it for newbies (especially about rough corners and stupid troubles, like "I ran `chmod -R 777 /'" or "I deleted my kernel"), and much, much less knowledgable people accessible to those newbies to learn from.

Now look at professional Linux sysadmins and programmers. They already have some knowledge about their OS and they're not likely to do a hard turn for different OS just for the sake of "understanding UNIX", as it would require quite large effort to just get to the point where they are already with Linux. Solaris would need to provide a really compelling function or feature to justify this effort, and all you said is "portability" (which most probably won't be necessary in most cases) and vague notion of "understanding UNIX".

EDIT: It was a little dishonest on my side to suggest that main selling points of Solaris derivatives were "understanding UNIX" and "writing portabile code". Solaris introduced ZFS and DTrace. But while it is said that ZFS with zones work well together, it still needs explaining why is it a significantly better combination than what Linux provides, and DTrace seems like a very distant benefit, less compelling each time a tool like Sysdig shows up.

Re:Understanding and Hardening Linux Containers [pdf] by Annatar on Monday April 25, 2016 @09:29AM (#11564008)
> They already have some knowledge about their OS and they're not likely to do a hard turn for different OS just for the sake of "understanding UNIX", as it would require quite large effort to just get to the point where they are already with Linux.

How about no priority 1 incidents at 02:03 in the morning, resulting in sleepless nights until 07:30 - 08:30 every day during one's on call, wasted on stupid problems like ext3 filesystem corruption caused by lost writes (again: http://danluu.com/file-consistency/)? Is that a compelling enough reason?

No incidents at dead hours of the night. A programming environment fully supporting standards. Same software which runs on Linux. Guaranteed application binary interface backward compatibility: compile on oldest SunOS 5.# you can find, run on all the versions newer than that one. Change schedulers on the fly, as you need them. Realtime subsystem. kdb and mdb: the best post mortem analysis tools in the industry (what are those system administrators going to do when the kernel crashes so bad that it locks up and doesn't dump a stack trace?) Full security isolation and bare metal performance with zones.

Exhaustive documentation ("he who can read is clearly at an advantage"), much of it applicable to illumos / SmartOS:

http://docs.oracle.com/cd/E19253-01/817-1985/

Should I go on?

Re:Understanding and Hardening Linux Containers [pdf] by dozzie on Monday April 25, 2016 @11:35AM (#11564875)
> How about no priority 1 incidents at 02:03 in the morning, resulting in sleepless nights until 07:30 - 08:30 every day during one's on call, wasted on stupid problems like ext3 filesystem corruption caused by lost writes (again: http://danluu.com/file-consistency/)? Is that a compelling enough reason?

I won't have calls about corrupted ext3, because I use XFS, so not much of a point.

"No priority 1 incidents" is not a function of an operating system, so it can't be a compelling reason to switch. You would still need to elaborate why exactly Linux doesn't work, especially when it works well enough in most admins' perception.

> A programming environment fully supporting standards.

Becasue Linux doesn't support a very important standard of... Could you help me here? Nothing comes to my mind.

> Same software which runs on Linux.

Sweet. I should learn Solaris, so I can run the same software as I currently can on Linux.

Do you hear how ridiculous this point is?

> Guaranteed application binary interface backward compatibility: compile on oldest SunOS 5. you can find, run on all the versions newer than that one.

Only ABI of kernel and (probably) OS-supplied libraries, which is not that much when it comes to what most of the companies run.

> Change schedulers on the fly, as you need them. Realtime subsystem.

Of course, because Linux doesn't allow to change schedulers and has no realtime subsystem.

> kdb and mdb: the best post mortem analysis tools in the industry (what are those system administrators going to do when the kernel crashes so bad that it locks up and doesn't dump a stack trace?)

This is a very distant benefit. One needs to be accustomed with kernel-level code in the first place to make any use of these.

> Full security isolation and bare metal performance with zones.

You get bare metal performance with Linux containers as well, so not a strong point. Only the "full security isolation" could be a sensible reason, but then, it should be proven by close scrutiny. I haven't heard too much about people analyzing Solaris zones, so I'm not likely to be convinced out of thin air that this argument applies.

> Exhaustive documentation ("he who can read is clearly at an advantage"), much of it applicable to illumos / SmartOS:

> http://docs.oracle.com/cd/E19253-01/817-1985/

OK, let's go to booting chapter: http://docs.oracle.com/cd/E19253-01/817-1985/hboverview-2546...

This is barely enough for a trained sysadmin to learn how booting works and where to nudge when things go awry or how to do less standard tasks (e.g. I'm not sure how much time would it take me if I needed to build a custom live CD image; probably quite a lot, judging from boot(1M) man page).

Even if it's exhaustive, it suffers from not having much between step-by-step-and-don't-think instructions and deep documentation that requires prior understanding of the system.

> Should I go on?

You haven't provided many arguments that would be sensible reasons (let alone being strong reasons) to invest in migrating to Solaris.

I think we should disengage from this discussion. If I was in (what I believe is) your state of mind, I couldn't muster any good argument that would convince my interlocutor.

Re:Understanding and Hardening Linux Containers [pdf] by dyn on Thursday April 21, 2016 @08:52AM (#11541670)
Hi author here.

Thanks! It was a lot of hard work, I'm glad it's being well received and seems like people are generally finding it useful- the main point of writing it.

I would have liked to talk a lot more about other non-Linux solutions, but that's where most of the NCC Group consulting work has been, which was a main driver for why this info needs to be out there (companies not understanding container security or not "turning it up to 11" in places it needs to be). My background is also mainly Linux, so I stuck to what I know.

Hopefully someone else will release a similar paper on non-Linux solutions/unikernels/hybrid platforms?

Re:Understanding and Hardening Linux Containers [pdf] by heroprotagonist on Thursday April 21, 2016 @12:15AM (#11539658)
I see nothing wrong with this. I think it's good to be specific. The Linux implementation is different from many other containerization solutions (eg, Solaris Zones, AIX WPAR, HPUX partitions, etc, so forth). The scope of "all container solutions for every platform" is too broad to be able to provide much depth, and the audience of people interested in container information for all platforms instead of a specific platform is fairly narrow.

There's plenty of room for someone to make a document specific to other platforms for a different audience. There's even room for someone to make a comparison of all platforms. It depends on your goal, your knowledge, and your intended audience.

Re:Understanding and Hardening Linux Containers [pdf] by X86BSD on Thursday April 21, 2016 @12:25AM (#11539689)
It's not that the paper is wrong for only focusing on Linux "containers".

It's that the paper, while describing the security short comings of the various solutions, illustrates the echo chamber I refer to.

"We have these security issues, that others do not have, but we are not going to look at how they solved these issues because, well, Linux!"

The paper, IMO, would have better served the audience had it included ideas and/or solutions people outside Linux-land developed to solve some of these issues that were written before Linux even attempted a "container".

You are free to disagree.

Re:Understanding and Hardening Linux Containers [pdf] by Scarbutt on Thursday April 21, 2016 @02:24AM (#11540039)
and how do you know the others solutions are really secure? Linux is going to get the most eyes, critics, scrutiny on this because is way more widely used than the other platforms that provide similar technologies.
Re:Understanding and Hardening Linux Containers [pdf] by AdieuToLogic on Thursday April 21, 2016 @03:59AM (#11540322)

  and how do you know the others solutions
  are really secure? Linux is going to get
  the most eyes, critics, scrutiny on this
  because is way more widely used ...
This is commonly known as the "given enough eyeballs" fallacy[1]. While it might help when more people have an opportunity to review security critical subsystems, a given OS' popularity certainly does not guarantee this. Heartbleed[2] is often cited as an exemplar for this very situation.

1 - http://www.amazon.com/Facts-Fallacies-Software-Engineering-R...

2 - https://en.wikipedia.org/wiki/Heartbleed

Re:Understanding and Hardening Linux Containers [pdf] by dap on Thursday April 21, 2016 @10:36AM (#11542448)
Because not only did the engineers who built it consider how the architecture would ensure security, they wrote all of the considerations and design choices down.[1] Even if they were completely wrong about everything, there's no reason to simply ignore all the thought that's been put into this problem.

[1] https://us-east.manta.joyent.com/bcantrill/public/ppwl-cantr...

Re:Understanding and Hardening Linux Containers [pdf] by i4k on Thursday April 21, 2016 @01:53PM (#11544040)
Linux being secure is a fallacy too.

Spender talks about it for a long time: https://grsecurity.net/~spender/interview_notes.txt

Linux developers fix security bugs in irresponsible ways, saying nothing about the attack vector or CVE's involved. Sometimes multiple vulnerabilities are addressed in the same patch, making hard to track the fix.

For more information see the exploit below: https://grsecurity.net/~spender/exploits/64bit_dos.c

The list of kernel vulnerabilities involving namespaces is bigger each day, as the paper shows, but the industry still uses "container" and "security" in the same quotes.

Re:Understanding and Hardening Linux Containers [pdf] by pyvpx on Thursday April 21, 2016 @04:25AM (#11540416)
other than something like seL4, no one _knows_ what is really secure. however it would be hard to argue that a decade+ of container systems such as FreeBSD Jail and Solaris' functional equivalent (the precise name escapes me) don't have useful lessons "Linux can learn from".
Re:Understanding and Hardening Linux Containers [pdf] by vacri on Thursday April 21, 2016 @03:27AM (#11540223)
> It's just simply frustrating to watch Linux reinvent some wheels, poorly, time after time.

Out of curiosity, why do you think that Linux got such traction and the BSDs didn't? I keep hearing from the BSD camp that the BSDs are better than Linux at, well, pretty much everything, but there must be some reason why BSDs were not as performant for people than Linux distros.

Re:Understanding and Hardening Linux Containers [pdf] by AdieuToLogic on Thursday April 21, 2016 @04:06AM (#11540349)

  Out of curiosity, why do you think that
  Linux got such traction and the BSDs didn't?
The AT&T lawsuit[1] against BSDi. Stymieing BSD in a well publicized lawsuit was enough to put a lot of people off from entertaining an x86 BSD (can't blame anyone for being prudent).

Right around the same time, one Mr. Linus Torvalds was making his own OS.

1 - https://en.wikipedia.org/wiki/UNIX_System_Laboratories,_Inc.....

Re:Understanding and Hardening Linux Containers [pdf] by vacri on Friday April 22, 2016 @02:55AM (#11547479)
Thanks for the info, folks. I was unaware of the lawsuit. Curious.
Re:Understanding and Hardening Linux Containers [pdf] by tachion on Thursday April 21, 2016 @04:18AM (#11540393)
The lawsuit against involving BSD back in the day was one thing, and another one is... spreading FUD. That's about the fact that BSD's haven't got traction. Of course, relatively to Linux its smaller, or, better say, less visible: Sony Play Station is FreeBSD, Netflix uses FreeBSD for content delivery servers, WhatsApp is FreeBSD, OSX is BSD user land and much, much more, simply less visible.
Re:Understanding and Hardening Linux Containers [pdf] by pyvpx on Thursday April 21, 2016 @04:28AM (#11540423)
the first version of Meraki were NetBSD -- same for Apple Airport devices. Junipers management plane has historically been FreeBSD (but that is unfortunately changing)
Re:Understanding and Hardening Linux Containers [pdf] by JamesSwift on Thursday April 21, 2016 @12:17PM (#11543269)
As an ignorant spectator I appreciate the balance provided by your comment. However, I wish you would have provided some concrete examples of which wheels they are reinventing, and how those wheels exist in other solutions.
Re:Understanding and Hardening Linux Containers [pdf] by X86BSD on Thursday April 21, 2016 @12:38PM (#11543427)
Listen to this discussion[0] which highlights the point I am making. I set the marker for the relevant "container" comment but watch the entire thing to understand more fully what I and others are saying and why.

[0] https://youtu.be/Ya6h2zKlpaQ?t=1h8m11s

Re:Understanding and Hardening Linux Containers [pdf] by i4k on Thursday April 21, 2016 @01:06PM (#11543664)
Container technology is grounded on namespaces and cgroups. Both technologies existed long time ago. Linux namespace is a very limited implementation of Plan9's namespace. Solaris and FreeBSD, to cite only the most famous, have resource limits database for process groups.