News for hackers. Stuff that matters.
Re:How We Build Code at Netflix by chei0aiV on Wednesday March 09, 2016 @11:27AM (#11253842)
> Immutable infrastructure is the future

Are there any downsides to immutable infrastructure?

Re:How We Build Code at Netflix by riskable on Wednesday March 09, 2016 @11:59AM (#11254060)
I'm on a team developing an immutable infrastructure solution with (Docker) containers at a large financial institution. The biggest downsides so far:

* It can get complicated very quickly since it involves a lot of stages, systems, and loads of brand new tools where you're unlikely to be able to hire existing talent (you need people who can just read the docs on a complicated tool/architecture like Kubernetes and then start using it).

* Speaking of talent, you need developers who are reasonably well-versed in Linux systems administration. Could your developers write a shell script that can configure every little aspect of a Linux host from packaging to authentication? Your developers also need to understand some hard core topics of TCP/IP (e.g. anycast), firewalls (e.g. port mapping), and DNS (TTLs, views, SRV records, and more). They also need to be well-versed in security (e.g. the impact of running as root inside a container) and especially authentication (e.g. HMAC, securely storing shared secrets) and encryption (SSL, managing certificates, knowing which hashing algorithm to use)!

* Dealing with existing bureaucracy. Especially rules and policies in regards to security. For example, some people will think that containers and immutable VMs should be treated like regular hosts. That can mean waiting hours for an inventory system to acknowledge their existence before you can put them into production. Or having to wait 30 minutes after every container/VM is brought up so that a security tool can perform a scan even though each container/VM would be identical in every way and have already been scanned before production deployment. Sigh.

Of course, a lot of that depends on where you work. At my work right now the biggest challenge is the bureaucracy for certain. If I can get containers classified differently than VMs then all my policy/BAU problems will (likely) go away. If I can't get that then, really, there's no point to our efforts since there would be no advantage to end users (the consumers of what we're building).

I may be able to live with a pointless, half-hour-long security scan every time we bring up a new container but I absolutely cannot live with an 8-hour-long wait for a container to show up in our system-of-record (inventory system).

Re:How We Build Code at Netflix by jfindley on Wednesday March 09, 2016 @12:26PM (#11254248)
We are doing something very similar (mesos rather than k8s), also in a financial organisation.

Our main issue is also security - as docker containers aren't very good at actually containing, from a security PoV, we're using SELinux MLS, assigning each container a unique SELinux level/category (fedora-atomic sets this up nicely for you if anyone wants to try).

This is great for security, but breaks so many things - lots of the cutting edge work with docker and related tech isn't done by people that care deeply about security, or really understand this sort of usecase, so we run into a huge amount of edgecases and bugs. We spend a lot of time fighting this.

It also makes having a really good build/deploy pipeline essential, as well as a good environment set up to reproduce and diagnose issues.

Re:How We Build Code at Netflix by tracker1 on Wednesday March 09, 2016 @01:54PM (#11254910)
Along a similar grain, I'm curious if anyone's used Joyent's infrastructure stack internally... I know they offer it, but not familiar enough with it. It seems Solaris containers as a base for docker containers is a better security model, but not sure what parts, if enough is open to implement without paying consultation from Joyent to get started even. It's definitely a compelling model.

I am only slightly surprised that there's still a lot of bugginess in this aspect, though I haven't had to deal with that level of security need on the infrastructure side in a number of years. I'd rather be doing app-dev over ops-dev.

Re:How We Build Code at Netflix by seanp2k2 on Wednesday March 09, 2016 @08:28PM (#11256886)
HashiCorp also has a pretty complete solution called Atlas : https://www.hashicorp.com/ ( different from the Netflix Atlas http://techblog.netflix.com/2014/12/introducing-atlas-netfli... )

Joyent's Triton looks cool: https://www.joyent.com/ as does CoreOS's Tectonic (built on top of k8s): https://tectonic.com/

Re:How We Build Code at Netflix by tracker1 on Thursday March 10, 2016 @10:03PM (#11264389)
I've been following CoreOS with great interest, I was just thinking from a security standpoint Joyent's Triton looks amazing... I don't think their pricing is competitive with Google/AWS/Azure though.

Hadn't looked at Atlas... I changed jobs into a larger company, and don't really have to deal with the ops side ever. Though eagerly awaiting a blessed docker based solution to become available in the org... it'll mean that projects can be developed more readily without being limited to the Java based infrastructure in place now.

Re:How We Build Code at Netflix by je42 on Wednesday March 09, 2016 @12:51PM (#11254449)
Do you have links pointing at what docker containers leak from a security PoV ?
Re:How We Build Code at Netflix by riskable on Wednesday March 09, 2016 @01:23PM (#11254652)
This largely depends on how you're implementing them. If you're using Kubernetes, for example, all Docker containers being managed by Kubernetes can talk to each other without restrictions. From each container's perspective they're part of a big, flat network with no firewalls between them.

So if you have a container running the database for client A that means that when client B spins up their little malicious container they can directly connect to client A's DB.

That's one way that Docker containers can "leak" (sort of). Another way is that the filesystem for running containers is directly accessible from outside the container. That means that anyone with the power to start/stop containers has the power to read the filesystems for all other containers.

There used to be a problem with, "root inside the container is the same as root outside the container" but that was fixed with Docker 1.9 via the new UID mapping feature. So the filesystem and processes running as UID 0 inside the container will actually be some high UID in the parent OS. So unless you're running Red Hat's version of Docker (which is currently hard set to be Docker 1.8!) you can upgrade and solve that problem pretty quickly with a few tweaks.

Re:How We Build Code at Netflix by pyvpx on Wednesday March 09, 2016 @12:58PM (#11254488)
I'm a network engineer with a strong interest in container networking -- where does anycast come into play in your immutable infrastructure? I am highly curious!
Re:How We Build Code at Netflix by riskable on Wednesday March 09, 2016 @01:18PM (#11254624)
Right now for us it mostly just DNS using anycast but we may use anycast as part of a (forthcoming) microservices architecture.

The current popular method is to simply query a service discovery resolver of some sort (e.g. https://docs.docker.com/swarm/discovery/) to figure out which server to query, say, the weather (very important in finance! haha =). That's great and necessary if your architecture spans across your internal cloud, Amazon, and cloud-provider-of-the-day but if you have a fixed set of data centers where you know your applications will always be running (say, because of legal reasons =) then you can just use anycast as long as you keep ports consistent.

For most microservices deployments I'd imagine they would want their stuff to be accessible via a global DNS name and if they're going to do that then they might as well take the next step and just standardize on a port as well. If you have a single (global) name and a single port for an API why do you need service discovery (for that)? You don't. You just need to make sure that when clients query your API that they don't wind up using that one server you've got on the other side of the world on a 56k ISDN line (sadly yes, that does happen).

Having said that we may end up using popular service discovery tools anyway because they're convenient and we can use them in horrible, do-as-I-say-not-as-I-do ways to hack legacy applications into containers =D

Re:How We Build Code at Netflix by alistproducer2 on Wednesday March 09, 2016 @01:48PM (#11254868)
Thanks for sharing your insight. Comments like this is why HN is my favorite place on the net.
Re:How We Build Code at Netflix by matt_wulfeck on Wednesday March 09, 2016 @12:14PM (#11254175)
Sounds like Devops.
Re:How We Build Code at Netflix by sytse on Wednesday March 09, 2016 @12:20PM (#11254207)
I think immutable infrastructure still requires a lot of tools and moving parts today. We're spending a lot of time thinking on how to improve this. Two idea's we have:

1. Deploy directly from GitLab to Kubernetes https://gitlab.com/gitlab-org/gitlab-ce/issues/14040 2. Add a container registry to GitLab itself https://gitlab.com/gitlab-org/gitlab-ce/issues/3299

Any comments or suggestions are welcome.

Re:How We Build Code at Netflix by phamilton on Wednesday March 09, 2016 @11:57AM (#11254048)
You need to commit to it. You just toe the line. If you currently depend on being able to SSH into boxes and tail logs you will feel a lot of friction.

However, practices like those tend to be a crutch anyway.

Re:How We Build Code at Netflix by dminor on Wednesday March 09, 2016 @11:31AM (#11253883)
Build wait times invoke 90's nostalgia.
Re:How We Build Code at Netflix by moondev on Wednesday March 09, 2016 @11:29AM (#11253863)
It takes a while to bake the image, and storing thousands of images can add up. However containers can address both points.