"distroless" is just Debian packages. Their self-description is fairly annoyingly misleading, since they don't mention that they are just using packages from Debian.

Why is that a bad thing? Binaries are binaries, whether you copied from a deb package or completely built from source code (assuming reproducible build, which Debian supports), they are the same.

It's absolutely not a bad thing, indeed I think it's a good thing to use binaries from Debian. I just think the name of the project is strongly misleading. "distroless" is built from the Debian distribution, not something new made from scratch.

Not sure how widespread that view is, but that's never the association I would have made.

"Xless" is just a different way of saying "without X". "fearless" = "without fear"; "serverless" = "without a server" (yes, I know, not really); so "distroless" = "(comes) without a distribution"

So in my opinon the name correctly suggests that it's "just the package" and comes "without a distro", and not "was built without the help of a distribution".

> Binaries are binaries, whether you copied from a deb package or completely built from source code

No, they are not.

Debian packages are deployable packages, which are built around packages and services and conventions adopted and provided by the target distribution.

it should probably be called "package-manager-less" because there's no package manager in the final build, but there's also no ls, etc, so distroless kinda makes sense. Maybe systemless?

For statically linked binaries, why wouldn't you use the SCRATCH (0 kb) 'image'?

You would. Except most statically linked binaries may still need ca-certificates, tzdata, and some other files that libraries expect it to be present on the system.

Not to mention, you still need runtimes if you are programming in Java, Python or many other languages. https://github.com/GoogleContainerTools/distroless project gives a way to have these runtimes + their lib dependencies while still maintaining a minimal attack surface.

For almost any serious job running in production, you might need CA certificates and openssl.

Unless you're behind a load balancer which terminates TLS and the traffic you deal with is purely http.

Please don't do this anymore. End-to-end encryption is extremely easy to set up and maintain. P2PE will absolutely lull you into a false sense of security.

Everyone seems to think E2E encryption is needed everywhere (I know because the security guys at work think it is needed everywhere, even for everything inside a VPC), but even AWS here is advertising the fact that you don't need to do this:

https://aws.amazon.com/blogs/aws/new-tls-termination-for-net...

>Today we are simplifying the process of building secure web applications by giving you the ability to make use of TLS (Transport Layer Security) connections that terminate at a Network Load Balancer (you can think of TLS as providing the “S” in HTTPS). This will free your backend servers from the compute-intensive work of encrypting and decrypting all of your traffic, while also giving you a host of other features and benefits:

If you trust Amazon to know your risk profile better than your security people, you have a management problem of some sort.

I trust myself who setup our infrastructure vs. the security guys who's automatic response to everything is deny all everywhere, encrypt everything everywhere (at rest encryption isn't enough, what can you do to get the db to work on the data encrypted internally 100% of the time?), and enable 2 factor on everything (the github gui has 2fa enabled, why aren't push/pull requests using 2fa?).

I think may main point is that kneejerk reactions to satisfy a security list checkbox are just as useless as a default "encrypt everything everywhere" stance must be better.

Sounds like you need better security people.

...which, sadly, is true of many larger organizations: "security" ends up as just another middle management approval committee whose only job is to apply byzantine security checklists dreamed up by some Certified Security Architect (tm) way too late in the development process, right when it's hardest for product teams to reshuffle their entire architecture to comply, and with no consideration to the actual circumstances / risk profile of specific projects.

IMHO this should be viewed as a big, glaring anti-pattern, as it fundamentally puts security team goals at odds with product team goals.

Agreed.

...Which sounds like a management problem.

Agreed.

but without tls amazon can "decrypt" your traffic and see whats inside. its one thing to have a backdoor inside a server that they rent to you that would have to be actively exploited and another to passively clone the traffic and analyze it in the name of making the service better.

If you believe that amazon are potentially an adversary, but you still want to host it on their servers, there is essentially nothing you can do to stop them getting at your data. At some point, to process the data, you have to do that unencrypted. That is an unpluggable achilles heel.

Why do they have to actively exploit hardware/vms that they own? Isn't it pretty trivial for a hypervisor to "passively clone" data right out of the memory of the VM? Or to use management interfaces/custom peripherals to exfiltrate data if it were bare metal? AWS is kind of a black box to me but it seems hopeless to try to protect data from people that physically control the systems.

i was especially talking about passively mirroring/analysing network traffic. afak there is no easy and trivial way to "passively clone" aka dump memory of the hypervisor all the time without it being detectable in slowdowns and so on. my concern was not that i need to protect myself from amazon for the fear that they will hack my server, but to the way that they can get insight into my customers, maybe get a snippet of the data i get and so on.

we once saw this from some other company where they noticed we where talking to the competitors and wanted to talk.

Last time I checked, mTLS incurred significant performance penalties and required significant soak testing to ensure that performance would be acceptable for a given application. If you're a small company, you have much lower hanging fruit to chase.

In my understanding there's additional overhead at handshake, but after that the performance is basically identical. The client certificate mostly acts to identify the client to the server, but otherwise the business of picking session keys etc is the same. At this point TLS overhead is close to free.

I think the start of this thread was a plea not to terminate HTTPS at the edge, but instead to plumb it all the way to the serving container. That's unlikely to be mTLS in any case.

What's an extremely easy solution to set up and maintain automated certificate signing and provisioning?

CloudFlare's cfssl: http://blog.cloudflare.com/introducing-cfssl

It does not matter how easy it is, what is the security threat you are mitigating with E2E encryption? In a large scale system it is often not trivial to build a proper E2E, far from being impossible though.

Eh, don't teach my about the systems I run. I'd love to run TLS end to end but in this one? Nah, not worth it.

"Don't teach me about ..." -- aren't we all here to learn? Let's keep the tone civil and assume the best.

I work at $CORP. I don't trust my enterprise IT department with unencrypted traffic for fear of falling victim to stupid traffic shaping or deep packet inspection intrusion prevention going haywire.

Good for you. In my current gig the trade-off is different. I don't work for Google.

> and the traffic you deal with is purely http.

Which is a truly rare case as many backend APIs these days are mandatory secured by HTTPS (or LDAPS, SMTPS, IMAPS to name a couple other openssl-based secure protocols).

sure, as long as you don't connect to ANY outside url's for anything

Which you probably shouldn't be doing for most of your services anyway.

I don't remember the last project I did which didn't have some sort of integration with an external service. I guess if you use microservices, most components won't need it; I mostly use monoliths.

I guess if everything is in a single service you're bound to have some sort of outbound connection at some point. Although, in my experience, you can go very far within your vpc.

I wonder if this would be important with service mesh and mutual tls...

Service meshes make in-cluster mTLS a more-or-less automatic feature, which is worth having. Some will terminate TLS at ingress and convert to mTLS internally. The argument above is that you shouldn't do this and should instead plumb that ingressing TLS traffic all the way to the container.

The downside of plumbing directly to the container is that you lose many of the routing features of a service mesh. If it can't inspect the traffic, it can't do layer 7 routing. It can only route and shape at layer 4.

This has nothing to do with being a LB, if you need to do outgoing calls with https you most likely need ca-certificates.

With static linking your binary would contain openssl.

Not the certificates though, unless you do some special tricks. I think GP is probably thinking of kerberos/NSS, which has a plugin system that requires dynamic linking.

There are two basic flavors of distroless image, one is base, the other one is static.

The `static` image is very bare minimal and only contains things like nsswitch.conf and ca certificates. This is the recommended base image for statically linked languages.

There's also a `base` image that I usually use as the base image of C programs, e.g. stunnel/unbound DNS. In those cases, I usually use Debian Stretch as build environment (distroless uses binary from Debian stable, so ABI is compatible), build a dynamically linked binary, then copy the result binary along with all dynamic object files to a distroless base image.

So when I talk about openssl in the image, I was referring to the `base` flavor. If you are happy with the provided openssl version, the openssl in the base image is indeed useful.

You can easily use multi stage docker files to copy the certs.

You might want at least a shell in the container for debugging?

Distroless has debug images for this purpose: https://github.com/GoogleContainerTools/distroless/blob/mast...

Adding a shell seems antithetical to deploying production code as a static-linked binary, not to mention an expansion of the attack surface of the container.

Without a shell, how does one debug if anything goes wrong?

You can start a container with a shell that shares the PID and network namespaces of the container you want to debug.

Reading logs/traces on your log aggregation service and reproducing in a dev system?

How do you debug in the dev env without a shell?

from the host system, containers don't exist in a vacuum

With remote debugging?

remote debugging is a shell

not necessarily. e.g. java runtimes can expose debugging ports when needed that operate on a custom protocol.

or you can just build gdb into the container and run the process under gdb, then attach to the tty.

or you can debug from the host system where the container's pid namespace is a descendant of the root namespace and the other namespaces can be accessed via /proc or unshare.

What I meant is having a remote debugger is as good as having a remote shell in terms of remote code execution.

Debugging is about when the difference between theory and practice breaks down.

You can use nsenter

Depends. Normally I would do with golang apps. But if I need to debug an issue I'll redeploy them with Ubuntu underneath so I can use the debugging tools.

If your binary is static, why do you need a container at all?

So you can justify using k8s and shine up your resume!

It’s the new 3x SQLServer + 2x IIS + 2x SharePoint cluster to serve an intranet for 100 staff when a single <insert oss wiki on linux> would be more than sufficient.

For Python I'd also highly recommend Clear Linux for raw performance. It's not quite as easy to get started as with something like Ubuntu though.

Checked it out, is Intel supported with compiler tweaks to get the most out of their CPUs, for libs like math, pandas, etc.

Clear Linux is definitely an interesting project, however, using base image with 'latest' tag (only tag existing for https://hub.docker.com/r/clearlinux/python/tags) in the production is not a best strategy as the breaking changes can arrive anytime

docker pull clearlinux/python

https://github.com/clearlinux/dockerfiles/tree/master/python

If you want small images, why not use a tool like https://github.com/docker-slim/docker-slim Then it doesn’t matter which distro you favor?

Wow, I've never heard of this before - I'm looking forward to seeing how much this shrinks my images!

Search for "awesome-docker" and make sure you have some free time. :)

I don't quite get it. If you've somehow statically compiled all your dependencies, shouldn't that just run without a container?

Perhaps the point is not to enable program execution, but to make use of the benefits that may come with container orchestration.

Well at the very least running in a container gives you filesystem and network and PID space isolation, optionally also user namespace isolation.