News for hackers. Stuff that matters.
Re:Musl 1.2.4 adds TCP DNS fallback by robinhoodexe on Sunday July 30, 2023 @01:38PM (#36933770)
We’re currently consolidating all container images to run on Debian-slim instead of a mixture of Debian, Ubuntu and alpine. Sure, alpine is small, but with 70% of our 500 container images being used for Python, R or node the final image is so large (due to libraries/packages), that the difference between alpine (~30 MB) and debian-slim (~80) is negligible. We’ve been experiencing the weird DNS behaviour of alpine and other issues with musl as well. Debian is rock solid and upgrading to bookworm from bullseye and even buster in many cases didn’t cause any problems at all.

I will admit though, that Debian-slim still has some non-essential stuff that usually isn’t needed at runtime, a shell is still neat for debugging or local development.

Re:Musl 1.2.4 adds TCP DNS fallback by adolph on Sunday July 30, 2023 @01:52PM (#36933969)
> the difference between alpine (~30 MB) and debian-slim (~80)

Given that it's a different layer, the your container runtime isn't going to redownload the layer anyway, right?

Re:Musl 1.2.4 adds TCP DNS fallback by robinhoodexe on Sunday July 30, 2023 @02:02PM (#36934083)
Exactly. Part of the appeal to consolidate all of our container images to use Debian-slim is the ability to optimise the caching of layers, both in our container registry but also on our kubernetes cluster’s nodes (which can be done in a consistent manner with kube-fledged[1]).

[1] https://github.com/senthilrch/kube-fledged

Re:Musl 1.2.4 adds TCP DNS fallback by robertlagrant on Sunday July 30, 2023 @06:16PM (#36936561)
Thanks for that - that operator sounds extremely useful!
Re:Musl 1.2.4 adds TCP DNS fallback by kstrauser on Sunday July 30, 2023 @04:43PM (#36935754)
And even if it did, in an ancient data center that only uses gigabit ethernet, that's only a .5s longer download. And even a $4 DigitalOcean server comes with 10GB of storage, so that 50MB is only 1/200th of the instance's store. (I'd also bet that nearly no one uses instances that tiny for durable production work where 50MB is going to make a difference.)
Re:Musl 1.2.4 adds TCP DNS fallback by leononame on Sunday July 30, 2023 @02:15PM (#36934251)
Can you point me on where to look for more details on securing a container? I'm a developer myself, and for me, the main benefit of containers is being able to deploy the app myself easily because I can bundle all the dependencies.

What would you suggest I restrict at runtime and can you point me to a tutorial or an article where I can go have a deeper read on how to do it?

Re:Musl 1.2.4 adds TCP DNS fallback by robinhoodexe on Sunday July 30, 2023 @02:36PM (#36934476)
What you want to read is the kubernetes pod security context fields[1].

In your Dockerfile, add a non-root user with UID and GID 1000, then in the end of your Dockerfile, right before a CMD or ENTRYPOINT you change to that user.

In your kubernetes yaml manifest, you can now set runAsNonRoot to true and runAsUser & runAsGroup to 1000.

Then there’s the privileged and allowPrivilegeEscalation fields, these can nearly always be set to false unless you actually need the extra privileges (such as using a GPU) on a shared node.

Then there’s seccomp profiles and the system capabilities. If you can run your container as the non-root user you’ve created, and you don’t need the extra privileges then these can safely also be set to the most restricted. Non-privileged non-root is the same as all capabilities are dropped.

The tricky one is the readOnlyRootfilesystem field. This includes /tmp, which is considered a global writeable directory, so the workaround is to make a in-memory volume and mount it at /tmp to make it writable. Likewise, your $HOME/.cache and $HOME/.local directories (for the user you created in your Dockerfile) are usually used by third party packages, so creating mounts here can be useful as well (if for some reason you can’t point it to /tmp instead).

[1] https://kubernetes.io/docs/tasks/configure-pod-container/sec...

Re:Musl 1.2.4 adds TCP DNS fallback by lenova on Monday July 31, 2023 @03:15AM (#36939630)
> In your Dockerfile, add a non-root user with UID and GID 1000, then in the end of your Dockerfile, right before a CMD or ENTRYPOINT you change to that user.

I'm not certain, but would it be more secure to create a non-root user in the Dockerfile, but _don't_ use UID 1000. Use volume mounts to grant host disk access instead.

Re:Musl 1.2.4 adds TCP DNS fallback by robinhoodexe on Monday July 31, 2023 @03:38AM (#36939750)
If you want to use the runAsUser and runAsGroup options in kubernetes that user needs to exist, hence why creating it is needed.

In our case, we have our own base container images that include this user. The base images are either just plain copies of official ones (such as dotnet or curl) or also includes stuff that almost everyone needs (such as MSSQL ODBC drivers, libxml2, curl etc) in the case of python/R/node.

Re:Musl 1.2.4 adds TCP DNS fallback by 4oo4 on Sunday July 30, 2023 @03:17PM (#36934888)
Re:Musl 1.2.4 adds TCP DNS fallback by vbezhenar on Sunday July 30, 2023 @03:45PM (#36935174)
Basically you need to work as unprivileged user and with immutable file system (of course you can have ephemeral /tmp or persistent /data, but generally the entire system should be treated as read-only).
Re:Musl 1.2.4 adds TCP DNS fallback by galangalalgol on Sunday July 30, 2023 @02:23PM (#36934336)
I've run into the same thing for large dev images, but using pure rust often means that musl allows for a single executable and a config file in a from scratch container for deployment. In cases where a slab or bump allocator are used, musl's deficiencies seem minimized.

That means duplication of musl in lots of containers, but when they are all less than 10MB its less of an issue. Linking against gnu libraries might get the same executable down to less than 2MB but you'll add that back and more for even the tiniest gnu nase images.

Re:Musl 1.2.4 adds TCP DNS fallback by synergy20 on Sunday July 30, 2023 @05:00PM (#36935921)
what is pure rust,rust needs its own libraries which is a few MB as I recall
Re:Musl 1.2.4 adds TCP DNS fallback by galangalalgol on Sunday July 30, 2023 @05:20PM (#36936102)
By pure, I just meant no dependencies on c or c++ bindings other than libc. If that is the case you can do a musl build that has no dynamic dependencies, as all rust dependencies are static. So then your only dependency is the kernel, which is provided via podman/docker. A decent sized rust program with hundreds of dependencies I can get to compile down to 1.5MB. But that is depending on gnu. So if you had 4 or 5 of those on a node, it might be less data to use one gnu base image that is really small like rhel micro, and build rust for gnu. But if you have cpu hungry services like I do, then you usually have only a couple per node, so from scratch musl can be a bit smaller.
Re:Musl 1.2.4 adds TCP DNS fallback by phamilton on Sunday July 30, 2023 @06:50PM (#36936808)
> cpu hungry services

Have you benchmarked musl vs glibc in any way? Data I've seen is all over the place and in curious about your experience.

Re:Musl 1.2.4 adds TCP DNS fallback by galangalalgol on Sunday July 30, 2023 @09:35PM (#36937919)
Very few system calls in the main loop. Not even close to being io bound, and as I said memory is all preallocated, so allocator efficiency isn't a factor.
Re:Musl 1.2.4 adds TCP DNS fallback by jonwest on Sunday July 30, 2023 @03:18PM (#36934894)
In the same boat here as well. Especially when you’re talking about container images using JavaScript or other interpreted languages that are bundling in a bunch of other dependencies, the developer experience is much better in my experience given that more developers are likely to have had experience working in a Debian based distro than an Alpine based one.

Especially when you’re also developing within the container as well, having that be unified is absolutely worth the convenience, and honestly security and reliability as well. I realize that a container with less installed on it is inherently more secure, but if the only people who are familiar with the system are a small infrastructure/platform/ops type of team, things are more likely to get missed.

Re:Musl 1.2.4 adds TCP DNS fallback by kachnuv_ocasek on Sunday July 30, 2023 @03:37PM (#36935074)
Do you have any tips regarding building R-based container images?
Re:Musl 1.2.4 adds TCP DNS fallback by robinhoodexe on Sunday July 30, 2023 @04:17PM (#36935496)
R is kinda difficult and I haven’t cracked this one. Currently we’re using the rocker based ones[1] but they are based on Ubuntu and include a lot of stuff we don’t need at runtime. I’ll look into creating a more minimal R base images that’s based on Debian-slim.

[1] https://github.com/rocker-org/rocker-versioned2

Re:Musl 1.2.4 adds TCP DNS fallback by Rapzid on Sunday July 30, 2023 @03:56PM (#36935275)
This here. Honestly most orgs with uhh.. Let's say a more mature sense of ROI tradeoffs were doing this from pretty much the very beginning.

Also, Ubuntu 22.04 is only 28.17MB compressed right now so it looks equiv to debian-slim. There are also these new image lines, I can't recall the funky name for them, that are even smaller.

Re:Musl 1.2.4 adds TCP DNS fallback by kstrauser on Sunday July 30, 2023 @04:36PM (#36935681)
I'm pushing to go back to Debian from Ubuntu. Canonical's making decisions lately that don't appeal to me, and especially on the server I don't see a clear advantage for Ubuntu vs good ol' rock solid Debian.
Re:Musl 1.2.4 adds TCP DNS fallback by 3np on Monday July 31, 2023 @05:23AM (#36940318)
Host OS vs container image base shouldn't be the same conversation.
Re:Musl 1.2.4 adds TCP DNS fallback by Fnoord on Monday July 31, 2023 @07:00AM (#36940905)
I've done this multiple times when new Debian Stable was released (Bookworm just got released). Its great, until it isn't (ie. you need more up-to-date software). Although to be fair, you might be comparing Ubuntu LTS with Debian releases and the minor Ubuntu releases get short support. Ubuntu is akin to a polished Debian Testing snapshot. I don't like them swapping to Snap either (snappy it ain't).
Re:Musl 1.2.4 adds TCP DNS fallback by synergy20 on Sunday July 30, 2023 @04:58PM (#36935902)
why is it so much smaller than debian slim?
Re:Musl 1.2.4 adds TCP DNS fallback by yjftsjthsd-h on Sunday July 30, 2023 @05:08PM (#36935990)
Fewer features (including less "we've supported doing that for 20 years and we're not cutting it now"), packages separated into parts (ex. where debian might separate "foo" into "foo" for the main package and "foo-dev" for headers, alpine will also break out "foo-doc" with manpages and such), general emphasis on being small rather than full-featured.
Re:Musl 1.2.4 adds TCP DNS fallback by Rapzid on Sunday July 30, 2023 @05:01PM (#36935924)
It's not, Debian slim is ~28.8MB compressed.
Re:Musl 1.2.4 adds TCP DNS fallback by ploxiln on Sunday July 30, 2023 @09:06PM (#36937790)
If you want to compare compressed size, then alpine is 3.25MB
Re:Musl 1.2.4 adds TCP DNS fallback by derkades on Monday July 31, 2023 @07:26AM (#36941073)
The parent comment was comparing Ubuntu and Debian-slim
Re:Musl 1.2.4 adds TCP DNS fallback by senknvd on Sunday July 30, 2023 @05:44PM (#36936332)
> There are also these new image lines, I can't recall the funky name for them, that are even smaller.

You might be thinking of the chiselled images. An interesting idea but very much incomplete[1].

[1]: https://github.com/canonical/chisel-releases/issues/34

Re:Musl 1.2.4 adds TCP DNS fallback by Rapzid on Sunday July 30, 2023 @11:53PM (#36938636)
Yes, that's it! I've been working with .Net a lot recently and recalled Microsoft's announcement about them. Specialized runtime images for JRE, .NEt, Asp.Net, and etc.
Re:Musl 1.2.4 adds TCP DNS fallback by nickjj on Sunday July 30, 2023 @06:57PM (#36936879)
I made the switch too around 4ish years ago. It has worked out nicely and I have no intention on moving away from Debian Slim. Everything "just works" and you get to re-use any previous Debian knowledge you may have picked up before using Docker.
Re:Musl 1.2.4 adds TCP DNS fallback by kelvie on Sunday July 30, 2023 @07:14PM (#36936993)
My biggest beef with the apt/deb-based distros for container images is that apt-get updates installs take frustratingly long, whereas apks always tend to be near instant. I wonder what it is in the postinst and related scripts that just take so long and can't be parallelized!

Most of the reason I switched from Ubuntu -> Arch is that working with alpine allowed me to realize that installing packages and their dependencies don't have to take so long.

Re:Musl 1.2.4 adds TCP DNS fallback by p_l on Sunday July 30, 2023 @07:59PM (#36937358)
Have you tried building the images with eatmydata library?

Essentially, dpkg is hilariously slow due to its approach in issuing fsync calls for every file - eatmydata wrapper injects a shared library which turns that into a no-op.

I haven't timed it rigorously, but I experienced up to 10x speedup in container builds by just wrapping dpkg with eatmydata.

Re:Musl 1.2.4 adds TCP DNS fallback by gorgoiler on Monday July 31, 2023 @12:03AM (#36938695)
[Edit: modern debian images have “force-unsafe-io” turned on in /etc/dpkg/ so eatmydata shouldn’t make a difference.]

I’m not the OP, but in an example I just tried it took build-essential from 48s to 39s (timed on the third step, without and then with the call to eatmydata):

  from debian:sid
  run apt update && env DEBIAN_FRONTEND=noninteractive apt-get install --yes eatmydata
  run apt update && eatmydata env DEBIAN_FRONTEND=noninteractive apt-get install --yes build-essential
Can you replicate something faster? Was I doing something wrong?
Re:Musl 1.2.4 adds TCP DNS fallback by TylerE on Monday July 31, 2023 @12:37AM (#36938858)
Yes, the second invocation will have hot caches. Basically the standard is to either test in a clean environment (e.g. immediately after a reboot) or to run the command until the times stabilize, and then take, say, the middle 3 out of 5 measurements.
Re:Musl 1.2.4 adds TCP DNS fallback by gorgoiler on Monday July 31, 2023 @01:06AM (#36939003)
What is cached though? DNS? IP routes to the Debian mirrors?

I ran it three more times. Plain install was 50s 48s 47s; eatmydata was 46s 46s 46s.

Installing build-essential probably does enough non-dpkg IO to account for the difference (install scripts etc, for example when libc changes and the system scans for running daemons) but it isn’t much and the delta may well be statistically insignificant.

Re:Musl 1.2.4 adds TCP DNS fallback by TylerE on Monday July 31, 2023 @01:11AM (#36939028)
File system. Drives cache recently read blocks. This isn’t something that’s neesesarily e en visible to the OS.
Re:Musl 1.2.4 adds TCP DNS fallback by lenova on Monday July 31, 2023 @02:37AM (#36939454)
I decided to benchmark these scenarios out of curiosity, because I hadn't heard of eatmydata before or of force-unsafe-io.

- Docker image: `debian:stable-slim`

- `docker system prune -a` ran before each test.

- Packages installed: `build-essential ca-certificates libssl-dev software-properties-common wget curl git`

Results:

Default Debian settings: 48.32 seconds

force-unsafe-io enabled: 55.86 seconds

eatMyData enabled: 35.04 seconds

Re:Musl 1.2.4 adds TCP DNS fallback by p_l on Monday July 31, 2023 @03:30PM (#36947607)
It's possible that the last time I dealt with that on debian force-unsafe-io wasn't set in /etc/dpkg.

I haven't really built debian-based containers in last two years.

Re:Musl 1.2.4 adds TCP DNS fallback by amouat on Monday July 31, 2023 @08:03AM (#36941375)
You might want to check out Wolfi and Chainguard Images. Wolfi is a Linux distro that we use to build minimal images that are roughly comparable to Alpine in size but, everything is compiled from source against glibc.

Our images come without a shell or pacakge manager by default, but there are -dev variants that include these.

https://github.com/wolfi-dev/ https://github.com/chainguard-images/images

Re:Musl 1.2.4 adds TCP DNS fallback by kapilvt on Monday July 31, 2023 @10:23AM (#36942968)
I was using wolfi for an oss project, but the recent attempts at forcing payments when pinning version was a huge red flag, as was the messaging around it, and I’ll be migrating away from it.

https://www.chainguard.dev/unchained/scaling-chainguard-imag...

Pinning a language version (say python 3.11) isn’t an optional thing its a best practice, and the notion that its because of security seems intentionally misleading as the images should be refreshed in place on the tag along with signatures.

Re:Musl 1.2.4 adds TCP DNS fallback by amouat on Monday July 31, 2023 @02:12PM (#36946581)
I'm very sorry that we broke things for you.

To be clear, nothing has changed with Wolfi. Wolfi is an open source community project and everything is still available there: https://github.com/wolfi-dev/.

We have made changes to Chainguard Images - our commercial product built on top of Wolfi - which mean you can no longer pull images by tag (other than latest). Chainguard images are rebuilt everyday and have a not inconsiderable maintenance cost (and the money we make here directly helps us support Wolfi).

The easiest way to avoid this is to build the images yourself. You can rebuild identical images to ours using apko and the source files in the images repo e.g: https://github.com/chainguard-images/images/blob/main/images... (note you can replace package names with versioned versions). You can also just use a Dockerfile with the wolfi-base image to "apk add" packages. Full details are here: https://www.chainguard.dev/unchained/a-guide-on-how-to-use-c...

I agree that pinning is a best practice. The above blog explains that you can still do it using a digest, but I accept this isn't the simplest solution.

If I can help any more, please feel free to get in touch - you can find me most places including twitter https://twitter.com/adrianmouat

Re:Musl 1.2.4 adds TCP DNS fallback by kapilvt on Monday July 31, 2023 @02:43PM (#36947041)
I sympathize, but if wolfi is oss, please update docs and downloads to use a separate registry then your commercial one for distribution.

else, frankly you claimed wolfi is oss, get customers to use your registry, and then bait and switched your early adopters.

aka, major version upgrades at random, have fun!

Re:Musl 1.2.4 adds TCP DNS fallback by amouat on Monday July 31, 2023 @03:07PM (#36947343)
Wolfi packages are served from https://packages.wolfi.dev/os which can be used with apk tools or apko.

The built Chainguard images are all on the cgr.dev registry.

Wolfi is completely OSS.

The policies regarding Chainguard Images have changed over time, so if there are docs that don't properly reflect this, please let me know and I'll get them updated.

Re:Musl 1.2.4 adds TCP DNS fallback by kapilvt on Tuesday August 01, 2023 @06:44AM (#36954483)
let me rephrase that, wolfi is an oss container image distro without an image, you get to pay us for the image. but here's a our tooling cause we like to call it oss, have fun building. either have those oss images on a non commercial registry, or let's stop misleading folks about what this is, aka pay to play/use oci images. that all wolfi docs / blog posts reference a commercial registry while touting it as oss, is pretty unethical imo, as its the definition of bait and switch.