News for hackers. Stuff that matters.
Re:VSCode, Dev Containers and Docker by dhruvrrp on Thursday January 21, 2021 @10:48AM (#25860092)
Slightly off-topic, it really irks me when people take about reproducible builds/environments while using docker, while having something along the lines of apt-get update/pip install in their dockerfile.

Like that completely destroys the reproducibility of your container!

Re:VSCode, Dev Containers and Docker by mumblemumble on Thursday January 21, 2021 @10:57AM (#25860195)
I can't speak for others, but, while we don't pin versions in the Dockerfiles for our build container images, we do version the images themselves.

Reproducibility is a continuum, not a binary. We've chosen a point on that continuum that we believe gives us the best trade-off between reliability and maintenance effort. 100% from-the-ground-up reproducibility would be ideal, of course, but there's a cost-benefit tradeoff, and we're not being paid to be perfectionists.

Re:VSCode, Dev Containers and Docker by hctaw on Thursday January 21, 2021 @11:05AM (#25860310)
I mean in a sense, the fucked up behavior of system package managers relative to the actual concerns of building and distributing software to the masses is the reason we need Docker to sandbox environments in the first place. It's 2021, there is next to no reason why the default behavior is installation to /usr/lib with shared objects that are rarely shared, with global access when installed for one application used by one user, and linked dynamically when the object never changes over the lifetime of the running application.

For all its warts this is something that npm gets right. Package management is a tool for software development, not software distribution to end users.

Re:VSCode, Dev Containers and Docker by nly on Thursday January 21, 2021 @11:59AM (#25861092)
You seem to be missing the point of what we call a "Linux distribution". Shared libraries are fantastic and everything in /usr/lib should be shared
Re:VSCode, Dev Containers and Docker by hctaw on Thursday January 21, 2021 @12:06PM (#25861199)
I don't appreciate the condescension, is it not fairly well known that shared libraries aren't typically shared at all? You can distribute software that is reusable and packaged for many machines (the ultimate sense of "shared" in my opinion) but that's difficult when it relies on distribution-specific nuances and packages.

Do you really enjoy having separate packages for the same software for each flavor of package manager that does the same thing? Wouldn't it be nice if we didn't have to use containerization to distribute the same software to machines running a kernel with a stable ABI?

Re:VSCode, Dev Containers and Docker by AnIdiotOnTheNet on Thursday January 21, 2021 @12:36PM (#25861561)
The majority of "shared" libraries are used by exactly one application: https://drewdevault.com/dynlib

It makes absolutely no sense to pollute a global namespace with these.

Re:VSCode, Dev Containers and Docker by rascul on Thursday January 21, 2021 @05:45PM (#25865605)
Out of 958 libs, 230 of mine on my desktop are only used by one application, according to the results after running the stuff from the link. I'm not disagreeing with you about polluting the global namespace, but at least in my case, it's certainly not a majority.
Re:VSCode, Dev Containers and Docker by FuckButtons on Thursday January 21, 2021 @01:46PM (#25862430)
Except that - Linux distributions aren't solely tools for developing software, and their package managers are intended for all their end users.
Re:VSCode, Dev Containers and Docker by hctaw on Thursday January 21, 2021 @03:30PM (#25863811)
That's not my point - dependency resolution is a build time requirement, not an install-time one. Requiring the user's machine to resolve dependencies to install your software is fragile, bug prone, and harms the user experience.

Like it or not, users don't care if your software has interchangeable parts. They care if it runs on their system. The only sane way to guarantee a piece of software runs outside your developer machine is to include its dependencies during distribution and packaging (not refer to them - which is what package managers require). The less sane way is to use containers, but those are required when developers don't package their software sanely.

This doesn't preclude users from installing software or replacing interchangeable components should developers support it. What it prevents is disgusting bugs and workarounds because dev A built on distro B while user C wants to use it on distro D but the packages have to be separately for everyone because the distro package managers don't agree with each other on what things are named or how they should be built.

Re:VSCode, Dev Containers and Docker by millstone on Thursday January 21, 2021 @11:28PM (#25868327)
No thanks. Apps like Firefox should use the libraries - windows, menus, etc. - installed on my system. Please don't ship your own libraries for this stuff.

Containers are an even more insane approach, so maybe we are in violent agreement.

Re:VSCode, Dev Containers and Docker by moistbar on Thursday January 21, 2021 @11:46AM (#25860890)
Both apt and yum support installing to whatever directory you like as long as you have write permissions there.
Re:VSCode, Dev Containers and Docker by AnIdiotOnTheNet on Thursday January 21, 2021 @11:52AM (#25860991)
Not really. They support dumping the package contents to an arbitrary directory, but you can't actually run the software from there without either the software having been written to support arbitrary paths (almost none is) or using namespacing and chroot to build it a sandbox wherein its baked-in paths actually work.

Contrast that with something like RiscOS AppDirs, classic Mac applications, or Next/Mac Application Bundles.

Re:VSCode, Dev Containers and Docker by moistbar on Thursday January 21, 2021 @12:01PM (#25861131)
You very much can run the software from those directories, what are you talking about? Those paths are handled by the OS itself as well through LD_PRELOAD and PATH.

If what you're saying were true, unmodified software wouldn't work in a Docker container, either.

EDIT: Here's the exact command to do what you're saying isn't possible: apt-get download package; dpkg -i --force-not-root --root=$HOME package.deb

Re:VSCode, Dev Containers and Docker by AnIdiotOnTheNet on Thursday January 21, 2021 @12:24PM (#25861421)
I was going to give you the benefit of the doubt and actually try this so I could show you that you are wrong, but I couldn't even get that far because dpkg always complains it is unable to access the dpkg status area. So clearly this is not as trivial as you make it out to be. I suspect because it expects a full filesystem in $HOME with its status file in the appropriate place. In other words, it is expecting a whole separate installation to be under $HOME.

Regardless, lets assume it did work. Here's what it would do: unpack the package replacing '/' with '$HOME' in the destination paths. That's it. That software will not magically be able to find its associated libraries and configurations without the user mucking with environment variables at best, or chrooting or sandboxing such that $HOME appears to it to be a wholly separate installation.

That's not how sane systems do this sort of thing. I have been trying to do this sort of thing in Linux for pretty much as long as I have been using Linux because I loathe the way Linux installs software, and in 20 years it has never been straight forward. AppImage is a close as we get and software needs to be carefully built and packaged for that.

> If what you're saying were true, unmodified software wouldn't work in a Docker container, either.

> [...] or using namespacing and chroot to build it a sandbox wherein its baked-in paths actually work.

Re:VSCode, Dev Containers and Docker by moistbar on Thursday January 21, 2021 @12:59PM (#25861852)
Point taken, though there's no reason the package manager couldn't change the environment variables based on the directory it's told to use for the install. The pieces are there, but nobody's really taken the time to put them together. For the filesystem, some simple bind mounts would probably suffice, though you'd still run into some permissions issues I imagine.

EDIT: It seems to be significantly easier on dnf, to the point that it could be trivial to add full support for home-directory installs.

Re:VSCode, Dev Containers and Docker by hctaw on Thursday January 21, 2021 @11:48AM (#25860940)
"support" and "default" are very different things. Especially with package managers like homebrew, which comes with a loud warning that non-default (root) paths may not be supported by dependencies.

It also doesn't really, since this is a system level issue. Applications need to package their dependencies, not the other way around. Dependencies form graphs, not flat lists. The existence of a global cache of libraries shared by all programs is a total inversion of requirements.

Re:VSCode, Dev Containers and Docker by chrisjc on Thursday January 21, 2021 @12:00PM (#25861117)
Depends on how you do it, right?

For instance, we create base images configured with SDKs, libraries, frameworks, configurations, binaries, etc...

Those base images are then built, versioned, tagged and then pushed to our container repos ready to be used by developers, CI/CD, etc...

Images based on these base images never need an apt-get, pip install, etc... If there is a dependency missing, updated needed, etc... we'll create a new base image with it, following the steps above.

I would love some constructive feedback.

Re:VSCode, Dev Containers and Docker by donmcronald on Thursday January 21, 2021 @03:33PM (#25863854)
This is what I think is the most practical approach. Pinning down ALL your dependencies to the exact version is much harder than it sounds. What you’re describing sound like the way Jib [1] does it. The pictures in this [2] blog post help visualize it.

The reason I like the approach you describe is because it keeps things simpler at the start of a project and consistent across most projects.

I also think it makes sense to have those support containers build on a schedule. For example, you build your build/CI container weekly and that’s the CI container for the week. On demand project builds use that CI container which has all dependencies, etc. baked in.

It would be nice if CI systems would let me explicitly tag builds as (non)reproducible.

1. https://github.com/GoogleContainerTools/jib

2. https://phauer.com/2019/no-fat-jar-in-docker-image/

Re:VSCode, Dev Containers and Docker by sp332 on Thursday January 21, 2021 @01:12PM (#25862034)
That solves the problem one way, but I don't know if I would call it reproducible. You're working around the fact that it's not reproducible by only doing it once. You don't get the benefits, e.g. everyone who uses your images has to trust that you built them the right way.
Re:VSCode, Dev Containers and Docker by chrisjc on Thursday January 21, 2021 @03:50PM (#25864056)
I guess what I mean is that everything built using this base image should be reproducible. There is no reason (hopefully) to reproduce the base image. Any changes to the base requirements (apt-get, pip, etc) requires a whole new build and results in an entirely different artifact.

And just to be clear, I'm not building (no human) the base image. The base image is also created within it's own build pipeline that has all of the necessary things to track its materialization and lineage. Logs, manifests, etc...

Once the image has been thoroughly tested and verified (both by humans and verification scripts) each time a change is merged, the git repo is tagged, docker image is built and tagged and then pushed to the container repo.

Perhaps you could explain what you mean by the other way? Why would you ever need to recreate the base image? Perhaps if the container repo dropped off the face of the earth and had to be created from scratch?

Re:VSCode, Dev Containers and Docker by amscanne on Thursday January 21, 2021 @11:20AM (#25860525)
The industry seems to have adopted “hermetic” as a word that describes truly reproducible builds, while “reproducible” has a lower standard. In many cases, it seems to be used to mean “not dependent on local build environment”.
Re:VSCode, Dev Containers and Docker by kiallmacinnes on Thursday January 21, 2021 @10:51AM (#25860137)
Yea - I get where you're coming from - you can't rebuild the container, but each team member can reuse the same container - which is at least a step towards being fully reproducible.
Re:VSCode, Dev Containers and Docker by umvi on Thursday January 21, 2021 @11:40AM (#25860798)
We don't rebuild image every time. We store images in GitLab registry. Also, just because we use apt-get to install python3 doesn't mean build isn't reproducible. The actual toolchain and sysroot are highly version controlled and python is just used to kick off the build script so it almost doesn't really matter which version python is as long as it's backwards compatible with the build script.
Re:VSCode, Dev Containers and Docker by physicsguy on Thursday January 21, 2021 @12:00PM (#25861110)
We tried this in our research group, and found that issues come in when it gets to things like...

Oh no! CMake is too old a version to support a dependency we have to build in the image construction. So we better pull in a version of CMake from a PPA which is community maintained, and build it from source/etc.

Re:VSCode, Dev Containers and Docker by cgdub on Thursday January 21, 2021 @03:46PM (#25864011)
If I use tools that can give me reproducible builds, like Bazel or Nix, I won't need or want to use containers for development.
Re:VSCode, Dev Containers and Docker by astrea on Thursday January 21, 2021 @12:52PM (#25861759)
I'm interested to know your alternate solution. How would you recommend getting the required libraries into your image?
Re:VSCode, Dev Containers and Docker by GordonS on Thursday January 21, 2021 @12:55PM (#25861810)
I guess the OP meant such that apt/yum/dnf/whatever runs every time the image runs, rather than just once when it's built. Not that that's something I see very often, mind.
Re:VSCode, Dev Containers and Docker by corytheboyd on Thursday January 21, 2021 @01:19PM (#25862103)
Isn’t the standard approach here then to derive from base images, which have exact versions? Being honest, I’m not a Docker/VM legend but I’ve seen a few attempts at managing this, and base images was one of them.

Personally, I don’t see the issue with it if you’re at least being a little careful— don’t make obvious mistakes like installing latest/nightly packages automatically, etc.

Re:VSCode, Dev Containers and Docker by GordonS on Thursday January 21, 2021 @01:29PM (#25862211)
Yes, that's the standard approach, but the base images are frequently updated. If you really want to pin at a specific image, you need to specify the image hash, rather than using the "latest", or even a version tag (e.g. "2.1").

In your image that extends from the base image, you'll typically update the package repo cache (it is typically cleared after building the base image, to reduce the size), then install whatever packages you want.

Like you, I don't see a particular issue with updating system-level packages - especially from a security standpoint.

Re:VSCode, Dev Containers and Docker by JMTQp8lwXL on Thursday January 21, 2021 @11:09AM (#25860372)
I never understood why commands like apt-get don't take a version with the dependency name.
Re:VSCode, Dev Containers and Docker by astrea on Thursday January 21, 2021 @11:19AM (#25860505)
They do. You can do `apt-get install package=version`
Re:VSCode, Dev Containers and Docker by wongarsu on Thursday January 21, 2021 @11:22AM (#25860565)
As far as I'm aware it's supported (something like `apt-get install virtualbox=5.0.18-dfsg-2build1 -V`). It's just not commonly used because you usually just choose a distro with the desired update granularity (whether you want to the newest version out there or a consistent version with backported security fixes, or something in between).
Re:VSCode, Dev Containers and Docker by falcolas on Thursday January 21, 2021 @12:54PM (#25861798)
Keeping apt-get in our container builds is vital, since it helps keep CVEs out of our containers. We can do automated re-builds of all of our containers weekly and (typically) it lets us keep on top of the CVE game.
Re:VSCode, Dev Containers and Docker by outside1234 on Thursday January 21, 2021 @10:50AM (#25860124)
And by that you mean that the version of those deps aren't pinned?
Re:VSCode, Dev Containers and Docker by beagle3 on Thursday January 21, 2021 @11:11AM (#25860402)
You depend on what the repository serves for "apt-get". You may pin it, but it still doesn't guarantee you're going to get the same version if it was replaced on the repository without changing the version.
Re:VSCode, Dev Containers and Docker by dhruvrrp on Thursday January 21, 2021 @11:14AM (#25860430)
At least for python, just pinning the top level dependencies are not enough. if you pin tensorflow==2.4.0, it doesn’t pin its required packages, rather it just defines a range. An example would be tf will try to get wheel>=0.26 (which was released in 2015 and is currently on release 0.36.2)
Re:VSCode, Dev Containers and Docker by collyw on Friday January 22, 2021 @09:03AM (#25871730)
Whats the correct way to get libraries into your docker container? We have plenty of apt-get / pip calls in the dockerfiles at my work (set up by someone else).
Re:VSCode, Dev Containers and Docker by NashHallucinate on Thursday January 21, 2021 @02:12PM (#25862778)
Not if you specify the versions to install, at least