Not using image layers to begin with. You can't reasonably do that with Docker or other software that isn't aware of the dependency graphs of the software that it runs. Docker just runs some arbitrary commands and maintains an opaque set of disk images. Seems to be the best you can do there. However, if one were to use software that actually understood the full dependency graph, you wouldn't need disk images at all and you'd be able to share storage without involving special file systems. In GNU Guix, a functional package manager, we can make containers for arbitary software without any duplication because we know the full dependency graph of the application. Applications may share sub-graphs so they are only on disk once. If two programs depend on libpng, libpng and its dependencies are only on disk in one place. With a functional package management system in place, containers become a relatively simple matter of creating namespaces and bind mounting the dependencies. No image layering, no special file systems, just simple bind mounts.

There are many problems a storage driver for docker need to solve. We are not replacing docker here, just providing a better solution for the storage driver (API of which is defined by docker) part of it. Solutions in use today are half baked and have too many problems to deal with.

Docker is already doing a good job of de-duplication by identifying shared data between applications and keeping a unique copy of shared data on disk. But many of the existing storage drivers today, would create multiple copies of such shared data in memory when many applications (containers) consume such shared data. Applications are running on top of file systems and each need a separate instance of the file system and Linux kernel does not know if all those file systems are sharing same data underneath. Each application also require a unique view of the file system and they may choose to modify their file system (thus their shadow copy of shared data) and such changes should be hidden from other applications (containers). That is where cloning technologies come into play. And there are many other things a docker storage driver need to solve. Rather than focusing on one aspect of the whole problem, it would be better to look at the whole picture and see if there are any solutions out there solving all those problems correctly and efficiently for a docker storage driver.

Yeah, Nix is cool but ain't nobody got time to learn it since Dockerfiles are so easy.

Easy on the surface, but very limiting and frustrating in practice.

I think all the energy spent around designing image layer file system after image layer file system is good evidence that the Docker approach isn't as easy as it may seem.

I would say image layer file system is not that hard to solve if you approach the problem correctly. It looks like all prior attempts tried to use general purpose solutions and in some cases a combination of those (for example, overlay on top of ext4) for dealing with a much smaller problem and ran into unnecessary complications. A lot of the complexities in a general purpose solution is unwarranted for a image layering file system.

We looked at the requirements of a docker storage driver and built a solution from scratch, just to address those efficiently and nothing else.

Interesting thought, however I think that the existing graph driver file systems are not actually an "image layer file system"... instead they are a general purpose file systems that are being used to store images.

For example, what one wants with launching multiple instances of an image is to be able to clone the image. Today, to accomplish that, file system snapshots are used to simulate this feature. And that is where a lot of the problems start happening. LCFS does things purpose built for these use cases.