News for hackers. Stuff that matters.
Re:Dive – A tool for exploring each layer in a Docker image by bepvte on Sunday November 25, 2018 @04:23PM (#18529141)
Is there a smart way to make more layers than splitting everything up into modules and ADDing each one? Why doesnt docker have delta transfers yet?
Re:Dive – A tool for exploring each layer in a Docker image by orf on Sunday November 25, 2018 @04:45PM (#18529262)
Layers are delta transfers. Maybe not as smart as they could be, but there is a lot of under-appreciated complexity hiding here.
Re:Dive – A tool for exploring each layer in a Docker image by bepvte on Sunday November 25, 2018 @05:42PM (#18529549)
layers are not delta transfers in my opinion, they are just new copies or removed files. its like saying sftp is delta transfer because you don't have to send your entire disk image every time. I love layers and dockers very nice reusing of base layers and such but i feel like delta transfers would dramatically improve my workflow and upload times.
Re:Dive – A tool for exploring each layer in a Docker image by Confiks on Sunday November 25, 2018 @05:55PM (#18529620)
> its like saying sftp is delta transfer because you don't have to send your entire disk image every time

I see your point, but the comparison to sftp seems really off. Overlay2 [1] indeed works on whole file differences, just as sftp would do.

Additionally, I don't think making actual byte-level deltas would yield you much improvement on container size. It would also increase access time unless you keep a cache that doubles storage requirements per image. Dockerfiles primarily add files rather than change them, and actual changed files are often small.

[1] https://docs.docker.com/storage/storagedriver/overlayfs-driv...

Re:Dive – A tool for exploring each layer in a Docker image by infogulch on Monday November 26, 2018 @01:49PM (#18535766)
> i feel like delta transfers would dramatically improve my workflow and upload times

If you think delta transfers would help, you're saying you have a bunch of data that doesn't change between image builds, but you have to re-upload all that data every time.

This actually sounds like a perfect use-case for image layers. If you can rearrange your Dockerfile to put the steps where files change more rarely before steps that change frequently, docker will automatically cache the earlier steps and not reupload if that layer already exists.

Image layers are at their heart already an implementation of a cache + delta transfer system. (Compared to some other delta systems it trades storage efficiency for computational efficiency.) You can get small delta transfers, but you have to work with docker via your Dockerfile to make it happen.

For example, that could mean: moving steps up or down the Dockerfile so more frequently changed layers come later; splitting a step into two steps to exploit the previous; making a step deterministic so you don't have to force rebuild; use multi-stage builds to make all of these easier to implement.

Re:Dive – A tool for exploring each layer in a Docker image by lstamour on Sunday November 25, 2018 @05:54PM (#18529616)
My understanding is the on-disk storage might not be delta, as tar files, but the transfer on the wire is... could be wrong, but I think they intentionally made delta-transfer a docker registry network concern? I suppose if you wanted delta storage on-disk that again could be an implementation concern, knowing where to pull and reassemble the bits into a standard image? Oh and it uses gzip compression on the wire and possibly elsewhere, again a form of delta compression.