"You should be able to work from a stable coffee shop WiFi connection."

I think there are different philosophies, just like 'no comments allowed in code', this is a matter of judgement.

I saw 5 different corporations setup centralised workflow, and usually you end up with a massive cloud bill, incorrectly configured admin rights, under-resourced VDI, and a ticketing system that takes a week to resolve the smallest issue.

If we assume for the moment that data in question is not highly sensitive (for example public satellite imagery), there is nothing wrong with getting every dev a beefy workstation with 50 TB of storage and a 20-core CPU, and it's often even cheaper.

Then again, if it is public (or not-sensitive) data, why make it part of your infra?

If you are working with ML and have public datasets, just use the original source. Make a public mirror if you want to collaborate. Make it accessible via BitTorrent. You can bet that the amount of savings you will get from AWS will be enough to give each dev a NAS with a few tens of TB, which then they can even then use to host/mirror the public data by themselves.

My specific example was a private data set hosted on-premises (satellite imagery).

The analysts were pulling down a ~0.1% sample to develop the ML algorithms locally, and then deploying the final thing in the cloud to run through the whole petabyte.

Local development has its advantages, such as high flexibility and very fast cycle times. It's fantastically expensive to outperform a modern PCI 4.0 NVMe drive in the cloud. Locally it's a $200 part.

If it’s on-premises, why are you talking about cloud costs?

5? Are you a consultant who goes in to fix these situations?

> If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong

When I worked at Vercel during the days we supported Docker hosting (back when it was still ZEIT), we saw that even among many tenants on the same machine, the "layers" thing wasn't really that beneficial - we still had to fetch a ton of stuff over and over again.

So this is great in theory, but it doesn't really pan out in practice.

I believe Docker has even throttled/blocked corporate users doing excessive layer pulls from Hub (and suggested they sign up for X plan).

Reasonable, IMHO.

That's a recent thing, as I understand it. After Hub switched to their enterprise plan setup.

ZEIT had a pretty aggressive cache anyway, for the short time it was implemented (before ZEIT moved away from docker hosting, at least).

I always thought it was a little strange given that layers are easily CDN-cacheable indefinitely. That said I'm sure they're using something like a CDN but even the cheapest bandwidth gets expensive at level of scale.

I also recall the saying "you can't save yourself rich"

There are some times when your thinking/workflow is constrained by your environment, sometimes significantly.

But the person is already super rich and has developed such an obscene yacht sinking habit that they are still money constrained.

> Docker images are layered for a reason. If you’re having to download multi gigabyte docker layers multiple times, something has gone very wrong.

Well the images need to be in some order, and if you are fine tuning the very initial parts of the image, this can happen.

But it seems a better transfer algorithm could be used for those individual layers. It rather seems like if they don't match, everything's transferred, while there could be a lot better options as well. If we consider that they are structured like file systems then it opens even more options.

---

So I took a peek on my local Docker installation. It seems the layer files on the drive are gzipped json.

That's right, .json.gz, _that includes the files BASE64-encoded_.

Yes, I believe there is room for improvement, if this is also the format they are transferred in.

base64-encoding data only poses a 1% overhead[1] when the data is subsequently gzipped, so this is hardly an issue in practice.

[1] head -c 100000 /dev/urandom | base64 | gzip | wc -c

I wasn't really consider space overhead, which as you notice is small, the encoding just something I noticed. Just json+bloatified binary+compression seemed the epitome of solving problems with a hammer when you know the hammer really well.

But as commented to me, that was incorrect, the payload in the files I saw was actually binary-encoded metadata. The actual archives are stored only in extracted form in the client and .tar.gz in the registry. Not that putting binary metadata in JSON is really optimal to me either, but maybe there are terrific reasons for doing it. Just check /var/lib/docker/image/overlay2/layerdb/sha256/*/tar-split.json.tar.gz; the fragments seem to be straight from tar, so maybe tar is used to interpret them?

Certainly .tar.gz files are not the most optimal for the purpose of incremental updates either. It's difficult to say how much a more sophisticated transfer protocol/storage format would benefit registry servers, though some clients could benefit a lot from it.

If a space-efficient merkle-tree-hashed archive-format suitable for efficient incremental updates doesn't exist, then someone should make one ;-).

json is used only for manifests; actual layers are .tar.gz.

You are correct; I too hastily took the "payload" in the tar-split.json.gz to be the binary itself, but it's something else.

Nevertheles, an archive format other than .tar.gz could be transferred more efficiently given a previous version of the same image. Or, even .tar.gz could be transferred more efficiently if both ends extract the file first, and if there is a need to preseve sha256, then that needs a bit more work.