When you want to feed your GPUs with what they need (model, weights, data), that kind of hoops slow you down exponentially. You need to be able to stream data to pinned memory, or to the GPU directly (hence GPUDirect) to keep your cards saturated.

This is why systems like Weka exist, and why Lustre is still being developed and polished. These systems reach tremendous speeds. This is not an exaggeration when you connect terabits of bandwidth to these storage systems.

I haven't had to feed a building full of GPUs, but I do wonder: does Ceph ever appear on your radar for this sort of work? And if not, why? Seems like it would fit rather well. Yet it never seems to get mentioned.

What if all your code is already written to use a filesystem, and you want to change the backing store from nfs to object store? Or what if you want to abstract away the specific blob store?

This was our exact use case backing store from nfs to object store. You can cloud cloud-specific mount providers and use a thin client in the middle

If you want de couple model loading and distribution, If you do it in application when you need application pulling from bucket can be slow