Re:Cloud Storage for $2 per TB per month
by toolslive on Saturday March 07, 2020 @03:39AM
(#22510605)
Some object stores do this, but aws S3 for example does not. You can list the contents of buckets, nicely sorted by name. You can mimic directory structures if you want.
However, you touched a key point: object stores are all about throughput, not latency. You can store at a GB/s (if you have the pipes), but even checking if an object exists will cost you a few milliseconds.
Re:Cloud Storage for $2 per TB per month
by statictype on Saturday March 07, 2020 @02:49AM
(#22510427)
Got it. Thanks for explaining.
With a standard fileshare, you need to walk the filesystem to retrieve your file - this incurs a ton of metadata overhead. It also means when you've got potentially billions of files in a directory, it can be slooooowww. All the metadata requests also make it very chatty - so doing it over a WAN link tends to be extremely painful if it works at all. Newer versions of SMB and NFS have done a lot to batch the metadata requests but they are still protocols meant to happen at extremely low latency inside a datacenter.