Re:Cloud Storage for $2 per TB per month
by sydney6 on Friday March 06, 2020 @04:26PM
(#22507465)
Storage Systems of this scale are thesedays almost exclusively built around object based storage rather than "legacy" block or file backed solutions. I guess today, min.io is would be the way to go. (to "go".. little pun on the end:)
[ Parent
| on HN ]
Re:Cloud Storage for $2 per TB per month
by statictype on Friday March 06, 2020 @08:04PM
(#22508933)
What does object-based mean? How is an object different from a file (which I presumed was a collection of blocks)?
It means you access your object through a GUID. Think about it like parking your own car vs. a valet. When you park your own car you need to know the address of the garage you parked in, the floor you were on, and the spot you were in. When you valet park, you hand the attendant a ticket and he brings your car back.
With a standard fileshare, you need to walk the filesystem to retrieve your file - this incurs a ton of metadata overhead. It also means when you've got potentially billions of files in a directory, it can be slooooowww. All the metadata requests also make it very chatty - so doing it over a WAN link tends to be extremely painful if it works at all. Newer versions of SMB and NFS have done a lot to batch the metadata requests but they are still protocols meant to happen at extremely low latency inside a datacenter.
Re:Cloud Storage for $2 per TB per month
by toolslive on Saturday March 07, 2020 @03:39AM
(#22510605)
Some object stores do this, but aws S3 for example does not. You can list the contents of buckets, nicely sorted by name. You can mimic directory structures if you want.
However, you touched a key point: object stores are all about throughput, not latency. You can store at a GB/s (if you have the pipes), but even checking if an object exists will cost you a few milliseconds.
Re:Cloud Storage for $2 per TB per month
by statictype on Saturday March 07, 2020 @02:49AM
(#22510427)
Got it. Thanks for explaining.
Re:Cloud Storage for $2 per TB per month
by tgsovlerkhgsel on Friday March 06, 2020 @08:13PM
(#22508983)
My guess: No random write access to objects, you can (at best) append-only but often you can only append until the object is finalized, and cannot read it until it is finalized.
Re:Cloud Storage for $2 per TB per month
by toolslive on Saturday March 07, 2020 @03:32AM
(#22510573)
it's like a key-value store, (or a dictionary). However, the values are objects (big blobs of data). This means you can't update parts of objects without rewriting the blob. However, most of the object stores offer metadata operations (move, tag, ...), concat of n objects into 1 and partial reads.
Re:Cloud Storage for $2 per TB per month
by londons_explore on Friday March 06, 2020 @06:24PM
(#22508342)
Google's clusters are all block and file based (depending on what layer you want to use them at)