Ambry: LinkedIn’s Scalable Geo-Distributed Object Store

Re:Ambry: LinkedIn’s Scalable Geo-Distributed Object Store by rad_gruchalski on Monday April 10, 2023 @06:37PM (#35518238)

Thank you. Can a large object be uploaded to Ambry using chunking? Like s3, gcs, and azure blob storage do? The upload client splits an object in 8KB chunks and uploads them in parallel using byte ranges. Each chunk upload has its own retry policy. This prevents a situation where bad egress would interrupt large upload as a whole.

Can this be done with Ambry? I don’t see any immediate mention of byte ranges in the source you linked.

[ Parent | on HN ]

Re:Ambry: LinkedIn’s Scalable Geo-Distributed Object Store by ambry on Thursday April 13, 2023 @10:41AM (#35556203)

Short answer: Yes.

However, Ambry doesn't use byte-ranges for the upload. Each chunk can be uploaded separately and the client then requests a _stitch_ operation[1]. In that operation, the client specifies the list of blob IDs for each chunk (in order) and asks Ambry to create a metadata blob listing those IDs. Ambry then returns the blob ID for the metadata blob.

[1]: https://github.com/linkedin/ambry/blob/9b7a49ac79b1678fd7fd7...

[ Parent | on HN ]