News for hackers. Stuff that matters.

A comment on Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze via a Single API. Parent comment

Re:Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze... by jamesblonde on Thursday November 10, 2016 @12:05PM (#12922103)
It's so bad that big companies, like Netflix, have to write a whole service to store a copy of the s3 metadata: http://techblog.netflix.com/2014/01/s3mper-consistency-in-cl...

Netflix have turned the S3 metadata eventually consistent problem into s3mpr metadata eventually consistent problem. The difference is that they can now inspect and reason about s3mpr's metadata.

Spotify have had to do the same thing for Google Cloud Engine. I can't help wonder if eventually consistent object stores will go the way of NoSQL databases, when a consistent, scalable hierarchical filesystem appears.

[ Parent | on HN ]
Re:Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze... by Scaevolus on Thursday November 10, 2016 @01:27PM (#12922851)
GCS is a bit better, since individual objects are strongly consistent, but I've been bitten by the eventually consistent listing.
[ Parent | on HN ]
Re:Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze... by Yrlec on Thursday November 10, 2016 @06:25PM (#12925646)
Also on GCS, if you do a HEAD after DELETE on a bucket that is under lifecycle management it returns 200 instead of 404. Not really a consistency issue but it can really come and bite if you if you're not aware of it. GET returns 404 but HEAD returns 200.

I reported it as a bug but Google said it was by design. More specifically they said: "You are correct, if the versioning enabled in your bucket then the object metadata is saved as an archive object in the bucket [1].This is the reason you are getting 200 for your HEAD request."

[ Parent | on HN ]
Re:Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze... by kevan on Thursday November 10, 2016 @12:11PM (#12922154)
>I can't help wonder if eventually consistent object stores will go the way of NoSQL databases, when a consistent, scalable hierarchical filesystem appears.

Of course they will. Eventual consistency is a huge tradeoff that I don't think anyone would make if they weren't forced to.

[ Parent | on HN ]
Re:Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze... by yandie on Thursday November 10, 2016 @12:22PM (#12922272)
Then you trade availability with consistency. No system can't escape CAP
[ Parent | on HN ]
Re:Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze... by kevan on Thursday November 10, 2016 @01:18PM (#12922761)
Exactly. Availability and partition tolerance are often hard requirements, so we have to do all sorts of gymnastics to deal with eventual consistency. In personal projects where availability isn't a big concern and my most precious resource is my own time I tend to make different tradeoffs.
[ Parent | on HN ]
Re:Show HN: Amazon S3, Microsoft Azure, Rackspace, Backblaze... by jamesblonde on Thursday November 10, 2016 @12:13PM (#12922176)
Maybe something like this - 1 million ops/sec on HDFS: http://www.logicalclocks.com/index.php/2016/10/14/hops-smash...
[ Parent | on HN ]