Re:S3 Strong Consistency
by banana_giraffe on Tuesday December 01, 2020 @07:41PM
(#25272003)
Yes, a ton. The big issue for me was Does key exist? -> No -> Upload object with key. Followed by Get key -> Not found. Less rarely, I'd see Upload object followed by an overwrite object, followed by a download download some combination of the two if the download was segmented (like AWS's cli will do).
The first one happened often enough to cause problems, the second one was a fairly rare event, but still had to be handled.
It might have been more of an issue with "large" buckets. The bucket in particular where I had to dance around the missing object issue had ~100 million fairly small objects. I ended up having to create a database to track if objects exist so no consumer would ever try to see if a non-existent object existed.
The first one happened often enough to cause problems, the second one was a fairly rare event, but still had to be handled.
It might have been more of an issue with "large" buckets. The bucket in particular where I had to dance around the missing object issue had ~100 million fairly small objects. I ended up having to create a database to track if objects exist so no consumer would ever try to see if a non-existent object existed.
Time to revisit all that mess, I suppose.