News for hackers. Stuff that matters.
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by bluedino on Monday March 15, 2021 @10:47AM (#26466654)
Do they de-dupe at all?
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by butt_hugger on Monday March 15, 2021 @11:15AM (#26466970)
Doubtful since they have an option for at-rest encryption.
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by tpmx on Monday March 15, 2021 @11:36AM (#26467236)
It's just that, an option, not the default. People tend to use the defaults.
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by amthrwy on Monday March 15, 2021 @12:11PM (#26467773)
Yes. The contents of zero-byte objects are deduplicated. (The metadata, of course, are not.)
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by jgalt212 on Monday March 15, 2021 @10:52AM (#26466707)
oops, I think you found their secret sauce for cost control and economies of scale.
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by weird-eye-issue on Monday March 15, 2021 @10:59AM (#26466783)
There is no way
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by tpmx on Monday March 15, 2021 @12:24PM (#26467971)
It's obviously technically possible.
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by weird-eye-issue on Monday March 15, 2021 @01:46PM (#26469072)
"Technically possible" that they do it, sure. But I would disagree it is possible that it is S3's "secret sauce".
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by tpmx on Monday March 15, 2021 @05:12PM (#26469153)
Because?
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by weird-eye-issue on Monday March 15, 2021 @05:21PM (#26469233)
... this is on you to prove it is their secret sauce. Throwing out some random feature and saying it is their secret sauce and putting the burden of proof on somebody else makes no sense.

But it would be because 1) they probably don't have many duped objects to begin with 2) the system to de-dupe items would be complex, error prone, increase latency, and just not worth it

Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by tpmx on Monday March 15, 2021 @05:46PM (#26469591)
> 1) they probably don't have many duped objects to begin with

That's crap reasoning. You have no way of knowing that.

> 2) the system to de-dupe items would be complex, error prone, increase latency, and just not worth

Would it really be that complex? I kinda doubt that.

Read latencies wouldn't be affected - except being improved.

Write latencies - the deduplication can happen after the write has been confirmed, so no extra latency added.

Whether or not it's worth it - I think it's very hard to to estimate the deduplication factor. My guess is 1.5x. At that point it would save 33% storage. That would instanly make a whole lot of extra complexity worth it.

Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by weird-eye-issue on Monday March 15, 2021 @06:05PM (#26469827)
It's like you are living in an alternate reality
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by tpmx on Monday March 15, 2021 @06:17PM (#26469971)
...because?

I dunno, I kinda think you might be out of your depth on this one, based on that response.

Also: none of your few previous posts are tech-related. They're all about personal finance.

Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by weird-eye-issue on Monday March 15, 2021 @06:38PM (#26470215)
So just to sum up your argument: You think 33% of objects in S3 are dupes and that removing those dupes would be S3's secret sauce?
[dead]
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by weird-eye-issue on Monday March 15, 2021 @07:11PM (#26470530)
You're incredibly condescending and just full of incorrect assumptions. Done with this conversation
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by tpmx on Monday March 15, 2021 @07:11PM (#26470533)
...
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by jgalt212 on Monday March 15, 2021 @11:51AM (#26467428)
prove me wrong.
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by babelfish on Monday March 15, 2021 @11:59AM (#26467576)
You have the burden of proof for your claim.
Re:Amazon S3’s 15th Birthday: 5,475 Days and 100T Objects by mathgladiator on Monday March 15, 2021 @12:05PM (#26467666)
no