News for hackers. Stuff that matters.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by dymk on Wednesday May 17, 2023 @01:35PM (#35978650)
> 99.99% availability

That seems... really bad, for the Core Core Service that Everything Depends On.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by abeppu on Wednesday May 17, 2023 @01:43PM (#35978778)
S3 standard storage class also says it's 99.99% availability so ... I wonder how hard it is to get beyond that?

https://aws.amazon.com/s3/faqs/

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by sosodev on Wednesday May 17, 2023 @01:55PM (#35978928)
It's extremely hard to get availability higher than that. Possible but not something companies are willing to promise in an SLA.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by awill on Wednesday May 17, 2023 @02:23PM (#35979274)
I'd say it's not so much that it's really hard, it's that the compromises aren't worth it.

note, I worked on S3 2015-2017

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by abeppu on Wednesday May 17, 2023 @02:31PM (#35979367)
So ... e.g. for 5 nines, the erasure code configuration would demand that do more writes in more locations and the cost impact is too high?
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by jl6 on Wednesday May 17, 2023 @04:09PM (#35980587)
I imagine that as you add more nines, you start to hit more problems that are more out of your control. Like, how many nines do your backup diesel generators have?
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by skrtskrt on Wednesday May 17, 2023 @04:25PM (#35980816)
or replication if you're not erasure coding.

So yeah you need more storage-optimized server racks and all the associated manpower and maintenance, you also need them to be distributed across different datacenters and zones which of course also impacts latency and your ability to provide some appearance of consistency, then you also need the same distribution for stateless services serving the data.

On and on and on and you might be nearly doubling cost to get to an extra 9 that almost all of your customers won't care about.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by sosodev on Wednesday May 17, 2023 @03:40PM (#35980210)
That's a fair point. It would be quite possible if the customers were willing to pay a lot more.

I worked on DigitalOcean's object storage for about a year not long ago. Makes sense that those of us who have been in this space would be interested in this article haha.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by foobiekr on Wednesday May 17, 2023 @11:16PM (#35983789)
Absent here is any statement about how available is defined in terms of being _meaningfully up_ like iops or bps.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by CobrastanJorji on Thursday May 18, 2023 @02:38AM (#35984738)
It's a cost problem. You can absolutely get higher availability, but it requires more hardware, uses more networking, and it makes maintenance and feature development more expensive. Blob storage customers are very price sensitive, but they also want lots of availability. That's why you'll see various "tiers" of storage, to try and capture all the combinations of availability and storage cost that various kinds of customers might want. Pretty much nobody wants to pay twice the cost to turn an exabyte of 99.99% available data to 99.995% available data, so it's not a product.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by petters on Wednesday May 17, 2023 @01:46PM (#35978825)
Not really bad. It’s less than an hour per year. Most users would not notice.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by sosodev on Wednesday May 17, 2023 @01:52PM (#35978897)
How is that bad? 99.99% means the system is down for less than an hour each year. https://uptime.is/99.99
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by foobiekr on Wednesday May 17, 2023 @11:14PM (#35983768)
of course, it actually depends on how they define "up."
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by rbanffy on Wednesday May 17, 2023 @01:59PM (#35978985)
When you build anything on it, you know it may not be available for a couple hours a year and architect around it. 99.99 isn't bad at all. Your ISP probably has worse SLAs.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by gumby on Wednesday May 17, 2023 @02:08PM (#35979067)
By comparison the old US POTS phone system was required to maintain downtime of less than five minutes per decade. I have no idea what rules, if any, apply to the modern wireless systems.

Note: The consent decree, the actual legal obligation of the Bell system, actually specified the percentage of times someone would pick up the phone and not get a dial tone or operator. It was surprisingly high -- IIRC something like 2%, which is why you can see people toggling the hook in old movies. The five minutes per decade constraint I know because some of my old customers made digital phone switches and they had to provide that SLA to their phone company customers, or else not get the order.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by _trampeltier on Wednesday May 17, 2023 @02:37PM (#35979422)
That was before we had to update every system every day :-)
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by biofunsf on Wednesday May 17, 2023 @02:56PM (#35979640)
I'm having trouble finding any reference for the "five minutes per decade" downtime limit. That would be "6 9s" or 99.9999% uptime which is just crazy in today's world. Even today VOIP providers only claim 99.999%, though it seems like in practice many fail to even get close too that. I think I found the 756 page consent decree between Bell and the US you mentioned but I can't find a reference there either, though it is quite a massive doc with so-so OCR: https://www.google.com/books/edition/Consent_Decree_Program_...

Stepping back a bit, it seems like the FCC would be in charge of establishing rules like this but that consent decree I found (which might be the wrong one), is with the house Antitrust committee. (After my own Googling failed I also asked GPT-4 and it isn't aware of this either)

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by sosodev on Wednesday May 17, 2023 @03:43PM (#35980253)
I think they got 5 nines mixed up with 6 nines. The 5 nines is 5 minutes per year. I asked ChatGPT and it claims they had a target of 5 nines which makes sense.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by gumby on Wednesday May 17, 2023 @04:18PM (#35980729)
Well as I noted in my comment the consent decree itself IIRC specified availability rather than uptime.

However the 5 min figure comes from my customers like DSC (R.I.P), Ericsson and Nokia building POTS switches. These guys were deadly serious, like the folks who made spacecraft and medical devices, plus the Ericsson and Nokia folks were nice too.

In DSC’s case they were so paranoid that they paid us a massive amount to maintain a special tool chain for just for them. It was frozen in time (no upgrades) and when they reported a bug and we sent them an updated tool chain they diffed the binaries and made sure that every delta was due to the big fix and nothing else (that the dev hadn’t snuck in some other patch for some reason)! They did some other headstands with their hardware and software, but in the end it didn’t save them.

It’s cool that the consent decree is online. That arrangement with the Bell system was very clever, though it led to a lot of weird anomalies and distortions but I think it did end up with a better phone system than the PTT model. It’s also been a better model than what’s happened with the power and water utilities.

The FCC back then was a better regulator for phone customers than the (now obsolete) ICC had been.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by didacticBuffalo on Wednesday May 17, 2023 @04:36PM (#35980946)
We kept a data closet in Manhattan in a sister building to 33 Thomas. Customers would ask for a BRP. We had N+2 for data level stuff and higher but some customers would get hung up on not having second location. We took the stance if the facility fails we all have bigger issues.

We had one 18 minute issue in the ten years of use. It was the best facility we ever used.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by gumby on Wednesday May 17, 2023 @06:14PM (#35981938)
I remember in the late 90s/early 00s a lot of cases when sites would go down (everybody self-hosted) due to backhoe events and the like.

These companies weren't idiots: they had replication for their databases and leased redundant transmission service because they knew this kind of thing could happen. The problem is you'd buy transmission from two providers...whose fiber turned out to be in the same conduit, or even both had rented bandwidth on the same fiber.

At least people are smarter these days, and with widespread cloud service there are fewer people who need to keep track of this stuff.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by kccqzy on Wednesday May 17, 2023 @07:08PM (#35982374)
They could be serious enough about it, and do everything right, and still have an outage. https://news.ycombinator.com/item?id=34665023
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by themerone on Wednesday May 17, 2023 @09:04PM (#35983081)
When I interned a phone company I was surprised by the lack of hardware redundancy.

The equipment was trusted to not fail in it's lifetime.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by zeroxfe on Wednesday May 17, 2023 @04:16PM (#35980697)
That's 5 minutes a year, not decade. (I worked in telecom for a very long time.)

Telco basically set the bar for "5 9s."

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by colesantiago on Wednesday May 17, 2023 @02:19PM (#35979222)
Oof.

You're not wrong, this is really really bad, especially for Dropbox, storage is their business so I expected way better.

These stats are no different to S3 at all. All of this engineering and moving away from AWS and for so few gain in availability.

I was initially excited when they moved away from AWS and expected industry leading higher availability when they moved away, but I was wrong.

This is disappointing for Dropbox which this is their main business, storing files without any minor hiccups or outages for years.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by rrdharan on Wednesday May 17, 2023 @02:31PM (#35979369)
> These stats are no different to S3 at all. All of this engineering and moving away from AWS and for so few gain in availability.

What makes you think a gain in availability matters or is necessarily a motivation for the project?

If they can achieve the same availability at far lower cost, it’s a win for them, which is why they would (and did) do it.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by colesantiago on Wednesday May 17, 2023 @02:44PM (#35979502)
> What makes you think a gain in availability matters or is necessarily a motivation for the project?

This isn't a win for enterprise / business / mission critical customers. Governments and public services cannot use this at all.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by jazzyjackson on Wednesday May 17, 2023 @02:57PM (#35979650)
You think 9-1-1 doesn't experience downtime?
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by colesantiago on Wednesday May 17, 2023 @07:18PM (#35982451)
Strawman. Who said that emergency systems don't ever experience downtime? Orgs using mission critical systems would trust a service that has less than 5x9s.

> Emergency response systems is 99.999% or “five nines” – or about five minutes and 15 seconds of downtime per year.

https://aws.amazon.com/blogs/publicsector/achieving-five-nin...

I see hospitals will be using AWS and other reliable hosts to use S3 rather than Dropbox.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by jazzyjackson on Wednesday May 17, 2023 @11:38PM (#35983920)
Hospitals have budgets too, not all of them will spring for the 5th 9
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by colesantiago on Thursday May 18, 2023 @12:53AM (#35984247)
And AWS is there for the 5th 9, not Dropbox. therefore, Dropbox is not suitable or reliable for these kind of orgs, even with this custom built system they built.

One can have a custom built or an experienced IT team using AWS and achieve the same or even better availability using AWS and other providers that care about critical availability, if architected correctly.

https://docs.aws.amazon.com/wellarchitected/latest/reliabili...

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by alex_lav on Wednesday May 17, 2023 @03:23PM (#35980000)
The propaganda is wild to make people believe governments and public services have even 99% uptime, let alone 99.99%.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by colesantiago on Wednesday May 17, 2023 @07:41PM (#35982614)
5x9s is more than possible and has been done. not propaganda, don't know what you've been reading.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by alex_lav on Wednesday May 17, 2023 @08:36PM (#35982912)
Yes 5x9s is possible. No, governments and utilities services do not keep 5x9s.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by CaveTech on Wednesday May 17, 2023 @02:52PM (#35979582)
They can, and do.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by colesantiago on Wednesday May 17, 2023 @07:12PM (#35982409)
They use AWS more than the unreliability of Dropbox.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by awill on Wednesday May 17, 2023 @02:24PM (#35979282)
this is an overreaction. Dropbox is mostly storing files that are already stored on the customer's device, so customers usually won't notice an outage.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by colesantiago on Wednesday May 17, 2023 @02:36PM (#35979417)
How is this an overreaction?

I would expect Dropbox, a file storage company that proudly invests heavily in tech and infrastructure to achieve a better availability than what they already were on AWS (99.99%)

In terms of availability the change is pretty much 0 and as a business / enterprise customer I might as well choose a different service with similar or higher 9s or (if my needs are complex) choose S3.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by nmjohn on Wednesday May 17, 2023 @02:42PM (#35979485)
Can you provide an example of an alternative service which will give higher than 4 nines for availability that an enterprise customer would pick instead if that < 1hr of downtime per year was too high?
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by colesantiago on Wednesday May 17, 2023 @02:47PM (#35979536)
AWS (Architected Correctly) which I am sure Dropbox has experience in.

https://aws.amazon.com/blogs/publicsector/achieving-five-nin...

Here is a service that has managed to achieve 5x9s of availability:

https://ably.com/

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by jazzyjackson on Wednesday May 17, 2023 @02:59PM (#35979679)
> achieve 5x9s of availability:

Guaranteed availability is a bet they're willing to make, a gamble they've been on top of so far, a risk that, should something fail, they will pay out on according to their SLA.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by colesantiago on Thursday May 18, 2023 @01:04AM (#35984284)
Seems to be working out for them (Ably) and this isn't a public billion dollar company like Dropbox.

I would have thought that given Dropbox's engineering talent, they would have designed a system that would account for 5x9s and even making that guarantee for enterprise or mission critical customers.

Can't even find their SLAs anywhere for these customers, so I presume that Dropbox doesn't care about them.

Guess I was wrong and this is just disappointing and made the move not worth it.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by prmoustache on Wednesday May 17, 2023 @03:20PM (#35979938)
I'd rather have maximum consistency and integrity than maximum availability.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by colesantiago on Thursday May 18, 2023 @01:20AM (#35984370)
Most tech these days have improved to offer maximum consistency, availability is also important, which is why it is a good thing that Dropbox still trusts AWS and especially S3, better than their own on-prem solution.

https://aws.amazon.com/solutions/case-studies/dropbox-s3/

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by rsync on Wednesday May 17, 2023 @03:26PM (#35980027)
"... storage is their business so I expected way better."

Storage is our business and we target an even worse 99.95% availability[1].

Availability has a cost. That cost is complexity.

We would very much prefer to have boring outages more often than have fascinating outages very rarely.

[1] https://www.rsync.net/resources/notices/sla.html

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by colesantiago on Thursday May 18, 2023 @01:34AM (#35984444)
> Availability has a cost. That cost is complexity.

It seems that Dropbox's system is complex enough but still isn't as designed to be highly available (5x9s) as a well architected system on AWS using S3.

Yet It is surprising a public billion dollar company like Dropbox cannot go beyond this for their mission critical customers (govt agencies, healthcare, military, etc) willing to spend hundreds of millions.

So yes, I expected way better.

Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by sosodev on Wednesday May 17, 2023 @03:45PM (#35980276)
Do you know of any companies that actually provide better reliability in their consumer product? The ones that lie or skew their uptime calculation don't count.
Re:Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System by colesantiago on Thursday May 18, 2023 @01:25AM (#35984402)