asymptotic

Deprecation of services is a common occurrence at AWS and many other tech companies. But it's never taken lightly. A mandatory step in this process is analyzing usage logs. We need to ensure customers have transitioned to the alternative. If they haven't, we reach out. We understand why. The idea of simply "nuking" customer data without a viable alternative is unthinkable.

The InfluxDB incident brings to light the ongoing debate around soft vs. hard deletion. It's unacceptable for a hard delete to be the first step in any deprecation process. A clear escalation process is necessary: notify the customer, wait for explicit acknowledgement, disable their APIs for a short period, extend this period if necessary, soft delete for a certain period, notify again, and only then consider a hard delete.

The so-called ["scream test"](https://www.v-wiki.net/scream-test-meaning/) is not a viable strategy for a cloud service provider. Proactive communication and customer engagement are key.

This incident is a wake-up call. It underscores the importance of data durability and effective, respectful customer communication in cloud services and platform teams. Communication is more than three cover-your-ass emails; it's caring about your customers.

> Traffic regularly exceeded 1,500 queries per second (QPS) at peak times

It may be quite cost prohibitive to use a managed service like AWS Global Accelerator.

One potential solution that hasn't been discussed much here is leveraging AWS managed services like AWS CloudFront or AWS Global Accelerator. It's worth noting that Dropbox's website already uses AWS CloudFront, so they are already leveraging AWS in some capacity. Based on my cost calculations, using AWS CloudFront would cost around $40k a month, while AWS Global Accelerator would be around $22k a month.

As of August 22, 2022, AWS CloudFront supports terminating HTTP/3 in a Point of Presence (POP), which could potentially help with the latency issues Dropbox is facing. AWS Global Accelerator, on the other hand, is designed to improve the performance of applications by terminating UDP/TCP as close to users as possible then routing user traffic through the AWS global network infrastructure. This could help reduce latency by ensuring that user traffic is routed through the most optimal path, even if the user is located far from a Dropbox data center.

It's hard to estimate what the potential latency reduction of using e.g. AWS Global Accelerator is, especially at higher percentiles. However, using https://speedtest.globalaccelerator.aws/, and assuming symmetry, my connections to Asia are 35-40% lower latency.

Of course, there are trade-offs to consider when using managed services like AWS CloudFront and AWS Global Accelerator. While they can provide significant performance improvements, they also come with additional costs and potential vendor lock-in. However, given the scale of Dropbox's operations and the importance of providing a fast, reliable search experience for their users, it may be worth exploring these options further.

---

Cost estimates

Assumptions:

    1. Dropbox's peak traffic is 1,500 queries per second (QPS).
    2. Average data transfer per query is 100 KB.
    3. 50% of the traffic comes from North America, 25% from Europe, and 15% from Asia (remaining 10% from other regions, for pricing purposes put it into Asia's calculations).

1) AWS CloudFront cost estimation:

    Data transfer:
    - North America: 1,500 QPS * 0.5 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 194.4 TB  
    - Europe: 1,500 QPS * 0.25 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 97.2 TB  
    - Asia: 1,500 QPS * 0.25 * 100 KB * 60 seconds * 60 minutes * 24 hours * 30 days = 97.2 TB

    Data transfer cost:
    - North America: 194.4 TB * $0.085/GB = $16,524
    - Europe: 97.2 TB * $0.085/GB = $8,262
    - Asia: 97.2 TB * $0.120/GB = $11,664

    Total data transfer cost: $16,524 + $8,262 + $7,006 = $36,200

    HTTP requests:
    Total requests: 1,500 QPS * 60 seconds * 60 minutes * 24 hours * 30 days = 3,888,000,000

    Using the updated AWS CloudFront pricing (as of May 22, 2023):
    - HTTP requests cost: 3,888,000,000 * $0.0075/10,000 = $2,916

---

2) AWS Global Accelerator cost estimation:

    Data transfer:
    - Total data transfer: 194.4 TB (NA) + 97.2 TB (EU) + 97.2 TB (Asia) = 388.8 TB

    Using the updated AWS Global Accelerator pricing (as of May 22, 2023):
    - Data transfer cost (averaged across regions): 388.8 TB * $0.035/GB = $13,608
    - (Also need to add EC2 egress cost, 388.8 TB * $0.02/GB = $7,776
    - Total data transfer cost = $21,384

    Accelerator:
    - Assuming 1 accelerator with 2 endpoints (1 for HTTP/2 and 1 for HTTP/3)
    - Accelerator cost: 1 * $18/accelerator/day * 30 days = $540

I’m not that familiar with S3, but I never noticed any concerns with S3 during an AZ outage. I’m not at all familiar with Aurora Serverless or ECS.

For all AWS services you can always ask AWS Support pointed, specific questions about availability. They usually defer to the service team and they’ll give you their perspective.

Also keep in mind that AWS teams differentiate between the availability of their control and data planes. During an AZ outage you may struggle to create/delete resources until an AZ evacuation is completed internally, but already created resources should always meet the public SLA. That’s why especially for DR I recommend active-active or active-“pilot light”, have everything created in all AZs/regions and don’t need to create resources in your DR plan.

But for the service team I worked for, our AZ-evacuation story wasn't great at the time and it took us tens of minutes to manually move out of the AZ, but at least there wasn't a customer-visible availability impact. Once we did it was just monitoring and baby-sitting until we got the word to move back in, I think it was 1-2 days later.

If you operate on AWS you work with the assumption that an AZ is a failure domain, and can die at any time. Surprisingly many service teams at AWS still operate services that don't handle AZ failure that well (at the time). But if you operate services in the cloud you have to know what the failure domain is.

1) avg mean (for throughput estimates), 2) p50 (typical customer / typical load), 3) p90 / p99 / p99.99 (whatever you think your tail is) 4) p100 (max, always useful to see and know, maybe p0).

Or throw in a histogram or kernel density estimate, sometimes there are really interesting patterns.

I’ve never seen a technical blog give such traffic or latency details though.

Edit: reading other comments, please do not read this as diminishing this blog post or Discord, great clear writing and impressive solution to an interesting problem.

This is the Way.

Can you go into this a bit more or provide a code reference? I'm new to writing async Rust code and am interested in this idea.

I've slowly developed a method which incorporates 1) culture, 2) popularity in different countries, and 3) pronunciation of names and attempts to recommend you names based on names you like. It kind of works, it's taken a lot of tuning to make it output something sensible. It's specifically designed to attempt to combine cultures together, which is a top request from customers I identified.

I've been working on releasing the app for a while. If you're interested in helping me test it before its release this month please feel free to sign up here:

https://www.reddit.com/r/namenerds/comments/qge6t9/im_lookin...

I will also look at this thread's comments closely before launching.

“To identify the location of your resources relative to your accounts, you must use the AZ ID, which is a unique and consistent identifier for an Availability Zone. For example, use1-az1 is an AZ ID for the us-east-1 Region and it is the same location in every AWS account”

So each separate AWS account will have a different AZ name that maps to use1-az1, but use1-az1 is a region-wide constant.

You can also hook up an Application Load Balancer to Lambdas if that works for you: https://aws.amazon.com/elasticloadbalancing/pricing/

I heavily relied on the following blog post: https://dev.to/robertohuertasm/rust-once-and-share-it-with-a...

Overall it's painful to set up and requires knowledge of the native platform (XCode/Swift, Android Studio/NDK/Kotlin), but overall it works and it was fun getting it working.

I'm happy to answer more questions about it, I've been meaning to write up the experience and details.

With the recent GA of AWS CDK [2] I would say that it is getting more and more possible to define your own development workflow that is tailored to your own static website hosting needs and better than just raw S3. AWS CDK is still difficult to use with some rough edges, and I know that setting up e.g. a CodePipeline and CI/CD is difficult.

Here is the CDK example for hosting a S3 origin + CloudFront CDN + custom TLS certificate + CloudFront invalidation on deployments static website [3]. Here is my personal version of it combining Hugo that took me a day to set up [4]. Still complicated, lot of rough edges, but offers a pleasant dev experience [5], I haven't yet built up a CI/CD pipeline.

Disclaimer: I work for AWS but my views are my own.

[1] https://aws.amazon.com/amplify/console/

[2] https://aws.amazon.com/cdk/

[3] https://github.com/aws-samples/aws-cdk-examples/blob/master/...

[4] https://github.com/asimihsan/asim_ihsan_io/blob/master/cdk/s...

[5] https://github.com/asimihsan/asim_ihsan_io/blob/master/READM...

I prefer applications to focus on one thing and do it very well, and leave other orthogonal functionality to other applications or the OS to then do the other parts very well. In this respect Quiver is well designed, I tip my hat to the author.

"As a security conscious user who follows the best practices like: using unique passwords, 2FA, only using a secure computer and being able to spot phishing attacks from a mile away, I would have thought my accounts and details would be be pretty safe? Wrong."

Are you sure the author enabled 2FA on his Amazon retail account, or was it only enabled on his AWS account? The two systems do not share the same 2FA.

FYI I enabled 2FA on my Amazon retail account and when I called customer support they verified it. Once the verification failed and they refused to give me support.

Anyone else confirm a similar story with 2FA and support? Anyone willing to explicitly test this out?

(I think this is true...I can't remember now actually).

"At the right vertex, we have Breiman's know-nothing approach—high-capacity models like neural nets, decision forests, and nonparametrics that will fit anything given enough data. This is engineering with less science (see these remarks). Deep learning people cluster here."

The phrase "will fit anything given enough data" is misleading and not correct about cutting-edge machine learning methods. "Fit" is a useless term, and you will instead find people talking about "bias" and "variance".

For any supervised method (you know the intended outputs) you apply to predict data, there are three sources of error: bias, variance, and random. Random error is some irreducible unpredictability that cannot be modeled. Bias occurs from bad assumptions made by the model itself (e.g. maybe the model is too simple). Variance is sensitivity to small changes in the data the algorithm is trained on.

High bias means the model is too simple to capture all the variations in the data set. High variance means the model is too overfit on the data at hand and it is not successfully generalizing to unseen data. In real-world problems there is a direct tradeoff between bias and variance. Nevertheless the goal of any supervised learning model is to have both low bias and low variance.

By splitting off a big (~10-20%) chunk of all data available into a "test" set, training the model on the remaining "train" set, then evaluating it on the "test" set, it's possible to estimate the generalizability of the model on future "unseen" data by whatever metric you want. By additionally plotting learning curves one can crudely estimate whether we have high bias or high variance.

Hence the insinuation that machine learning blindly "fits" data as much as possible is false. Sophisticated (yet not difficult) methods both minimize and estimate the generalizability of the model to future unseen data (minimizing variance), inevitably at the cost of some notion of accuracy (increasing bias).

I think the OP's objection is that such ML methods "know nothing". This is a trivial statement to make. Rather, I would turn the objection on its head and ask "If our methods achieve acceptable estimated generalizability on unseen data, do we need to know anything?". This reminds me of Alan Turing's arguments about machines passing the Turing Test vs. "are they really human?".

This is purely based on the author's excellent description of the format in his arq_restore tool: https://www.arqbackup.com/s3_data_format.txt

    "WHO KNOWS WHATS GONNA HAPPEN. DO IT FOR SCIENCE."

    While our universities are turning out very
    able graduates, well versed in the sexy new
    languages of Java, .Net and the like...

    It's the knowledge that no matter how sucky a
    day turns out to be, I have done something to
    better myself.