(I work for AWS. Opinions expressed here are my own and not necessarily those of my employer.)

> If I have a filer with 500TB of disk I should be getting 5-10GiBy/s reads

That kind of performance requires a 50-100 GbE SAN fabric. I'm unaware of any cloud provider that offers this yet, let alone an on-prem fabric that isn't extremely expensive. (Our customers can get close with the u-24tb1.metal instance type, which offers 28 Gbps dedicated EBS throughput.)

> Likewise EBS and emphemeral SSDs top out at 250MiBy/s, which is just abysmal.

There are plenty of EC2 instance types that offer 1750MiB/s of EBS throughput - see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptim... for a list. You'll need to stripe some 1.3TB EBS volumes together to achieve it, but it's absolutely attainable. (I did just that when I worked at a well-known YC company to build their high-throughput Kafka clusters.)

As for instance storage, I just ran a fio test on an i3.large's NVMe SSD and it performed at 80k read IOPs, 311MiB/s sustained sequential read. These too can be striped for higher performance -- I was able to achieve nearly 24GiB/sec (at 644k IOPs) on an i3.16xlarge instance.

Thanks for doing this. Many times in my career I've had to deal with long-standing assumptions about storage performance in cloud infrastructure without having done any testing.

It's trivially easy to spin up an instance or three and run FIO.

Thanks for the EBS tips, I'll give it a go with these instances. Still limited by the EFS speed though, which means multiple EFS volumes to shard writes, additional code to handle that etc. (If anyone knows how to use a VFS/fuse adapter to stack/union these into one volume please shout out.)

Since 40-100GBE is extremely cheap now, I don't really see how the bandwidth can be so low. When we're building compute clusters it is multiple rails of 25/50/100GBE (or other fabric like IB). Is Amazon using anemic 2x10GBE LOM or something?

Relatedly, its always mysterious debugging on AWS because you have to theorise about the underlying hardware and topology. Maybe there is a VIP level of CloudWatch where you get told how it works underneath...

> AWS security is so complicated now, at the control plane layer, that hardly anyone understands it.

YES! Here is an example.... I wanted to use Elastic Container registry and have Fargate pull from it. I had to figure out that ECR is actually implemented like a private s3 bucket and my app needed read access to S3. Why should I be exposed to the implementation details as someone granting permissions to ECR?! Also to figure this out it is not trivial as the only way to get a clear error message is to run Cloud Trail and wait 10-15 minutes for the error to be reported there as there is a delay. Also another instance where something needed permission to an weird internal S3 endpoint for gateway access on a VPC....

I have experience deploying to Fargate from ECR and we didn't need to configure any sort of S3 access to do it, certainly for the ExecutionRoleArn. ecr:BatchCheckLayerAvailability, ecr:BatchGetImage, and ecr:GetDownloadUrlForLayer on the ECR repos was sufficient.

I recently learned GCP GCR does this same thing with Cloud Storage permissions.

You miss why folks pick AWS I think.

When you say get a filer with 500TB that can do 10GiBy/s as a developer I might hear this as the reality.

Talk to infra group -> who will sit with a bunch of sales people and go round and round then get some quotes, then go back and forth with dev group trying to really spec out what is needed, then get finance / accounting approvals, more questions and answers, then put the order in, then have sales people apologize for this and that delay, then get filer onsite, the a big todo to install, then find the gigabit Ethernet can't actually push 10GiBy/s, then have to find add-in cards for all the servers and do a big network redo to get some kind of speed to the actual servers. Then after developers play with this a bit they decide to go a different route and the dance starts again.

Contrast with this AWS. I can try out my flow with a few clicks on console.

Security is painful though - no question.

TBF that is a problem with your company (many companies and govt), not non-cloud hardware. If you have good management practices then all the groups involved will share ownship of delivering the successful outcome (whatever you need your giant high performance filer for), which also includes empowering the lead engineer to hire/train (which is often better IMHO) people with the requisite skills.

So what you are arguing is that many places have limited competency, and you can outsource the need for that to AWS -- at significant cost. I dealt with a fair number of turf guarding drongos so I'm sympathetic to this idea, but it remains true that you can get much more peak performance, and perf/$ from your own hardware.

In case my point wasn't clear - for many situations AWS is FASTER in terms of development, and once you've built your app on AWS it can be painful to switch to an on-prem solution and rebuild on the "easy" 10GB/s EBS. And by the time you are done dealing with add-in cards / driver bugs etc...

It sounds like the ease of using AWS means the technical staff capability has atrophied. If you're doing on prem all the time it isn't an issue. Of course you have to have the scale to have a tech staff, including not losing critical capability when someone leaves.

On prem cloud is still mayhem, VMWare is very very bad, Openstack is baroque. Moving anything from AWS or Azure to on prem cloud would be very frustrating.

> AWS security is so complicated now, at the control plane layer, that hardly anyone understands it.

IAM and security management are the worst parts of AWS. It's complex an unintuitive. I always second guess my choices, wondering if I've left a gaping security hole somewhere.

AWS has an automated reasoning group working on tools to help answer the high level questions about complicated policy configurations. [0] has been submitted four times to Hacker News, and received a total of 4 points and 0 comments.

[0] https://aws.amazon.com/blogs/security/protect-sensitive-data...

[1] https://aws.amazon.com/security/provable-security/

I too have been wondering the same. Does anyone else see a need for some external service that examines resources/permissions and give a clear picture and ongoing monitoring for changes? Are there any services that already do this well?

Amazon makes one https://aws.amazon.com/config/

What I'd like to see is larger space for custom rules. You can't have a single policy that defines a whitelist of functions you can do on an EC2 instance per user.

> AWS security is so complicated now, at the control plane layer, that hardly anyone understands it.

Yep, and in the context of OP asking to pull out authN and ID, it's a lot clearer how to pop in an alternate DB than IAM. Even pulling out Cognito makes some sense, but not IAM. Notice OP didn't even mention authZ because IAM roles and policies are so intertwined with every AWS service, there's another major disincentive to pulling something out. It seems any pull-out solution will need to wrap IAM such that it will incur treble complexity and performance hits.

Ideas?

What would you be willing to pay for “physical like” performance?

The same prices that are common for physical hardware:

https://www.hetzner.com/dedicated-rootserver/matrix-ax?count...

For the lazy among us, how much does Hetzner charge for 500TB of disk?

you can get 150TB for 310€/month: https://www.hetzner.com/dedicated-rootserver/sx292

2000 eur/mo

Physical-like prices?

What happens when the SSD in your colo server malfunctions? What happens when the (an) SSD attached to your cloud VM malfunctions? Why do you think those two setups should cost the same?

Because it happens at a low enough frequency that dealing with it at places I've managed physical racks is pretty much a rounding error relative to the hardware costs.

And AWS has other complexities, so the devops costs with AWS are different, but rarely lower than managing physical hardware.

I'd never pick AWS based on total cost - it's an expensive convenience that first starts to pay off in terms of cost once you're big enough to negotiate huge discounts.

If you can afford it, it's great, but a lot of people who use AWS have no idea how much of a premium they pay for it.

If you go with dedicated servers instead of colo, the provider will still take care of swapping faulty SSDs. Prices are similar to colo, only downside is that you're restricted with the hardware the vendor offers and cannot guarantee that all servers are in the same rack. But neither of that is true with cloud providers.

Yes, which are mostly SRE salaries.

I like with Google cloud how disk throughput scales with size. It's simple compared to dealing with the crazy multitude of options, striping, and provisioned IOPs AWS uses