Thanks for doing this. Many times in my career I've had to deal with long-standing assumptions about storage performance in cloud infrastructure without having done any testing.
It's trivially easy to spin up an instance or three and run FIO.
Re:Unbundling AWS
by angry_octet on Monday December 02, 2019 @09:13PM
(#21689323)
Thanks for the EBS tips, I'll give it a go with these instances. Still limited by the EFS speed though, which means multiple EFS volumes to shard writes, additional code to handle that etc. (If anyone knows how to use a VFS/fuse adapter to stack/union these into one volume please shout out.)
Since 40-100GBE is extremely cheap now, I don't really see how the bandwidth can be so low. When we're building compute clusters it is multiple rails of 25/50/100GBE (or other fabric like IB). Is Amazon using anemic 2x10GBE LOM or something?
Relatedly, its always mysterious debugging on AWS because you have to theorise about the underlying hardware and topology. Maybe there is a VIP level of CloudWatch where you get told how it works underneath...
> If I have a filer with 500TB of disk I should be getting 5-10GiBy/s reads
That kind of performance requires a 50-100 GbE SAN fabric. I'm unaware of any cloud provider that offers this yet, let alone an on-prem fabric that isn't extremely expensive. (Our customers can get close with the u-24tb1.metal instance type, which offers 28 Gbps dedicated EBS throughput.)
> Likewise EBS and emphemeral SSDs top out at 250MiBy/s, which is just abysmal.
There are plenty of EC2 instance types that offer 1750MiB/s of EBS throughput - see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptim... for a list. You'll need to stripe some 1.3TB EBS volumes together to achieve it, but it's absolutely attainable. (I did just that when I worked at a well-known YC company to build their high-throughput Kafka clusters.)
As for instance storage, I just ran a fio test on an i3.large's NVMe SSD and it performed at 80k read IOPs, 311MiB/s sustained sequential read. These too can be striped for higher performance -- I was able to achieve nearly 24GiB/sec (at 644k IOPs) on an i3.16xlarge instance.