News for hackers. Stuff that matters.

A comment on SSDs Have Become Fast, Except in the Cloud. Parent comment

Re:SSDs Have Become Fast, Except in the Cloud by jsnell on Tuesday February 20, 2024 @12:51PM (#39444373)
I think you're wrong about that. AWS calls this class of storage "instance storage" [0], and defines it as:

> Many Amazon EC2 instances can also include storage from devices that are located inside the host computer, referred to as instance storage.

There might be some wiggle room in "physically attached", but there's none in "storage devices located inside the host computer". It's not some kind of AWS-only thing either. GCP has "local SSD disks"[1], which I'm going to claim are likewise local, not over the network block storage. (Though the language isn't as explicit as for AWS.)

[0] https://aws.amazon.com/ec2/instance-types/

[1] https://cloud.google.com/compute/docs/disks#localssds

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by wstuartcl on Tuesday February 20, 2024 @04:58PM (#39447509)
the tests were for these local (metal direct connect ssds). The issue is not network overhead -- its that just like everything else in cloud the performance of 10 years ago was used as the baseline that carries over today with upcharges to buy back the gains.

there is a reason why vcpu performance is still locked to the typical core from 10 years ago when every core on a machine today in those data scenters is 3-5x or more speed basis. Its cause they can charge you for 5x the cores to get that gain.

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by wmf on Tuesday February 20, 2024 @06:56PM (#39448553)
vcpu performance is still locked to the typical core from 10 years ago

No. In some cases I think AWS actually buys special processors that are clocked higher than the ones you can buy.

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by gowld on Tuesday February 20, 2024 @09:19PM (#39449463)
You are talking about real CPU not virtual cpu
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by wmf on Tuesday February 20, 2024 @10:44PM (#39449912)
Generally each vCPU is a dedicated hardware thread, which has gotten significantly faster in the last 10 years. Only lambdas, micros, and nanos have shared vCPUs and those have probably also gotten faster although it's not guaranteed.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by jandrewrogers on Wednesday February 21, 2024 @01:56AM (#39450872)
In fairness, there are a not insignificant number of workloads that do not benefit from hardware threads on CPUs [0], instead isolating processes along physical cores for optimal performance.

[0] Assertion not valid for barrel processors.

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by phanimahesh on Wednesday February 21, 2024 @12:14AM (#39450390)
The parent claims that though aws uses better hardware, they bill in vcpus whose benchmarks are from a few years ago, so that they can sell more vcpu units per performant physical cpu. This does not contradict your claim that aws buys better hardware.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by wmf on Wednesday February 21, 2024 @12:39AM (#39450513)
It's so obviously wrong that I can't really explain it. Maybe someone else can. To believe that requires a complete misunderstanding of IaaS.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by asalahli on Wednesday February 21, 2024 @09:15PM (#39462313)
GP is probably refferring to blurbs like these

> Amazon SimpleDB measures the machine utilization of each request and charges based on the amount of machine capacity used to complete the particular request (SELECT, GET, PUT, etc.), normalized to the hourly capacity of a circa 2007 1.7 GHz Xeon processor. See below for a more detailed description of how machine utilization charges are calculated.

https://aws.amazon.com/simpledb/pricing/

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by wmf on Wednesday February 21, 2024 @10:48PM (#39463018)
SimpleDB is over 15 years old. I guess it's the only service still using "normalized" pricing. Newer services like RDS tell you exactly which processor you're getting and how many cores.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by flaminHotSpeedo on Wednesday February 21, 2024 @12:27AM (#39450455)
> there is a reason why vcpu performance is still locked to the typical core from 10 years ago

That is transparently nonsense.

You can disprove that claim in 5 minutes, and it makes literally zero sense for offerings that aren't oversubscribed

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by pclmulqdq on Tuesday February 20, 2024 @02:14PM (#39445545)
That's the abstraction they want you to work with, yes. That doesn't mean it's what is actually happening - at least not in the same way that you're thinking.

As a hint for you, I said "a network", not "the network." You can also look at public presentations about how Nitro works.

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by jasonwatkinspdx on Tuesday February 20, 2024 @04:50PM (#39447443)
Both the documentation and Amazon employees are in here telling you that you're wrong. Can you resolve that contradiction or do you just want to act coy like you know some secret? The latter behavior is not productive.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by stingraycharles on Wednesday February 21, 2024 @12:16AM (#39450406)
The parent thinks that AWS' i3 NVMe local instance storage is using a PCIe switch, which is not the case. EBS (and the AWS Nitro card) use a PCIe switch, and as such all EBS storage is exposed as e.g. /dev/nvmeXnY . But that's not the same as the i3 instances are offering, so the parent is confused.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by jsnell on Tuesday February 20, 2024 @03:57PM (#39446809)
I've linked to public documentation that is pretty clearly in conflict with what you said. There's no wiggle room in how AWS describes their service without it being false advertising. There's no "ah, but what if we define the entire building to be the host computer, then the networked SSDs really are inside the host computer" sleight of hand to pull off here.

You've provided cryptic hints and a suggestion to watch some unnamed presentation.

At this point I really think the burden of proof is on you.

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by stingraycharles on Tuesday February 20, 2024 @09:31PM (#39449527)
You are correct, and the parent you’re replying to is confused. Nitro is for EBS, not the i3 local NVMe instances.

Those i3 instances lose your data whenever you stop and start them again (ie migrate to a different host machine), there’s absolutely no reason they would use network.

EBS itself uses a different network than the “normal” internet, if I were to guess it’s a converged Ethernet network optimized for iSCSI. Which is what Nitro optimizes for as well. But it’s not relevant for the local NVMe storage.

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by MichaelZuo on Wednesday February 21, 2024 @10:42AM (#39455152)
The argument could also be resolved by just getting the latency numbers for both cases and compare them, on bare metal it shouldn't be more than a few hundred nanoseconds.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by fcsp on Wednesday February 21, 2024 @02:47AM (#39451140)
I see wiggle room in the statement you posted in that the SSD storage that is physically inside the machine hosting the instance might be mounted into the hypervised instance itself via some kind of network protocol still, adding overhead.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by eek2121 on Wednesday February 21, 2024 @04:19PM (#39459703)
At minimum, the entire setup will be virtualized, which does add overhead.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by jng on Tuesday February 20, 2024 @02:44PM (#39445944)
Nitro "virtual NVME" device are mostly (only?) for EBS -- remote network storage, transparently managed, using a separate network backbone, and presented to the host as a regular local NVME device. SSD drives in instances such as i4i, etc. are physically attached in a different way -- but physically, unlike EBS, they are ephemeral and the content becomes unavaiable as you stop the instance, and when you restart, you get a new "blank slate". Their performance is 1 order of magnitude faster than standard-level EBS, and the cost structure is completely different (and many orders of magnitude more affordable than EBS volumes configured to have comparable I/O performance).
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by rcarmo on Wednesday February 21, 2024 @09:21AM (#39454138)
This is the way Azure temporary volumes work as well. They are scrubbed off the hardware once the VM that accesses them is dead. Everything else is over the network.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by dekhn on Tuesday February 20, 2024 @04:39PM (#39447308)
it sounds like you're trying to say "PCI switch" without saying "PCI switch" (I worked at Google for over a decade, including hardware division).
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by pclmulqdq on Tuesday February 20, 2024 @09:42PM (#39449574)
That is what I am trying to say without actually giving it out. PCIe switches are very much not transparent devices. Apparently AWS has not published anything about this, and doesn't have Nitro moderating access to "local" SSD, though - that I did get confused with EBS.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by pzb on Tuesday February 20, 2024 @11:40PM (#39450226)
AWS has stated that there is a "Nitro Card for Instance Storage"[0][1] which is a NVMe PCIe controller that implements transparent encryption[2].

I don't have access to an EC2 instance to check, but you should be able to see the PCIe topology to determine how many physical cards are likely in i4i and im4gn and their PCIe connections. i4i claims to have 8 x 3,750 AWS Nitro SSD, but it isn't clear how many PCIe lanes are used.

Also, AWS claims "Traditionally, SSDs maximize the peak read and write I/O performance. AWS Nitro SSDs are architected to minimize latency and latency variability of I/O intensive workloads [...] which continuously read and write from the SSDs in a sustained manner, for fast and more predictable performance. AWS Nitro SSDs deliver up to 60% lower storage I/O latency and up to 75% reduced storage I/O latency variability [...]"

This could explain the findings in the article - they only meared peak r/w, not predictability.

[0] https://perspectives.mvdirona.com/2019/02/aws-nitro-system/ [1] https://aws.amazon.com/ec2/nitro/ [2] https://d1.awsstatic.com/events/reinvent/2019/REPEAT_2_Power...

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by rowanG077 on Wednesday February 21, 2024 @02:00AM (#39450902)
Why are you acting as if PCIe switches are some secret technology? It was extremely grating for me to read your comments.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by stingraycharles on Wednesday February 21, 2024 @08:34AM (#39453547)
Because the parent works/worked for Google, so obviously it must be super secret sauce that nobody has heard of. /s

Next up they’re going explain to us that iSCSI wants us to think it’s SCSI but it’s actually not!

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by the-rc on Wednesday February 21, 2024 @08:51AM (#39453766)
Although it used them for years, the first mention by Google of PCIe switches was probably in the 2022 Aquila paper, which doesn't really talk about storage anyway...
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by rowanG077 on Wednesday February 21, 2024 @01:00PM (#39457173)
I don't understand why you would expect Google to state that. They have been standard technology for almost 2 decades. You don't see google claiming they use jtag or using SPI flash or whatever. It's just not special.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by the-rc on Wednesday February 21, 2024 @10:36PM (#39462921)
Google didn't invent the Clos network, either, but it took years before they started talking about its adoption and with what kind of proprietary twists. Same with power supplies. You're right, a PCIe switch is not special, unless maybe it's integrated in some unconventional way. It's in Google's DNA to be cagey by default on a lot of details, to avoid giving ideas to the competition. Or misleading others down rabbit holes, like with shipping container datacenters.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by sitkack on Thursday February 22, 2024 @07:29AM (#39466137)
No, it dismisses technology until it does a 180 and then pretends it innovated in ways everyone is too stupid to understand. Google exceptionalism 101.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by dekhn on Wednesday February 21, 2024 @01:39PM (#39457669)
Like many other people in this thread, I think we disagree that a PCI switch means that an SSD "is connected over a network" to the host bus.

Now if you can show me two or more hosts connected to a box of SSDs through a PCI switch (and some sort of cool tech for coordinating between the hosts), that's interesting.

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by 20after4 on Tuesday February 20, 2024 @12:58PM (#39444464)
If the SSD is installed in the host server, doesn't that still allow for it to be shared among many instances running on said host? I can imagine that a compute node has just a handful of SSDs and many hundreds of instances sharing the I/O bandwidth.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by discodave on Tuesday February 20, 2024 @01:07PM (#39444584)
If you have one of the metal instance types, then you get the whole host, e.g. i4i.metal:

https://aws.amazon.com/ec2/instance-types/i4i/

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by aeyes on Tuesday February 20, 2024 @01:22PM (#39444763)
On AWS yes, the older instances which I am familiar with had 900GB drives and they sliced that up into volumes of 600, 450, 300, 150, 75GB depending on instance size.

But they also tell you how much IOPS you get: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/stora...

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by queuebert on Tuesday February 20, 2024 @02:59PM (#39446130)
How do these machines manage the sharing of one local SSD across multiple VMs? Is there some wrapper around the I/O stack? Does it appear as a network share? Geniuinely curious...
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by felixg3 on Tuesday February 20, 2024 @03:07PM (#39446222)
Probably NVME namespaces [0]?

[0]: https://nvmexpress.org/resource/nvme-namespaces/

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by bravetraveler on Tuesday February 20, 2024 @05:54PM (#39448056)
Less fancy, quite often... at least on VPS providers [1]. They like to use reflinked files off the base images. This way they only store what differs.

1: Which is really a cloud without a certain degree of software defined networking/compute/storage/whatever.

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by magicalhippo on Tuesday February 20, 2024 @04:04PM (#39446886)
In say VirtualBox you can create a file backed on the physical disk, and attach it to the VM so the VM sees it as a NVMe drive.

In my experience this is also orders of magnitude slower that true direct access, ie PCIe pass-through, as all access has to pass through the VM storage driver and so could explain what is happening.

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by bravetraveler on Tuesday February 20, 2024 @05:56PM (#39448073)
The storage driver may have more impact on VBox. You can get very impressive results with 'virtio' on KVM
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by magicalhippo on Tuesday February 20, 2024 @07:15PM (#39448690)
Yeah I've yet to try that. I know I get a similar lack of performance with Bhyve (FreeBSD) using VirtIO, so it's not a given it's fast.

I have no idea how AWS run their VMs, was just saying a slow storage driver could give such results.

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by bravetraveler on Tuesday February 20, 2024 @08:24PM (#39449157)
> just saying a slow storage driver could give such results

Oh, absolutely - not to contest that! There's a whole lot of academia on 'para-virtualized' and so on in this light.

That's interesting to hear about FreeBSD; basically all of my experience has been with Linux/Windows.

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by dan-robertson on Tuesday February 20, 2024 @03:12PM (#39446276)
AWS have custom firmware for at least some of their SSDs, so could be that
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by icedchai on Tuesday February 20, 2024 @04:56PM (#39447488)
With Linux and KVM/QEMU, you can map an entire physical disk, disk partition, or file to a block device in the VM. For my own VM hosts, I use LVM and map a logical volume to the VM. I assumed cloud providers did something conceptually similar, only much more sophisticated.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by bravetraveler on Tuesday February 20, 2024 @05:58PM (#39448087)
Files with reflinks are a common choice, the main benefit being: only storing deltas. The base OS costs basically nothing

LVM/block like you suggest is a good idea. You'd be surprised how much access time is trimmed by skipping another filesystem like you'd have with a raw image file

[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by jethro_tell on Wednesday February 21, 2024 @10:41AM (#39455138)
Heh, you'd probably be surprised, there's some really cool cutting edge stuff being done in those data centers but a lot of what is done is just plan old standard server management without much in the way of tricks. Its just someone else does it instead of you and the billing department is counting milliseconds.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by icedchai on Wednesday February 21, 2024 @11:32AM (#39455931)
Do cloud providers document these internals anywhere? I'd love to read about that sort of thing.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by jethro_tell on Wednesday February 21, 2024 @12:45PM (#39456966)
Not generally, especially not the super generic stuff. Where they really excel is having the guy that wrote the kernel driver or hypervisor on staff. But a lot of it is just an automated version of what you'd do on a smaller scale
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by ownagefool on Tuesday February 20, 2024 @01:25PM (#39444820)
PCI bus, etc too
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by throwawaaarrgh on Tuesday February 20, 2024 @02:44PM (#39445938)
Instance storage is not networked. That's why it's there.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by Hewitt821 on Tuesday February 20, 2024 @08:53PM (#39449306)
Local SSD is part of the machine, not network attached.
[ Parent | on HN ]
Re:SSDs Have Become Fast, Except in the Cloud by reactordev on Wednesday February 21, 2024 @01:57AM (#39450882)
AWS is so large, every concept of hardware is virtualized over a software layer. “Instance storage” is no different. It’s just closer to the edge with your node. It’s not some box in a rack where some AWS tech slots in an SSD. AWS has a hardware layer, but you’ll never see it.
[ Parent | on HN ]