What we learned about SSDs in 2015

Re:What we learned about SSDs in 2015 by wtallis on Sunday December 20, 2015 @01:16PM (#10767737)

M.2 slots don't make much sense for servers, at least if you're trying to take advantage of the performance benefits possible with a PCIe interface. Current M.2 drives aren't even close to saturating the PCIe 3.0 x4 link but they're severely thermally limited for sustained use and they're restricted in capacity due to lack of PCB area. Server SSDs should stick with the traditional half-height add-in card form factor with nice big heatsinks.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by lsc on Sunday December 20, 2015 @02:23PM (#10767956)

most of the NVMe backplanes I've seen give full 'enterprise 2.5" drive clearance to the thing, so if they are actually as thick as consumer SSDs, as most current SATA 'enterprise SSD' are, there's plenty of room for a heatsink without expanding the slot. The supermicro chassis (and I've only explored the NVMe backplanes from supermicro) usually put a lot of effort into drawing air through the drives, so assuming you put in blanks and stuff, the airflow should be there, if the SSD are setup to take advantage of it.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by wtallis on Sunday December 20, 2015 @03:09PM (#10768115)

You need to be more precise with your terminology. There is no such thing as a NVMe backplane. NVMe is a software protocol. The backplanes for 2.5" SSDs to which you refer would be PCIe backplanes using the SFF-8639 aka U.2 connector. None of the above is synonymous with the M.2 connector/form factor standard, which is what I was talking about.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by lsc on Sunday December 20, 2015 @03:11PM (#10768120)

edit: okay, I re-read what you said and yes, these won't support M.2 drives, if I understand what's going on here, and it's possible I still don't. (I have yet to buy any non-sata SSD, though I will soon be making experimental purchases.)

I was talking about these:

http://www.supermicro.com/products/nfo/NVMe.cfm

Note, though, it looks like if you are willing to pay for a U.2 connected drive, you can get 'em with the giant heatsinks you want:

http://www.pcper.com/news/Storage/Connector-Formerly-Known-S...

further edit:

http://www.intel.com/content/dam/www/public/us/en/documents/...

available; not super cheap, but I'm not sure you'd want the super cheap consumer grade stuff in a server, anyhow.

further edit:

but I object to the idea of putting SSDs on PCI-e cards for any but disposable "cloud" type servers (unless they are massively more reliable than any that I've seen, which I don't think is the case here.) just because with a U.2 connected hard drive in a U.2 backplane, I can swap a bad drive like I would swap a bad sata drive; an alert goes off and I head off to the co-lo as soon as convenient and I can swap the drive without disturbing users, whereas with a low-profile PCI-e card, I've pretty much gotta shut down the server, de-rack it, then make the swap, which causes downtime that must be scheduled, even if I have enough redundancy that there isn't any data loss.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by wtallis on Sunday December 20, 2015 @04:24PM (#10768352)

Take a look at how much stricter the temperature and airflow requirements are for Intel's 2.5" U.2 drives compared to their add-in card counterparts. (And note that the U.2 drives are twice as thick as most SATA drives.)

M.2 has almost no place in the server market. U.2 does and will for the foreseeable future, but I'm not sure that it can serve the high-performance segment for long. It's not clear whether it will reach the limits on capacity, heat, or link speed first, but all of those limits are clearly much closer than for add-in cards.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by lsc on Sunday December 20, 2015 @05:37PM (#10768584)

>M.2 has almost no place in the server market. U.2 does and will for the foreseeable future, but I'm not sure that it can serve the high-performance segment for long. It's not clear whether it will reach the limits on capacity, heat, or link speed first, but all of those limits are clearly much closer than for add-in cards.

No argument on m.2 - it's a consumer grade technology. No doubt, someone in the "cloud" space will try it... I mean, if you rely on "ephemeral disk" - well, this is just "ephemeral disk" that goes funny sooner than spinning disk.

But the problem remains, If your servers aren't disposable, if your servers can't just go away at a moment's notice, the form factor of add-in cards is going to be a problem for you, unless the add-in cards are massively more reliable than I think they are. Taking down a whole server to replace a failed disk is a no-go on most non-cloud applications...

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by kijiki on Sunday December 20, 2015 @06:31PM (#10768774)

You're probably tired of hearing this from me, but if you distribute the storage, you can evacuate all the VMs off a host, down it, do whatever, bring it back up, and then unevacuate.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by lsc on Sunday December 20, 2015 @08:56PM (#10769152)

> You're probably tired of hearing this from me, but if you distribute the storage, you can evacuate all the VMs off a host, down it, do whatever, bring it back up, and then unevacuate.

Yes. The same conversation is happening right now in the living room, because security has forced three reboots in the last year, after having several years where simply by being xen and pv and using pvgrub (rather than loading a user kernel from the dom0) we haven't been vulnerable to the known privilege escalations. This is a lot more labor (and customer pain) when you can't move people.

No progress on that front yet, though.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by wtallis on Sunday December 20, 2015 @08:03PM (#10769012)

And if uptime is that important, you can just buy a server that supports PCIe hotswap.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by lsc on Sunday December 20, 2015 @08:32PM (#10769093)

can you point me at a chassis designed for that?

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by wmf on Sunday December 20, 2015 @06:11PM (#10768700)

OTOH if you can fit 24 U.2s but only 6 AICs in a server, maybe RAIDed U.2 will still win.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by wtallis on Sunday December 20, 2015 @07:58PM (#10768994)

Only if you postulate that your drives won't be able to saturate their respective number of PCIe lanes. The total number of lanes in the system is fixed by the choice of CPU, and having drives with a total lane count in excess of that only helps when your drives can't fully use the lanes they're given or if you're trying to maximize capacity rather than performance.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by scurvy on Sunday December 20, 2015 @01:57PM (#10767873)

2.5" NVMe is fine and more scalable than add-in cards (more slots) with increased serviceability. The equivalent add-in card is usually much more expensive than the 2.5" version.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by djsumdog on Sunday December 20, 2015 @06:45PM (#10768810)

I thought this was the point of SATA Expresses, but the last I read, SATA Express actually holds on par and even falls behind M.2.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by tw04 on Sunday December 20, 2015 @01:38PM (#10767795)

How exactly does connecting to the PCIe bus obviate the need for RAID? RAID isn't about connecting drives, it's about not losing data. If you don't need RAID for your application today, you can purchase a non-RAID SAS adapter for a couple hundred bucks, or just use the onboard ports that are sure to be there.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by tehwalrus on Sunday December 20, 2015 @01:55PM (#10767867)

> RAID isn't about connecting drives, it's about not losing data.

RAID-0 is used as a way to get faster performance from spinning disk drives, as you can return parts of each read request from different (striped) drives.

You also get better write performance, as your writes are split across the drives.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by lsc on Sunday December 20, 2015 @02:19PM (#10767945)

uh, this shouldn't be voted down. Now, I don't use RAID for performance, I use raid for reliability, like parent said, and I've never actually been in a position where it would make sense, but people do use raid0 to increase performance. It happens, even if it's not nearly as common as using raid to prevent data loss.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by toomuchtodo on Sunday December 20, 2015 @03:19PM (#10768157)

Back in the day, folks would RAID0 AWS EBS volumes to get the level of performance they needed (this was long before SSD EBS volumes).

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by bigiain on Sunday December 20, 2015 @05:44PM (#10768611)

Now you're making me feel old - when "back in the day" and "AWS" get used in the same sentence... :-)

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by MichaelGG on Sunday December 20, 2015 @03:43PM (#10768232)

This is still (AFAIK) the recommended way to get higher IOPS out of Azure SSDs.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by mirimir on Sunday December 20, 2015 @06:06PM (#10768683)

I like RAID10. It's almost as reliable as RAID1, and you get about half the capacity and performance boost of RAID0. With four drives, RAID10 gives you the same capacity as RAID6. It's only ~30% slower, and it rebuilds much faster after drive failure and replacement. With more drives, you get 100% of added drive capacity for RAID6 vs 50% for RAID10, but rebuild time gets crazy huge.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by burntwater on Sunday December 20, 2015 @04:16PM (#10768327)

High-end video servers used in the entertainment industry typically use RAID 0. My last project, for example, used 10 250GB SSDs in RAID 0.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by ersii on Sunday December 20, 2015 @05:37PM (#10768586)

What kind of video servers are you talking about? For ingesting from a feed/camera in? Streaming out video? Curious.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by burntwater on Sunday December 20, 2015 @11:34PM (#10769594)

While these servers can ingest video feeds, they wouldn't (typically) be saving the incoming feeds. These servers play out video over many projectors blended into a single seamless image. If you've watched the past few Olympics opening ceremonies, you most certainly saw the projections.

My recent project, which was relatively small, played back a single video over 4 blended projectors. Each video frame was a 50MB uncompressed TGA file, 30 times a second. On a more complex show, you could be trying to play back multiple video streams simultaneously.

D3 - http://www.d3technologies.com/ and Pandora - http://www.coolux.de/products/pandoras-box-server/

are two of the big players in the industry.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by pjc50 on Monday December 21, 2015 @05:53AM (#10770540)

Each video frame was a 50MB uncompressed TGA file, 30 times a second.

Ouch! Not even a PNG?

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by burntwater on Tuesday December 22, 2015 @03:48PM (#10780306)

Uncompressed TGA is used for two reasons: Quality (this particular image was going on a screen three stories tall) and, as the other commenter mentioned, the time required to decompress images.

I'm not an expert on the exact architecture, but I'm guessing that with uncompressed TGA you just throw the bits at the GPU and they get displayed, while if you have to uncompress images that first gets handled by the CPU (?).

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by tracker1 on Monday December 21, 2015 @12:59PM (#10772347)

I'd guess the time to do compression from TGA to PNG may be more than 1/30th of a second. Or at least reliably/consistently so.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by bigiain on Sunday December 20, 2015 @07:18PM (#10768897)

A friend of mine does a lot of programming and installations using Dataton Watchout - where they use 4 or more 4K projectors stitched together into one big image. He's regularly got war stories about problems with the latest batch of SSDs not playing well in RAID0 causing stutters on simultaneous playback of multiple 4K video streams.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by im2w1l on Monday December 21, 2015 @03:29AM (#10770183)

Is there an advantage compared to putting each stream on a different drive?

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by burntwater on Tuesday December 22, 2015 @03:49PM (#10780312)

It's possible, and probably advantageous to a point. Eventually you'll hit bottlenecks somewhere, at which point you throw more servers at it.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by otakucode on Monday December 21, 2015 @03:27AM (#10770179)

Sure. My first purchase of SSDs were 3 80GB Intel SSDs which I put into RAID-0 and used for my primary system drive on my gaming machine. RAID-0 provides very-near 100% performance boost per drive added. Of course, it also provides the same bonuses to the likelihood of total data loss... but that was a risk I was OK with taking (and which never bit me!).

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by tracker1 on Monday December 21, 2015 @01:01PM (#10772359)

Mine was the same drive, my first SSD was the first 80GB Intel one, and just over a year of use, it started reporting itself as an 8MB drive. You never know for sure what will remain good, or be good consistently.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by dajonker on Sunday December 20, 2015 @02:21PM (#10767952)

RAID 0 does not have any level of redundancy, so you might as well remove the 'R' in RAID and replace it by 'S' for striping or something. However, people would probably get confused if you start calling it SAID.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by bigiain on Sunday December 20, 2015 @07:23PM (#10768910)

Complaining about industry-standard terminology like this is mostly a waste of time.

The sort of people who need to know that but don't aren't going to learn it from a post on HN, and the sort of people who'll read it on HN mostly don't need to be told.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by mirimir on Sunday December 20, 2015 @11:05PM (#10769519)

Sad but true. I gather that RAID0 has been the default for notebooks with two drives. And for the first 1TB MyBooks.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by tracker1 on Monday December 21, 2015 @01:02PM (#10772368)

0 == no redundancy...

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by fulafel on Monday December 21, 2015 @05:17AM (#10770427)

> RAID [...] it's about not losing data

It's a pretty ineffective way to guard against data loss, you need backups.

RAID [1-6] is an availability solution, saving the downtime of restoring from backup in the case of the most predictable disk failures. It doesn't help with all the other cases of data loss.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by derefr on Sunday December 20, 2015 @04:55PM (#10768427)

If we can connect SSDs directly to the PCIe bus... why are there no cute little Thunderbolt flash drives that run at full NVMe speed?

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by wtallis on Sunday December 20, 2015 @05:11PM (#10768489)

Windows has had astoundingly bad Thunderbolt hot-plug support given that ExpressCard and CardBus existed. NVMe support for non-Apple SSDs under OS X can be problematic. Thunderbolt PHYs are expensive. USB 3.0 is fast enough for now, and USB 3.1 exists.

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by something2324 on Monday December 21, 2015 @12:53AM (#10769793)

This is a bit of a question from my lack of understanding of disk IO:

But would this in practice play well with the CPU prefetcher? If you're crunching sequential data can you expect the data in the L1 cache after the initial stall?

[ Parent | on HN ]

Re:What we learned about SSDs in 2015 by dragontamer on Monday December 21, 2015 @02:04AM (#10769978)

SSDs are still grossly slower than RAM. The fastest SSD I know of is the Intel 750, which is like ~2.0 GigaBYTES/second (or for what is more typical for storage benchmarks: about 16Gbps or so over PCIe x4).

Main DDR3 RAM is something like 32GigaBYTES per second, and L1 cache is even further.

What I think the poster was talking about, is moving from disk-based databases to SSD-based databases. SSDs are much faster than hard drives.

L1 Cache, L2 Cache, L3 Cache, and main memory are all orders of magnitude faster than even the fastest SSDs today. Thinking about the "CPU prefetcher" when we're talking about SSDs or Hard Drives is almost irrelevant due to the magnitudes of speed difference.

[ Parent | on HN ]