In the end, the RTX 3090 is still better at ML workloads than every card other than the RTX A6000 or the A100. Although the 3090 theoretically falls behind the V100, Titan V and Titan RTX in terms of pure mixed-precision matrix math, in practice the 3090 still performs better on almost all workloads.

So maybe not 100% designed for ML, but better than anything else out there unless you want to sell a kidney.

Because in reality, it’s only gimped for certain scenarios, and compared to the 2080TI, it’s a monster upgrade for ML!

Here’s a dirty secret: there is a ton of ML prototyping done on GeForce level cards, and not just by enthusiasts or at scrappy startups. You’ll find GeForce level cards used for ML development in workstations at Fortune 50 companies. NVIDIA would love everyone to be using A100s to do their ML work (and V100s before that,) but the market isn’t in sync with that wish. The 2080TI remains an incredibly popular card for ML even with only 11GB. Upping to 24GB, even with the artificial performance limitations for certain use-cases, enables new development opportunities and use-cases to explore.

When it comes to product stratification, the hard rule according to driver EULAs is that GeForce cards can’t be used in data centers. For serious ML development at scale, NVIDIA has their DGX lineup. In the middle are the Quadro cards, but they tend to be a poor value for ML. The cost differential with Quadro is largely due to optimizations and driver certification for use with tools like Catia or Creo (CAD/CAM use,) which don’t intersect with ML.

The Titan RTX may not have the gimped drivers, but the 3090 beats the Titan in many benchmarks nonetheless. Is the 3090 the best NVIDIA PCIe form factor card for ML? No. The A100 is still king of the crop and is the only Ampere card with HBM memory, and even the A6000 will outperform for many use-cases with 48GB of RAM. Still, the 3090 will be the optimal card for many.

I’m one of the lucky few to have a 3090 in my rig. I lead of team of volunteers doing critical AI prototyping and POC work in an industry give-back initiative, and price was not a leading factor in my decision to procure a 3090 over a Quadro. I chose the 3090 principally because I didn’t want a loud blower card in my computer (and I don’t need 48GB.) If someone donated an A100 to our efforts, I’d gladly take it, but it wouldn’t replace the 3090. It’s not a graphics card and it won’t play games, which indeed is an important value-added benefit of the 3090 :)

> gimped ML functionality

curious what is that?

The 3090 supports but doesn't let you enable the titan driver paths, SR-IOV, NGX, and critically, is capped to half-rate tensor operations with FP32 accumulate.

An official firmware release could change this but they're likely saving it for the Titan cards down the road.

Where do you see that this is artificial (i.e. in software/firmware) rather than just missing some hardware that's present on the Titan/Quadro cards?

The same thing happened with the 2080 / Titan and looks to be confirmed for the 3000 series by AnandTech. https://twitter.com/RyanSmithAT/status/1301996479448457216

The Reddit AMA implies the Ampere architecture supports this configuration but is software limited under the FP32 section: https://www.reddit.com/r/nvidia/comments/ilhao8/nvidia_rtx_3...

To epistemologically correct, I don't believe one will ever see a statement from NVIDIA confirming that this is a software limitation. I believe it's just inferred.

edit: This is close to confirmation: https://imgur.com/a/RH8vyz9 -though I may suspect there are non reversible hardware fuses at play too.

They also did this with the TITAN in the past, I think, so it's not entirely unprecedented.

TITAN X got access to pro drivers shortly after AMD announced the Vega Frontier Edition.

Maybe it is a type of binning. Maybe the machine learning specific hardware is more prone to errors so they just disable that on subpar chips and sell them for use in graphics cards?

The AMA section on FP32 seems to imply it's in hardware though, unless I'm missing something – datapath is a hardware term, at least.

I mean, I really hope that it's just in the driver so that an enterprising reverse engineer can hack the driver and re-enable full FP16/FP32 accumulate :)