That’s what Nvidia faces. Doesn’t matter how good the current in-house teams are using direct hardware, the trend in corporate is shift to a vendor (Google/AWS).
Nvidia can watch this inevitable shift or get ready to offer itself as a platform too.
AI graphics service that renders games in the cloud (photorealistic), concluding its epic journey of being an amazing graphics card company.
Kinda cool when you think about it.
It feels like reading that setting up something like Facebook would be extremely challenging for a company like SpaceX.
But the fattest profit margins are in selling to big corporations. Big corporations who already have accounts with the likes of AWS. They already have billing set up, and AWS provides every cloud service under the sun. Container registry? Logging database? Secret management? Private networks? Complicated-ass role management? Single-sign-on integration? SOC2/PCI/HIPAA compliance? A costs explorer with a full API? Everything a growing bureaucracy could need. Getting your GPU VMs from your existing cloud provider is the path of least resistance.
The smaller providers often compete by having lower prices - but competing on cost isn't generally a route to fat profit margins. And will folks at big corporations care that you're 30% cheaper, when they're not spending their own money?
nvidia could definitely launch a focused cloud product, that competes on price - but would they be happy doing that? If they want to get into the business of offering everything from SAML logon integration to a managed labelling workforce with folks fluent in 8 languages - that could be a great deal of work.
To give an example in a closer domain: Look at how long Google lost money on cloud services through 2022 (over $15B in loses), and now only makes money by creative accounting (bundling "cloud services" together versus breaking out GCP; Microsoft does something similar with Office 365 and Azure).
Like many potential customers, I would not consider GCP because:
1) Google "support" is a buggy, automated algorithm which randomly thwacks customers on the head
2) Google randomly discontinues products
3) I've seen a half-dozen to a dozen instances where buying from Google was penny-wise and pound-foolish, and so have many other engineers I've worked with.
Google's overall attitude is that I'm a statistic defined by my value to Google. Google can and will externalize costs onto me. That attitude is 100% right for adwords and search, which are defined by margins, but not for something like GCP. If I am going with a cloud service/platform, I'll go with Amazon, Microsoft, or just about anyone else, for that matter.
That's not that Google is a bad company. Google actually did have the skill set to build the software and data centers for a very, very good cloud provider. It's just that Google's core competencies lie very far from providing reliable service to customers, customer support, or all the things which go into providing me with stability and business continuity.
"Fixing" this would require a wholesale culture, value, and attitude change, and developing a core competency very far from what Google is good at.
I put "fixing" in quotes since if you develop too many core competencies, you usually stop being good at any of them. Focus is important, and there's a reason many businesses spin out units outside of their domains of focus. If Google is able to become good at this, but in the process loses their edge in their current core competencies, that's probably a bad deal.
FWIW: I haven't yet formed an opinion on NVidia's cloud strategy. However, their core competencies appear to be very much in the "hard" domains like silicon, digital design, machine learning rather than "soft" ones. Another relevant example for what can happen when hard skills are de-emphasized at engineering-driven companies is Boeing (if you've been following recent stories; if not, watch a documentary).
One is a publicly listed business with as much of an objective look at real time "worth" as possible in today's world, and the other is a private business with confidential financials.
Seems like you would be unable to even calculate SpaceX's net worth, much less compare them to a business with the most objective measure of "worth".
A private investment at this scale should have a lot more transparency and due diligence than disclosures from a SEC disclosures. If I were investing $750M, I'd have engineers under NDA review SpaceX technologies, financial auditors, legal auditors, etc.
Secondary sales place it a little bit higher (but those typically have all the issues you describe).
What I've heard is Azure is a pain in the ass, and things take three times as long to set up there, for some reason. There's also Oracle cloud but you hear way less about them. out there.
> What I've heard is Azure is a pain in the ass, and things take three times as long to set up there, for some reason.
Doesn't matter. The cost here is a rounding error. What does matter is something like this:
https://developer.chrome.com/docs/extensions/develop/migrate...
https://workspaceupdates.googleblog.com/2021/05/Google-Docs-...
https://workspace.google.com/blog/product-announcements/elev...
https://www.tomsguide.com/news/g-suite-free-shutdown
Etc.
These sorts of behaviors take out whole swaths of businesses wholesale. It's random, and you never know when it will happen to you.
It's the difference between managing a classroom with:
- an annoying kid throwing spitballs every day (Azure)
- the quiet kid who, one day, brings an assault rifle, a few extra mags, and starts spraying bullets into the cafeteria (Google).
Yes, one is a constant source of annoyance, but really, it's very manageable when you consider the alternative.
Just because an organisation is extremely good at one thing doesn't mean it can easily apply that to another field. I would guess that SpaceX probably does have the talent on hand to throw together a Facebook clone, but equally I think they would struggle to actually complete with Facebook as a business at scale.
NVIDIA is not entering an empty marketplace here. Also there's already enough know-how to make things run on cloud, OpenStack, HPC, etc.
Unless they make things very difficult for independent platforms, they can't force their way in that much, from my perspective.
Now they do all re-sell Nvidia GPUs in their cloud businesses, but there the cost will be passed very directly on to infrastructure customers, who will see the competing higher level services from those cloud providers (hosted models) likely at lower or competitive prices, and it's going to be harder to justify renting CUDA cores for custom software.
NVIDIA could voluntarily open the standard to avoid this ligitation if they wanted to, though, and IMO it would be the smart thing to do, but almost every corporation in history has chosen the ligitation instead.
If AMD isn't a competitor before government intervention, I don't the government forcing nvidia to open up CUDA changes much. CUDA's moat isn't due to some secret sauce - nvidia put in the developer hours; and if AMDs CUDA implementation is still broken, people will continue to buy nvidia.
There has been a lot of trying to get AMD to work - Hotz has been trying for a while now[1] and has been uncovering a ton of bugs in AMD drivers. To AMD's credit, they have been fixed, but it does give you a sense of how far behind they are in regards to their own software. Now imagine them trying to implement a competitor's spec?
[1] https://twitter.com/__tinygrad__/status/1765085827946942923
The issue that AMD has is they had a long period where they clearly had no idea what they were doing. You could tell just from looking at websites, CUDA pretty much immediately gets to "here is a library for FFT", "here is a library for sparse matricies". AMD would explain that ROCM is an abbreviation of the ROCm Software platform or something unspeakably stupid. And that your graphics card wasn't supported.
That changed a few months ago; so it looks like they have put some competent PMs in the chair now or something. But it'll take months for the flow on effects to reach the market. They have to figure out what the problems are which takes months to do properly; then fix the software (1-3 months more minimum); then get it into the open and the foundational libraries like PyTorch pick it up (might take another year). You can speed that up, but more cooks in the kitchen is not the way. Bandwidth use needs to be optimised.
It isn't like ROCm seems lacks key features; it can technically do inference and training. My card crashes regularly though (might be a VRAM issue) so it is useless in practice. AMD can check boxes but the software doesn't really work and grappling with that organisationally is hard. Unless you have the right people in the right places, which AMD didn't have up to at least mid 2023.
Pytorch has been "supporting" rocm for all last 2 years
It makes perfect sense that, organisationally, they were focused on that battle. If you remember the Athlon days, AMD beat Intel before, but briefly. It didn't last. This time it looks like they beat Intel and have had the focus to stay. Intel will come back and beat them some cycles, but there is no collapse on the horizon.
So it makes sense that they started looking at nVidia in the last year or so. Of course nVidia has amassed an obscene war chest in the meantime...
If even that. A few years ago they managed to break basic machine learning code on the few commonly-used consumer GPUs that were officially supported at the time, and it was only after several months of more or less radio silence on the bug report and several releases that they declared those GPUs were no longer officially supported and they'd be closing the bug report: https://github.com/ROCm/ROCm/issues/1265
If they paid everyone $1M/year salaries maybe more people would consider going into aerospace engineering.
Right now though Boeing's starting salaries aren't that much higher than what an Uber driver in the bay area makes.
If it was a clean room implementation of the API NVIDIA wouldn’t care. Heck that’s exactly what AMD did with HIP.
But what you cannot do is essentially intercept calls to and reverse engineer NVIDIA binaries in real time because you can’t be arsed to build your own.
And this is precisely what anti-trust ligitation would allow them to do.
Preventing someone from reverse-engineering a product with the sole intention of maintaining monopoly status may be seen as anti-competitive.
AMD makes really, really good CPUs now, but only after ligitation against Intel allowed them to keep up with evolving x86 standards.
It's not about being "arsed" to build your own, the problem is NVIDIA can add to CUDA at any point in time and launch a new GPU at the same time and the market will buy them because the cutting edges of the ecosystem are all CUDA-based, so people are forced to buy NVIDIA if they want to stay on that cutting edge and not wait a year for AMD to reverse engineer it.
What ZULDA did wasn’t to maintain a compatibility with the CUDA API and provided an open implementation of it but rather use all the CUDA based libraries that NVIDIA provided on top of it.
The equivalence again would be that not only Google implemented their own Java compatible API but rather that they used the now Oracle owned JVM to do so and redistributed it.
AMD already implemented CUDA essentially one to one in the form of HIP in ROCm. The issue that they face is that they don't have all the equivalent middleware to make copy pasting code actually work and this was what ZULDA did but instead of building a HIPdnn they just reused NVIDIA binaries.
The problem currently, as people like Hotz and many others are discovering, it not the lack of CUDA. Most people use PyTorch and don't care what the underlying software is. Infact most CUDA is hand tuned to nvidia hardware anyways and is optimized to make the most on nvidia. The problem is AMD's drivers - the piece that actually sends the code to run on the GPU, tends to be broken. AMD cannot "sponsor" an outsider to fix this. A legal, but broken, AMDCUDA will not be any better than the current situation; so no, having CUDA on AMD wouldn't change anything.
The problem is not "CUDA is not AMD", the problem is AMD has not, does not, and for some reason will not invest adequately in GPU compute. CUDA is a mirage; if AMD had a similar platform someone would have done the work already to ensure PyTorch works on it. PyTorch already supports ROCm, people don't use it because the performance is bad and it's buggy. When nvidia had this problem, nvidia hired engineers to work on open source projects and debug issues in open source libraries (not even limited to AI, you will find nvidia engineers debugging issues in a wide range of CUDA projects). When AMD has this issue, they barely acknowledge it.
CUDA is such a misnomer. Amd doesn't have tensorRT, cuDNN, cutlass, etc. Forcing Nvidia to make these work on AMD is like forcing Microsoft to make windows work on apple hardware... Not going to happen.
You can absolutely use the names of the functions and the programming model. Like I said, HIP is literally a copy. Llama.cpp changes to HIP with a #define, because llama.cpp has its own set of custom kernels.
And this is what I've said before, CUDA is hardly a moat. The API is well-known and already implemented by AMD. It's all the surrounding work: the thousands of custom (really fast!) kernels. The ease-of-use of the SDKs. The 'pre-built libraries for every use case'. You can claim that CUDA should be made open-source for competition, but all those libraries and supporting SDKs represent real work done by real engineers, not just designing a platform, but making the platform work. I don't see why NVIDIA should be compelled to give those away anymore than Microsoft should be compelled to support device driver development on linux.
The MI300 smokes the H100 yet here we are.
Microsoft has partnetship with OpenAI and also with Mistral.
Present convenience may not hold true in future. Nvidia knows that well.
Even if AWS has its own hardware+software solution for neural networks it would still take years if not decades to tear off the CUDA platform
I can understand from a growth perspective why it’s more profitable for Nvidia if it can become more of a platform service for AI. However, that’s difficult to balance that and partnerships the company already has with AWS and Microsoft. I’d expect to see some acquisitions or competing custom solutions in the future. Fortunately for Nvidia, a lot of AI is still dependent on CUDA. I’m interested to see how this plays out.