AMD has had poor management in the GPU computing space since Raja Koduri's time (he put the best engineering resources on VR during his tenure and ignored deep learning). Subsequent directors have not had a long term vision and left with a few years for greener pastures.
Looks like Lisa Su has corrected this now - they seem to have moved AMD software engineers en masse to work under Xilinx management on AI. Remains to be seen if this new management hierarchy will have a better vision and customer focus.
Perf isn't the biggest problem though, many AI chips can do this or a bit better on benchmarks, if you invest the engineering time to tune the benchmark.
The really hard part is getting a complete software stack running.
It took us over 3 years because many of the layers just didn't exist, e.g. scale out LLM inference service that supports multiple requests with fine-grained batching across models distributed over multiple GPUs.
On Instinct, ROCm gets you the ability to run most pytorch models on one GPU assuming you get the right drivers, compilers, framework builds, etc.
That's a good start, but you need more to serve a real application.
The difference is that it didn't get a lot of love as part of the overhyped python LLM movement.
We are working on it, but it takes time.
Contributions to foundational layers like ROCBlas, pytorch, slurm, Tensile, huggingface, etc would help.
* Drives down price
* Enhances product features (I see them competing on VRAM first)
* Helps to insulates buyers from supply issues
Nvidia has kneecapped their consumer grade hardware to ensure the gaming market still has scraps to buy in spite of crypto mining and the AI gold rush. All AMD would have to do to eat into Nvidia marketshare is remove the hardware locks in low-end cards and ship one with 64GB+ of VRAM.
This of course would only work if they have comparable/usable software support. Any improvements to ROCm will be a boon for any company that doesn't already have or can't afford huge farms of high-end Nvidia chips.
More than 10x cheaper than allocating machines on a tier 1 cloud - AWS, Azure, GCP, Oracle, etc
More memory - 128GB HBM per GPU - means bigger models fit for training/inference without the nightmare of model parallelism over MPI/infiniband/etc
Longer term - finetuning optimizations
Does AMD provide something similar to nvlink, and even libraries like cudnn?
Also, last I checked none of the public clouds offered any of the latest gens MI GPUs, so I wasn't aware that it had good availability! Azure had a preview but I'll look more into it now.
Thank you for your answer btw!
On the plus side, it was drastically cheaper and now we can just slot in machines.
I would prefer that a tier 1 cloud made MI GPUs available though. It would make it so much more accessible.
We will open source pieces of it over time. Our strategy is open source functional core. Eventually we will have an open source dev environment that runs on a personal scale computer. We already have this for some configurations, but we don't do enough testing to ensure perf/functionality on many different systems.
We are mainly bottlenecked by resources as a 12-person startup.
We have released some open source SDKs here:
https://github.com/orgs/lamini-ai
This class has some training recipe code:
https://www.deeplearning.ai/short-courses/finetuning-large-l...
One thing I'd like to push back to open source is the scale out AMD SLURM support.
I'm really really worried about AMD, and whether they're going to care about anyone else. They might just care about Instinct, where margins are so high, and ignore consumer cards or making more friction and segmentation for consumer cards.
Part of what made CUDA so successful was that the low hardware barrier to entry created such a popular offering. Everyone used it. I really hope AMD realizes that, and really hope AMD invests in consumer card software too. Just making it work on the high end doesn't seem enough to get the kind of mass-movement ecosystem success AMD really needs. I'm afraid they might go for a smaller win, try to compete only at the top.
Lisa tweet: https://x.com/LisaSu/status/1706707561809105331?s=20
Lamini tweet: https://x.com/realSharonZhou/status/1706701693684154766?s=20
Blog: https://www.lamini.ai/blog/lamini-amd-paving-the-road-to-gpu...
Register: https://www.theregister.com/2023/09/26/amd_instinct_ai_lamin... CRN: https://www.crn.com/news/components-peripherals/llm-startup-...
The hard part about using any AI Chips other than NVIDIA has been software. ROCm is finally at the point where it can train and deploy LLMs like Llama 2 in production.
If you want to try this out, one big issue is that software support is hugely different on Instinct vs Radeon. I think AMD will fix this eventually, but today you need to use Instinct.
We will post more information explaining how this works in the next few weeks.
The middle section of the blog post above includes some details including GEMM/memcpy performance, and some of the software layers that we needed to write to run on AMD.