>> We are interested in feedback on what people want and can be agile enough to customize our purchases for your needs.

I would like you to not by pallets of these chips so that I and others may be able to get our hands on a couple for our own personal projects.

It was very difficult for me to get access to these chips and I'm building a whole business around it. It won't be possible for individuals for a long long time, if ever. Maybe in a few years on the secondary market.

Case in point... try searching ebay for the previous generation... MI250.

If you want access to these chips, you're going to have to rent them. Given that nobody is really renting them currently, at least I'm working to get you access to them at all.

Was it difficult though because you lack the credentials of an established large corporation, or because there is very low stock at this point?

If NVIDIA has 98% of the market, I don't think stock is an issue. ;-)

The chip industry runs on relationships.

What’s to stop the government from taking your business from you for national security reasons?

Correct me if I'm wrong, but I don't think the US government can take over a business like that. They can take the land where we host the equipment, though if that happens, I guess I'd just move it somewhere else?

I did personally sign some documentation (EAR) saying that I wouldn't export the equipment to a whole list of countries and I definitely won't do that. I also won't rent to anyone in those countries either, just to cover my bases. I have high ethics and have no desire to get in any sort of trouble.

He's not in China

Very very true.

"needs work" is an understatement. ROCM only just came out for windows(!) and so far everyone in the LLM and SD scene says that ROCM on linux is a pain in the ass to get working, requires a lot of hackery and override environment variables, is buggy, and also only supports a small handful of cards - whereas CUDA runs on nearly everything NVIDIA makes, from laptop GPUs to datacenter compute cards. And there's very little support on the third party software side at the moment.

That "no consumer hardware support" sounds stupid, but people getting into ML/AI, grad students, etc want to be able to mess around and develop/prototype on local hardware they already own.

You cannot do that with ROCM, because they only support a very, very small number of consumer-level cards - and very oddly, it's a mix of the very high end and low end, nothing in the middle.

AMD is also massively behind hardware-wise with only the current 7xxx series cards that just came out having AI-specific hardware, whereas NVIDIA has had tensor cores for three generations of their cards.

But of course AMD can't do anything right, so they nerfed the GPU's graphics processing cores when they added the AI stuff, making the cards a worse deal from a pure gaming standpoint. The 7000 series cars are basically a tiny bump at best, and in many games worse, than the 6000 series equivalents. Their only advantage is better hardware video encoding support and somewhat better power consumption.

All of what you're saying is true and I can't argue that.

The only difference is that previously you couldn't even rent time on these super high end AIA's. They were reserved for supercomputers only. Even the MI250x sku is government/research only and even I am unable to buy that over the MI250.

So, we need to work on building that developer flywheel. My view is that if you can rent some time on one of these systems, that is a good first step. Nowhere else can you load a model into 192gb of RAM on a top end system.

For AI you have to support PyTorch. That works well now on the big AMD GPUs. Consumer gpu capabilities do not matter here.

I think that PyTorch is part of the puzzle and it certainly helps that it is supported by AMD [0]. That said, there is code that needs to run closer to the metal too.

[0] https://pytorch.org/blog/experience-power-pytorch-2.0/

that's awesome! what support are you giving for the rewriting that has to happen for things to run on ROCm or whatever instead of CUDA?

Thanks! Because we are just starting out, in order to stay lean, we are focused on the hardware side at this point. The capex/opex involved with deploying this level of compute is rather insane and I feel that it will be too distracting to solve both problems at the same time.

I do fully recognize that in order to build the developer flywheel, we need to solve both problems in the long run. For now, there are other great companies that are focused on solving the software side, such as NeuralFabric, EmbeddedLLM, Lamini and MK1. Our intention is to partner and be friends with all of them. There is a lot of room in the picks/shovels ecosystem for all of us.

Exciting news. How's software story on this? Currently we are on CUDA, etc. We have nice big AMD Epycs (1GB of cache!) but use Nvidia for GPU. Mi300x at $40k would be tempting, but hard to justify with porting effort required.

I would start to look into using ROCm/HIP to port the code. Some of it can be done automatically and some will need customizations [0]. Then, you rent time on my systems and see what the performance is. If it is good, you rent more, for longer to bring the pricing down. Effectively, we are taking the capex/opex risk off your shoulders. The key point here is that we have actual availability and we aren't just reselling someone else's hardware.

Dual 9754's are indeed quite nice. ;-)

[0] https://github.com/ROCm/HIPIFY

Thank you for the link and advice. We're using 9684X for the cache. But dual 9754 does sound good!

Oh yea, great chip as well. That cache certainly is expensive! Can I ask... what is the usecase for the cache?

We use the chip general purpose. HFT simulation code, empirically faster.

The GPUs are to train a model. Separate use-case, but we use same hardware for both use-cases so that's why the big cache thing.