While NVIDIA provides the best absolute performance for training, Intel (i.e. Gaudi) already provides much better training performance per dollar.

The funny thing is that this fact has been shown inadvertently by NVIDIA:

https://www.servethehome.com/nvidia-shows-intel-gaudi2-is-4x...

That article shows that it takes about 50x as long to train gpt-3 with intel's offering vs Nvidia. At least in the current environment, if you are training llms I think almost no amount of cost savings can justify that.

That 50X is only if you can afford one thousand NVIDIA H100.

There cannot be more than a handful of companies in the entire world that could afford such a huge price (tens of millions of $).

In comparison with a still extremely expensive cluster of 64 NVIDIA H100, the difference in speed would reduce to only two to three times, and paying several times less for the entire training becomes very attractive.

You’ve got a really warped sense of scale if you think only a handful of companies couldn’t afford tens of millions of dollars.

That’s complete pocket change for any of the Fortune 500.

The problem is not having so much money available.

Such a big expense only makes sense for a company where spending that amount would bring hundreds of millions of $ of additional revenue.

I doubt that any of the companies that have already spent such amounts have recovered even a small part of their expenses. It is more likely that they bet on future revenues, but it remains to be seen who will succeed to achieve that.

Kinda, companies of that scale regularly spend more than that on (often) random R&D.

Sure if there is a plausible ROI, they’d have no issues dropping that much money (actually far more). Revenues for fortune 500’s are going to be in the 10’s of billions anyway, and it wouldn’t be hard to make an argument that random AI project could increase that by a couple percent or decrease costs a couple percent, which would more than provide that ROI.

Their biggest issue is usually having anyone in leadership that has a clue enough to even propose something plausible, let alone get a team together to give it a plausible go.

If they have that, Capital is not the issue.

huggingface supports multi-node training?? https://huggingface.co/docs/optimum/habana/usage_guides/mult...

am i missing something here? just like you'd want to scale an h100 cluster out beyond one box of 8, you'd do the same for gaudi2?

Yeah no one really cares about this for large models. Iteration speed matters, and no one is waiting 50 times as long to train a LLM.

Intel has always published the "training per dollar" because no one else competes.

Even for fine tuning you are almost always better off getting smaller GPU cloud instances.

Amazon announced Trainum last year and recently announced their partnership with Anthropic, where Claude will be trained on Trainium.

https://press.aboutamazon.com/2023/9/amazon-and-anthropic-an...

Shouldn't inference be 99.999% of the compute cost over time? Especially for MS. Look how many Copilots they are cramming into their products

It's probably 25-75 for inference vs training. I estimate 10K tokens/month * 100M users/month = 1T tokens of inference/month. Training is 2-3 times more expensive, the training set is 15T tokens, so about equivalent to 45T tokens of inference. But the lifetime of a model ~1 year, they spend 45T tokens to train and 12T tokens for inference. So the inference cost is about 21% of the total cost.

This estimate is based on a conservative 10K tokens/user/month. In reality it could be much higher if they send PDFs or other input documents. That might increase the ratio maybe to 50-50, but in any estimate it's not 100:1.

The Maia chip Microsoft announced it's also for training.