Re:Nvidia gets 20% weighting, more investor demand; Apple de...
by hansvm on Monday June 17, 2024 @03:29PM
(#40710011)
> inference more expensive than training
That's a gamble. It's the whole idea behind a pre-trained model (GPT). If you make a really good model then you can use it a lot of times. That seems to be playing out as intended at the moment.
> very very very big matrix operation
To do efficiently, it's many slightly smaller very very big matrix operations.
> does it require advanced GPU capabilities
No, but GPUs don't have a lot of overhead for those very very big matrix operations.
> high costs because models are too large to fit on a single device
Engineer costs are high, and communication costs are also high. Those are mostly just barriers to entry though. Compute still dominates.
> competitors with more RAM
It's a little more complicated than that because extra RAM has speed-of-light delays and other such problems. If you don't expose a CUDA interface then you have a long hill to climb to get even a couple customers. A couple extra caches or other more esoteric memory APIs won't be easy for the industry to adopt (maybe a clever electrical engineer can fix that; I don't have hard proofs of those bounds).
That's a gamble. It's the whole idea behind a pre-trained model (GPT). If you make a really good model then you can use it a lot of times. That seems to be playing out as intended at the moment.
> very very very big matrix operation
To do efficiently, it's many slightly smaller very very big matrix operations.
> does it require advanced GPU capabilities
No, but GPUs don't have a lot of overhead for those very very big matrix operations.
> high costs because models are too large to fit on a single device
Engineer costs are high, and communication costs are also high. Those are mostly just barriers to entry though. Compute still dominates.
> competitors with more RAM
It's a little more complicated than that because extra RAM has speed-of-light delays and other such problems. If you don't expose a CUDA interface then you have a long hill to climb to get even a couple customers. A couple extra caches or other more esoteric memory APIs won't be easy for the industry to adopt (maybe a clever electrical engineer can fix that; I don't have hard proofs of those bounds).