Re:Intel CEO: 'The entire industry is motivated to eliminate...
by selfhoster11 on Thursday December 14, 2023 @06:20PM
(#38648817)
The amount of VRAM is the absolute killer USP for the current large AI model hobbyist segment. Something that had just as much VRAM as a 3090 but at half the speed and half the price would sell like hot cakes.
Re:Intel CEO: 'The entire industry is motivated to eliminate...
by eyegor on Thursday December 14, 2023 @08:21PM
(#38649920)
You are describing the ebay market for used nvidia tesla cards. The k80, p40, or m40 are widely available and sell for ~$100 with 24gb vram. The m10 even has 32gb! The problem for ai hobbyists is it won't take long to realize how many apis use the "optical flow" pathways and so on nvidia they'll only run at acceptable speeds on rtx hardware, assuming they run at all. Cuda versions are pinned to hardware to some extent.
Re:Intel CEO: 'The entire industry is motivated to eliminate...
by icelancer on Thursday December 14, 2023 @09:29PM
(#38650462)
Yep. I have a fleet of P40s that are good at what they do (Whisper ASR primarily) but anything even remotely new... nah. fp16 support is missing so you need P100 cards, and usually that means you are accepting 16GB of VRAM rather than 24GB.
Still some cool hardware.
Re:Intel CEO: 'The entire industry is motivated to eliminate...
by muxr on Friday December 15, 2023 @08:14AM
(#38653836)
For us hobbyists used 3090 or new 7900xtx seem to be the way. But even then you still need to build a machine with 3 or 4 of these GPUs to get enough VRAM to play with big models.
Re:Intel CEO: 'The entire industry is motivated to eliminate...
by icelancer on Friday December 15, 2023 @02:36PM
(#38657820)
For sure - our prod machine has 6x RTX 3090s on some old cirrascale hardware. But P40s are still good for last-gen models. Just nothing new unfortunately.
Re:Intel CEO: 'The entire industry is motivated to eliminate...
by selfhoster11 on Friday December 15, 2023 @03:15AM
(#38652112)
Out of these three, only P40 is worth the effort to get running vs the capabilities they offer. That's also before considering that other than software hacks or configuration tweaks, those cards require specialised cooling shrouds for adequate cooling in tower-style cases.
If your time or personal energy is worth >$0, these cards work out to much more than $100. And you can't even file the time burnt on getting them to run as any kind of transferable experience.
That's not to say I don't recommend getting them - I have a P4 family card and will get at least one more, but I'm not kidding myself that the use isn't very limited.
Re:Intel CEO: 'The entire industry is motivated to eliminate...
by icelancer on Friday December 15, 2023 @02:39PM
(#38657864)
The K80 is literally worth < $0 given how ridiculous it is to configure and use.
P40s are definitely the best value along with P100s. I still think there is a good amount of value to be extracted from both of those cards, especially if you are interested in using Whisper ASR, video transcoding, or CUDA models that were relevant before LLMs (a time many people have forgotten apparently).
I tried the a770, but returned it. Half the stuff does not work. They have the CPU side and GPU development on different branches (GPU seems to be ~6 months behind CPU) and often you have to compile it yourself, (if you want torchvision or torchaudio) it also currently on 2.0.1 of pytorch so somewhat lagging, and does not have most of the performance analysis software available. You also, do need to modify your pytorch code, often more than just replacing cuda for xpu as the device. They are also doing all development internally, then pushing intermittently to public. A lot of this would not be as bad if there was a better idea of feature timeline, or if they made their CI public. (Trying to build it myself involved a extremely hacky bash script, that inevitably failed halfway through.)