My understanding is that grayskull is a devboard for inference, Wormhole is supposed to do both, because you can scale it out with multiple cards.
> With CUDA you get a lot of flexibility which you don't get with Grayskull
Do you? From what I can tel tt-metalium gives you quite low level access.
Grayskull/Wormhole/... basically have a bunch of tensix cores, which each have 5 small rv32 cores, where some are used for data movement and some for computation. IIRC two (could be one) drive a SIMT compute unit.
From what I can tell there is no "matrix in matrix out" style fixed accelerator, the SIMT unit looks decently flexible, the grayskull one is a bit limited isa wise, wormhole is supposed to improve on that.
tt-buda is supposed to be the high level API with pytorch, tensorflow, ... support, but idk how much compatibility there is at the moment. I assume there is a lot of software work left.
Yet you got me to reading the following:
> t-Series Workstations
> Our t-series workstations are turnkey solutions for running training and inference on our processors, from a single-user desktop workstation in the t1000 up to the t7000 rack-mounted system designed specifically to function as a host with our Galaxy 4U Server
> 8 Grayskull Cards
> 16 Grayskull Chips
So these cards can also do training.
These are roughly analogous to compile time vs runtime for compiled programming languages.
Training is in general a more intensive task. However, in an ideal scenario training is run once and inference is run millions of times, so the lifetime cost of inference is bigger - this is why it might make sense to optimize for intense.
With CUDA you get a lot of flexibility which you don't get with Grayskull, at the expense of having a more layered/patched API which has been evolving for many years while R&D came up with new solutions to new problems.
I wonder what Nvidia's capabilities are in creating new, highly optimized cards, similar to Grayskull, but I wouldn't be surprised if they don't have any interest in creating them since their current products are already consuming all their resources.
It's great to see that companies like Tenstorrent are offering more optimized and pricewise more accessible products. It would also ease the situation with how hard it is to get Nvidia cards, because even companies doing only inferencing are buying chips which are developed to be able to do much more than that.
[0] https://tenstorrent.com/cards/