This is purely a developer kit, letting people get used to the hardware configuration and the software stacks before the big hardware comes later in future generations.

I'd like to know what different precisions/ quantizations if any are supported. For LLMs 8GB is fine for playing around if weights are quantized. And as the article mentions it's more than enough for lots of computer vision models.

Grayskull supports FP8, FP16, BFP16, VFP2, BFP4, BFP8 [0], and some sort of 19-bit floating point in the SFPU. [1]

I don't know how and with what performance those are supported.

FP4 has 221 TFLOPS on the Grayskull e75, and 332 TFLOPS on the Grayskull e150. [0]

While wormhole has 82 INT8 TOPS per card. [2]

I haven't found any other numbers.

Edit: Some more info about data formats: https://docs.tenstorrent.com/tenstorrent/v/tt-buda/dataforma...

[0] https://docs.tenstorrent.com/tenstorrent/add-in-boards-and-c...

[1] https://tenstorrent-metal.github.io/tt-metal/latest/tt_metal...

[2] https://tenstorrent.com/systems/galaxy/

Cool, thank you!

That's a lot of formats for FP8, FP4, and even FP2! (never heard of). I wonder how they did it. Nice.

What's the memory interface like? Is an extreme NUMA thing with several orders greater potential for parallelization?

I'm probably dreaming; new CPU core designs are one thing, a completely new memory design is likely high fantasy..

I mean it says LPDDR4, nothing about how many channels but I would be surprised if it's anything more than dual-channel. Big cache but not anything not already seen in the AMD 3XD chips.

The quoted ~100-120GBps is going to be quad-channel LPDDR4

Will graphics cards eventually get renamed?

Are the optimizations for inference basically identical to graphics?

I imagine the workloads are different, and that inference has been the main market for years now...

The new things are already getting renamed, but the dust hasn't settled yet.

NPU: https://en.wikipedia.org/wiki/AI_accelerator

TPU: https://en.wikipedia.org/wiki/Tensor_Processing_Unit

why bother renaming? just re-acronym to "greatly parallel unit"

The website indicates each Tensix core ~1TOP theoretical. So not amazing.

It will have to compete on price/memory in the crowded inference space.

Maybe power usage ?

This card isn't competing on price - it's a dev kit.

Why are they so skimpy on memory capacity? I guess this is a dev kit for anything but LLMs?

Grayskull is make before LLMs being a thing. And their plan is like Groq, to distribute the compute graph across multiple processors to get higher effective memory and throughput by pipelining. But better-ish by having RAM so you can fit models on much less cards. Grayskull doesn't have this ability. The next generation Wormhole does by having 100GbE interfaces on the cards.

Also the CPUs on Grayskull is 32bit. Memory is addressed through the bank address so it works for now. But they'll have to upgrade to 64bit soon.