From the page : "Andromeda, a 13.5 Million Core AI Supercomputer". Blown away by the number of cores (I considered myself lucky to have 2 10000+ cores GPU in my workstation) I then realized that the word "core" is singular in the sentence. Is it just a mistake or does it mean something else ? (genuine question, English is not my first language)

EDIT: Ahhh a bit below on the page it is written "13.5 million AI-optimized cores" and there it's plural. So it was probably just a mistake.

It is not a mistake, that is how you phrase it in English when the noun ("core" in this case) is being used as part of a compound adjective. The convention is to keep the noun singular.

e.g.

"He commanded a ten thousand man army." (not men)

"Andromeda, a trillion star galaxy, is 2.5 million lightyears away." (not stars)

etc

Not a native English-speaking person, but shouldn’t it be "He commanded a ten-thousand-man army" and "Andromeda, a trillion-star galaxy, is 2.5 million lightyears away"?

* * *

I'm now remembering some WW1 posters from my GCSE History lessons that hyphenated "to-day" like this: https://www.pinterest.com/pin/483644447456350306/

That’s optional, but definitely more clear. The kind of thing a newspaper editor would insist on, but not necessarily seen outside of that.

To expand on the sibling comment, this is when pedantic people start talking about hyphenation. The clearer way to say this is "13.5-million-core AI supercomputer".

Yes, Google TPUs are sold to consumers under the Coral brand.

https://coral.ai/products/

This is certainly interesting, but doesn't seem quite like what I want. It only seems to offer "Edge TPUs" which...

> The Edge TPU ... supports only TensorFlow Lite models that are fully 8-bit quantized and then compiled specifically for the Edge TPU.

It's a different chip

Clearly; they're tiny and for embedded applications.

They even wrote "The products offered by Google are unrelated to the products offered under the CORAL trademarks" in the page footer.

These things come with a lot of limitations, like not being able to work with 4D and larger tensors, plus the 8 bit quantization. Getting models to run on them is a real pain.

This is like saying Fisher Price toy cars are equivalent to F1 cars.

Absolutely. On an hourly basis. Support is extra. https://cloud.google.com/tpu/pricing

Renting isn't purchasing.

Cerebres systems are whole rack with high cost and cooling requirement, pci cards are more sensible for home gamers:

https://tenstorrent.com/

How useful are these for workstation-class scientific computing, vs. a GPU? Not ML necessarily, but I do a lot of matrix multiplication for example when solving systems of equations.

We dont know yet, because they're not available. But Jim Keller has been all-in there for 2 years and they're supposed to be generally programmable.

Cant wait for tenstorrent card to come to public.

Grayskull is supposed to be A100 performance for $1000, with some cool features (horizontally scalable by plugging them to each other over ethernet, C++ programmable, sparse computation, etc.)

I really wonder how well it's going to perform given that it's 600 TOPS / 16GB DRAM 200 Gbit/s setup. I was told that in Transformer training, memory bandwidth is key. However, neither 16GB single-card memory capacity, nor bandwidth sounds terribly attractive but most important of all, judging from their FAQ, their driver is likely to be proprietary and would require Internet access which is worrying still.

Over the last couple weeks, I've seriously considered purchasing AMD Instinct MI50 (32GB HBM2, 1 Gbit/s) card which goes for under a $1000. I know it lacks Tensor Cores, and can only offer 53 TOPS which sounds silly compared to Grayskull's 600 TOPS. However, isn't it the case that for the most part these cores are idle, & waiting for memory still? At any rate, you're not going be able to run, say Llama 30B which won't fit already into 16GB but must fit comfortably in a 32GB system— on a single card. But perhaps most importantly, amdgpu driver is actually open source and allows PCIe passthrough unlike other vendors so considering all of the above, it almost seems like a no-brainer for a trusted computing setup.

I wonder if my logic is correct.

MI50: https://www.amd.com/en/products/professional-graphics/instin...

The idea is that the tenstorrent cards dynamically prune out codepaths that aren't used.

Transformers naturally have a big part of the network that is unused on any particular token flowing through the model. You could see that by how little RAM ended up being used in llama.cpp when they moved to mmaping the model.

So my understanding is that the tenstorrent cards are drastically more efficient, even on "dense" models like transformers because of the sparsity of any specific forward pass.

Also: I wouldn't bet on AMD accelerators for ML. They've disappointed every time. I would trust Jim Keller, whose every project in the last decade ended up being impactful.

I think the internet access is just to download the driver. It's not some sort of DRM setup where it needs to be always-online.

I understand this pruning is the difference between "silicon"and "software-assisted" TOPS in their documentation but I still don't see how exactly that's going to address the fact that to fit a 30B parameters model into memory, you need at least that. So basically to go 30B and up, you would need at least two, if not three cards and I couldn't find any details on how interlink is going to be implemented except that it's apparently done via Ethernet, limiting it to 100 Gbps which again, seems like a hard bandwidth limitation out-of-place when comparing it to impressive compute. Also: "online installer required" is not a good look, at least for me personally and the security model of my system. AMD cards, however, are affordable commodity hardware, offering 32GB HBM2 and the driver is open source so I wouldn't really discount it even considering their lacklustre performance. Cloud VMs do away nicely for most use-cases but as long as soon as hard security comes into focus, you just can't afford to have unauditable blobs from NVIDIA, or any other vendor for that matter. I'm still looking forward to learn more about Grayskull, especially on its memory capacity/bandwidth limitations and what it means for ever-growing large language models. Hopefully, they can open-source their driver stack.

Notice they require 32gb RAM and encourage 64gb RAM.

Presumably the architecture keeps the model in CPU RAM and shuffles it dynamically to the PCIe card network? I'm guessing here.

Whatever quirks come out of their hardware, I want to keep an eye on it.

I also think that comparing the current generation of models, which are built and trained to maximize GPU or TPU bandwidth, could be improved if someone architected models to maximize their model to greyskull advantages. Given PyTorch runs on it, I don't think it'd be too hard to do.

Thank you

> You could see that by how little RAM ended up being used in llama.cpp when they moved to mmaping the model.

From what I've read that was just an error in reading memory consumption after switching to the mmap version and not more memory efficient at all in the end.

Not exactly. It's that the model is loading less stuff out of the mmap'ed weights that you would expect.

The author of the mmap patch chimes in here:

https://news.ycombinator.com/item?id=35393615

> Transformers naturally have a big part of the network that is unused on any particular token flowing through the model. You could see that by how little RAM ended up being used in llama.cpp when they moved to mmaping the model.

Dense transformers (GPT-3 AFAIK is dense) don't.

They do though. A ton of the neurons end up not activating on any particular layer, leading to a huge waste as you're passing a zero down the layers.

I spoke with a PM at TT and he told me an important idea is that we're spending a lot of electricity multiplying things with zeros.

Seeing people calling this an NVDA A100 killer, being passed around. Any thoughts on this having as much of an impact as stated.

To be honest even if it fulfills half the expectations itll sell like hotcakes