News for hackers. Stuff that matters.

A comment on TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning. Parent comment

Re:TPU v4: An Optically Reconfigurable Supercomputer for Mac... by lostmsu on Wednesday April 05, 2023 @08:35PM (#35462534)
> Transformers naturally have a big part of the network that is unused on any particular token flowing through the model. You could see that by how little RAM ended up being used in llama.cpp when they moved to mmaping the model.

Dense transformers (GPT-3 AFAIK is dense) don't.

[ Parent | on HN ]
Re:TPU v4: An Optically Reconfigurable Supercomputer for Mac... by VHRanger on Wednesday April 05, 2023 @10:00PM (#35463173)
They do though. A ton of the neurons end up not activating on any particular layer, leading to a huge waste as you're passing a zero down the layers.

I spoke with a PM at TT and he told me an important idea is that we're spending a lot of electricity multiplying things with zeros.

[ Parent | on HN ]