YaLM-100B: Pretrained language model with 100B parameters

Re:YaLM-100B: Pretrained language model with 100B parameters by joshvm on Thursday June 23, 2022 @07:28AM (#31847589)

You could just run this on a desktop CPU, there's nothing stopping you in principle, you just need enough RAM. A big memory (256GB) machine is definitely doable at home. It's going to cost 1-2k on the DIMMs alone, less if you use 8x32GB, but that'll come down. You could definitely do it for less than $5k all in.

Inference latency is a lot higher in relative terms, but even for things like image processing running a CNN on a CPU isn't particularly bad if you're experimenting, or even for low load production work.

But for really transient loads you're better off just renting seconds-minutes on a VM.

[ Parent | on HN ]

Re:YaLM-100B: Pretrained language model with 100B parameters by sascha_sl on Thursday June 23, 2022 @08:11AM (#31847923)

From the readme, it looks like you need that RAM on your GPU.

[ Parent | on HN ]

Re:YaLM-100B: Pretrained language model with 100B parameters by joshvm on Thursday June 23, 2022 @02:06PM (#31852954)

There isn't any reason you can't run a neural net on a CPU. It's still just a bunch of big matrix operations. The advantage of the GPU is it's a lot faster, but "a lot" might be 1 second versus 10 seconds, and for some applications 10 seconds of inference latency is just fine (I have no idea how long this model would take). All the major ML libraries will operate in CPU-only mode if you request it.

[ Parent | on HN ]

Re:YaLM-100B: Pretrained language model with 100B parameters by visarga on Thursday June 23, 2022 @04:58PM (#31855075)

They are pretty slow even on GPU. The problem is that it's an autoregressive model. So it needs to do a forward pass for each token.

[ Parent | on HN ]