News for hackers. Stuff that matters.

A comment on Megatron-Turing NLG 530B, the World’s Largest Generative Language Model. Parent comment

Re:Megatron-Turing NLG 530B, the World’s Largest Generative ... by fspeech on Monday October 11, 2021 @02:31PM (#28831352)
A 10 trillion parameter model was mentioned here: https://mobile.twitter.com/ethancaballero/status/14458268620...
[ Parent | on HN ]
Re:Megatron-Turing NLG 530B, the World’s Largest Generative ... by ImprobableTruth on Monday October 11, 2021 @02:35PM (#28831395)
That's MoE.
[ Parent | on HN ]
Re:Megatron-Turing NLG 530B, the World’s Largest Generative ... by rolisz on Monday October 11, 2021 @04:19PM (#28832576)
Mixture of Experts, aka not all 10 trillion parameters are used at the same time, just a subset that is an "expert" on the "task at hand".
[ Parent | on HN ]
Re:Megatron-Turing NLG 530B, the World’s Largest Generative ... by jobigoud on Tuesday October 12, 2021 @02:28AM (#28836697)
That would also describe an organic brain though.
[ Parent | on HN ]