Re:GPT-3: A Disappointing Paper?
by p1esk on Saturday May 30, 2020 @05:51PM
(#23365091)
An average cost of an NLP researcher is probably around $300k/year. If buying 10 $5k cards makes them twice more productive, then it's a no brainer.
[ Parent
| on HN ]
Re:GPT-3: A Disappointing Paper?
by mattkrause on Saturday May 30, 2020 @08:28PM
(#23366102)
The average cost of an academic NLP researcher is probably closer to $30k/year.
Most universities have access to supercomputers, including GPU clusters. But that's not the point, not every NLP problem requires experimenting with 175B parameter models.
Academic researchers shouldn't try to compete with Google or OpenAI in scaling up models. They should try to come up with new approaches. Our brains have been evolving under tight constraints (size, energy, noise, etc). Maybe a good academic problem to solve is "how can I do what GPT-3 does if I only have an 8 GPU workstation?" This might lead to all kinds of breakthroughs.