Re:We read the paper that forced Timnit Gebru out of Google
by confuseshrink on Saturday December 05, 2020 @09:33AM
(#25314790)
This is a very valid argument but it's hard to know what scaling a transformer will really do without trying (looking at you GPT-3). This is probably an issue for ML in general at this point.
I think a more nuanced conversation around these topics will look at exactly what you bring up, how do we properly trade the potential knowledge benefit against the costs?
It pains me that entirely valid avenues of research like this get covered up in nonsense and drama and their message seemingly lost in the midst of it.
Re:We read the paper that forced Timnit Gebru out of Google
by CydeWeys on Saturday December 05, 2020 @09:35AM
(#25314802)
Yes, an in-depth nuanced conversation is absolutely merited here, and Gebru and her colleagues were having it in the most rigorous way possible -- in peer-reviewed papers at AI conferences. I won't pretend to remotely be contributing as much to the discussion as they were.
It's not a fixed cost though, it's just how much was spent on this year's iteration of the model. The overall point being made is that model training costs are growing unbounded. Next year it could be 30 cars' worth or whatever for BERT-2, then 600 cars' worth for BERT-3 the year after. That's what it's warning against. At some point it isn't worth it.