Re:Pathways Language Model (PaLM): Scaling to 540B parameters
by aceon48 on Monday April 04, 2022 @09:58PM
(#30914451)
Would that even work? Using the pretrained model, wouldn’t you then be in the “local” optimum?
Although from a transfer learning perspective, chopping off last layers and retraining… You are prob right
I believe we are near the end of that. As models get more and more expensive to train, we'll see future huge models being 'seeded' with weights from previous models. Eventually nation-state levels of effort will be used to further train such networks to then distribute results to industry to use.
A whole industry will be built around licensing 'seeds' to build ML models on - you'll have to pay fees to all the 'ancestors' of models you use.