Re:Meta AI Unleashes Megabyte, a Revolutionary Scalable Mode...
by godelski on Wednesday May 24, 2023 @02:16AM
(#36054709)
> if for each step of improvement you need 10x parameters and 100x training, you quickly run into a brick wall
Btw, PALM2 has far fewer parameters than V1.
> The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses
more training compute. Our evaluation results show that PaLM 2 models significantly outperform PaLM on a variety
of tasks, including natural language generation, translation, and reasoning. These results suggest that model scaling
is not the only way to improve performance.
From the leak we know that the large version is 340B parameters, compared to the original 540B parameters. From Table 2 in the document we see that the small version (unknown size) is on par with version 1.
ML typically follows a cycle. Improve, distill, repeat. It is unfortunate that the big labs lead these efforts because many smaller labs try to work on the distill part in parallel (out of necessity) but works get rejected (due to lack of SOTA) or ignored. Like all research, we need to be careful and nuanced in our evaluations. There's a lot of hype and many trying to take advantage of the confusion and sell snake oil. I think AI/ML is and will continue to change our world, but we have to be careful to not let the salesmen dictate the conversations.
Btw, PALM2 has far fewer parameters than V1.
> The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute. Our evaluation results show that PaLM 2 models significantly outperform PaLM on a variety of tasks, including natural language generation, translation, and reasoning. These results suggest that model scaling is not the only way to improve performance.
From the leak we know that the large version is 340B parameters, compared to the original 540B parameters. From Table 2 in the document we see that the small version (unknown size) is on par with version 1.
ML typically follows a cycle. Improve, distill, repeat. It is unfortunate that the big labs lead these efforts because many smaller labs try to work on the distill part in parallel (out of necessity) but works get rejected (due to lack of SOTA) or ignored. Like all research, we need to be careful and nuanced in our evaluations. There's a lot of hype and many trying to take advantage of the confusion and sell snake oil. I think AI/ML is and will continue to change our world, but we have to be careful to not let the salesmen dictate the conversations.
PaLM https://arxiv.org/abs/2305.10403
PaLM 2 https://arxiv.org/abs/2204.02311