Apple: Transformer architecture optimized for Apple Silicon

Re:Apple: Transformer architecture optimized for Apple Silicon by month13 on Thursday March 23, 2023 @07:28PM (#35282996)

This is an older paper, but DeepMind alleges in their Chinchilla paper that far better performance can be extracted with fewer parameters; quote

"We find that current large language models are significantly under-trained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant."

It's difficult to evaluate a LLM's performance as it's all qualitative, but Meta's LLaMA has been doing quite well, at even 13B parameters.

[ Parent | on HN ]

Re:Apple: Transformer architecture optimized for Apple Silicon by astrange on Thursday March 23, 2023 @10:22PM (#35284279)

Chinchilla is aimed at finding a cost-performance tradeoff as well, not the optimal amount of training. If cost is no barrier because it'll be used forever, then probably there's no amount of training that's good enough.

[ Parent | on HN ]