News for hackers. Stuff that matters.
Re:Meta AI Unleashes Megabyte, a Revolutionary Scalable Mode... by visarga on Wednesday May 24, 2023 @01:43AM (#36054502)
> I don't buy the idea (with either architecture) that "10x"-type scaling is required for another breakthrough.

Scaling can happen in two dimensions - model size and dataset size. What counts is the product of n_examples x n_parameters. That's why we have the super-Chinchilla laws, where n_examples >> 20*n_parameters. Scale the data, keep the model lean. Not to mention dataset quality - if you got clean diverse data you need less of it.

Another important trend is using LLMs to generate training sets. Example: ConstitutionalAI (pure RLAIF), TinyStories (scaling down LLMs), Alpaca (borrowed RLHF), AlpacaFarm - a recent paper promising fine-tuned models for $200 cost in 24h.

Re:Meta AI Unleashes Megabyte, a Revolutionary Scalable Mode... by BulgarianIdiot on Wednesday May 24, 2023 @02:06AM (#36054635)
Scaling can happen in many other places. Such as deeper iterative thought during inference such as Chain of Thought, Tree of Thought, which extracts increasingly better performance out of existing parameter and data sizes.