Re:$1T in Tech Stocks Sold Off as Market Grows Skeptical of AI
by candiddevmike on Saturday November 08, 2025 @10:33AM
(#45857358)
We need a fundamental paradigm shift beyond transformers. Throwing more compute or data at it isn't pushing the needle.
[ Parent
| on HN ]
Re:$1T in Tech Stocks Sold Off as Market Grows Skeptical of AI
by marcosdumay on Saturday November 08, 2025 @10:45AM
(#45857475)
Just to point, but there's no more data.
LLMs would always bottleneck on one of those two, as computing demand grows crazy quickly with the data amount, and data is necessarily limited. Turns out people threw crazy amounts of compute into it, so the we got the other limit.
Re:$1T in Tech Stocks Sold Off as Market Grows Skeptical of AI
by Mistletoe on Saturday November 08, 2025 @10:55AM
(#45857564)
Yeah I’m constantly reminded of a quote about this- you can’t make another internet. LLMs already digested the one we have.
Re:$1T in Tech Stocks Sold Off as Market Grows Skeptical of AI
by bigyabai on Saturday November 08, 2025 @10:46AM
(#45857493)
Synthetic data works.
Re:$1T in Tech Stocks Sold Off as Market Grows Skeptical of AI
by marcyb5st on Saturday November 08, 2025 @11:03AM
(#45857630)
There's a limit to that according to: https://www.nature.com/articles/s41586-024-07566-y . Basically, if you use an LLM to augment a training dataset it will become "dumber" every subsequent generation and I am not sure how you can generate synthetic data for a language model without using a language model
Re:$1T in Tech Stocks Sold Off as Market Grows Skeptical of AI
by yorwba on Saturday November 08, 2025 @11:36AM
(#45857950)
Synthetic data doesn't have to come from an LLM. And that paper only showed that if you train on a random sample from an LLM, the resulting second LLM is a worse model of the distribution that the first LLM was trained on. When people construct synthetic data with LLMs, they typically do not just sample at random, but carefully shape the generation process to match the target task better than the original training distribution.
Re:$1T in Tech Stocks Sold Off as Market Grows Skeptical of AI
by bubblelicious on Saturday November 08, 2025 @05:34PM
(#45860659)
Epoch has a pretty good analysis of bottlenecks here:
https://epoch.ai/blog/can-ai-scaling-continue-through-2030
There is plenty of data left, we don’t just train with crawled text data. Power constraints may turn out to be the real bottleneck but we’re like 4 orders of magnitude away
Re:$1T in Tech Stocks Sold Off as Market Grows Skeptical of AI
by bubblelicious on Saturday November 08, 2025 @10:36AM
(#45857395)
And you don’t think that’s already happening? Also where is your evidence for this?
Re:$1T in Tech Stocks Sold Off as Market Grows Skeptical of AI
by bigyabai on Saturday November 08, 2025 @10:46AM
(#45857486)
> Also where is your evidence for this?
The fact that "scaling laws" didn't scale? Go open your favorite LLM in a hex editor, oftentimes half the larger tensors are just null bytes.
Re:$1T in Tech Stocks Sold Off as Market Grows Skeptical of AI
by bubblelicious on Saturday November 08, 2025 @10:57AM
(#45857588)
Show me a paper, this makes no sense of course scaling laws are scaling