When I say models will plateau I don't mean there will be no progress. I mean progress will slow down since we'll be scraping the bottom of the barrel for training data. We might never quite run out but once we've sampled every novel, web site, scientific paper, chat log, broadcast transcript, and so on, we've exhausted the rich sources for easy gains.

Chat logs don’t run out. We may run out of novelty in those logs, at which point we may have ran out of human knowledge.

Or not - there still knowledge in people heads that is not bleeding into ai chat.

One implication here is that chats will morph to elicit more conversation to keep mining that mine. Which may lead to the need to enrage users to keep engagement.

The necessity of higher quality data from vetted experts is why Mercor just raised at 10B

I’mafraid I don’t share your optimism. I think we are more or less seeing the limitations of the transformer architecture.