That’s the rub, though. The bar for most tasks isn’t “decent” level of expertise. We want genuine expertise. It doesn’t matter if your Rust developer Jerry also knows how to write Italian operas about SpongeBob. He needs to write code that is big-free or be able to address bugs as they come up. As long as SOTA models are only “decent” Jerry keeps his job.

If it sounds like I’m moving the goalposts, I’m not. I acknowledge that this is impressive in the abstract. It’s fun to play around with. But I’m also predicting that we’re at a local maxima: there are diminishing returns to the architectures we’ve developed so far. Throwing more data and compute won’t solve the problems we have.