> so parameter scaling is the primary way corporations are achieving increases in ability
Thinking tokens have nothing to do with parameter count and where some of the biggest gains have come from.
We're really past the point of scaling parameters being viable. We simply lack the content to make "bigger" better at this point.
No, parameter scaling still works because it broadly increases sample efficiency. We don't have the data to make a Chinchilla optimal 100T, but that doesn't mean there are no gains in building up to 100T.
I don't think "biology" and "elegantly" belongs in the same sentence. Let alone the one you wrote.
I suspect that the "elegant" workings of the human brain are a pile of hacks. Evolution crammed the brain full of patterns that are a good fit for the specific problems it typically solves - the way curve detectors and scale invariance are a good fit for CNNs.
We don't have millions of years of evolution, but what we have is enormous datasets and compute at scale.
I forget who said it, but a trenchant insight is that current neural networks are generalized function approximators and essentially they are doing through brute force what biology accomplishes elegantly. Our hardware (both in the sense of our brains in terms of elegance and our computer chips in terms of raw throughput) is amazing.