Re:What does a leaked Google memo reveal about the future of...
by candiodari on Monday May 15, 2023 @07:41AM
(#35946619)
The points raised in the paper are that
1) algorithmic improvements beat processing capacity. One of the examples in the paper was that $100 in training compute budget was enough to get within 2% of Google Bard performance
2) data quality (even for "tiny" data sizes) beats dataset size in improvements
Hence both aspects of the moat of large companies (dataset size and available compute) matter less than you'd expect.
In contrast the advantage of "open source"/academics/"the market", the many warm bodies trying out new strategies, yields improvements at a rapid pace that Google, even with a huge team, cannot match.
1) algorithmic improvements beat processing capacity. One of the examples in the paper was that $100 in training compute budget was enough to get within 2% of Google Bard performance
2) data quality (even for "tiny" data sizes) beats dataset size in improvements
Hence both aspects of the moat of large companies (dataset size and available compute) matter less than you'd expect.
In contrast the advantage of "open source"/academics/"the market", the many warm bodies trying out new strategies, yields improvements at a rapid pace that Google, even with a huge team, cannot match.